What is Sparse Retrieval

How It Works

Sparse retrieval represents documents and queries as high-dimensional vectors where each dimension corresponds to a vocabulary term. Traditional methods like BM25 use term frequency statistics, while learned sparse methods like SPLADE use neural networks to assign importance weights to terms. Retrieval uses inverted indices for efficient lookup of documents matching query terms.

Technical Details

Classical sparse vectors have dimensionality equal to vocabulary size (typically 30K-100K) with only a few hundred non-zero entries per document. SPLADE and other learned sparse models expand documents with related terms and learn term weights end-to-end. Sparse retrieval excels at exact matching, entity lookup, and queries with rare technical terms that dense models may not handle well.

Best Practices

• Use learned sparse models (SPLADE) over raw BM25 for significantly better relevance
• Combine sparse and dense retrieval in hybrid search for the best of both approaches
• Tune the sparsity regularization parameter to balance effectiveness and efficiency
• Index sparse vectors in inverted indices for sub-millisecond retrieval

Common Pitfalls

• Dismissing sparse retrieval as outdated when learned sparse methods are competitive with dense
• Not applying proper tokenization and stemming for term-based sparse representations
• Over-regularizing sparse models, leading to too few terms and poor recall
• Ignoring the complementary strengths of sparse and dense retrieval

Advanced Tips

• Use SPLADE with distillation from a cross-encoder for state-of-the-art sparse retrieval
• Implement efficient sparse-dense hybrid scoring with learned interpolation weights
• Apply document expansion techniques to enrich sparse representations with related terms
• Use sparse retrieval as a first-stage candidate generator before dense reranking