The Retrieval Funnel: Why Your Reranker Sees 400 of 10 Million
Summary
Ten million documents could answer one query, and the model best qualified to judge them, a cross encoder that reads query and document together, is the one you can least afford to run across a corpus. That single constraint is why production retrieval is a ladder rather than a model. ANN search over embeddings goes first because it is cheap per document, so it runs against everything and returns a shortlist. Filters cut further, and the order matters: filtering after scoring and filtering before scoring are different operations with different recall. Only then does the cross encoder run, affordable precisely because it inherits a shortlist. Cheap stages absorb the volume, costly stages inherit a shortlist. Full breakdown: mixpeek.com/glossary/multi-stage-retrieval-pipeline
About this video
Ten million documents could answer one query, and the model best qualified to judge them, a cross encoder that reads query and document together, is the one you can least afford to run across a corpus. That single constraint is why production retrieval is a ladder rather than a model. ANN search over embeddings goes first because it is cheap per document, so it runs against everything and returns a shortlist. Filters cut further, and the order matters: filtering after scoring and filtering before scoring are different operations with different recall. Only then does the cross encoder run, affordable precisely because it inherits a shortlist. Cheap stages absorb the volume, costly stages inherit a shortlist. Full breakdown: mixpeek.com/glossary/multi-stage-retrieval-pipeline