What is multi-stage retrieval?
Multi-stage retrieval is a search pipeline that chains several narrowing steps instead of answering a query with one lookup. A cheap, high-recall stage pulls a few hundred candidates, then progressively more expensive stages filter, re-score and trim them down to the handful a user or an agent actually sees. The point is economic: the accurate scoring method is too slow to run over the whole corpus, so you only run it on candidates something cheaper already shortlisted.
Almost every production search system is multi-stage, including ones that do not describe themselves that way. A vector search followed by a reranker is a two-stage pipeline.
Why not just run one vector search?
Because a single approximate nearest-neighbour search optimises for the wrong thing at the wrong scale. It is fast because it does not compare your query against every document properly: it walks an index structure and returns things that are probably close. That approximation is fine for finding 200 plausible candidates and visibly imperfect at deciding which 5 are best.
The accurate method, scoring the query against each document jointly, is a cross-encoder. It is far better at ranking and far too slow to run over a million documents, because it cannot precompute anything: every query-document pair is a fresh forward pass.
So you split the work. Cheap and approximate for recall, expensive and accurate for precision, over a candidate set small enough to afford.
What does each stage cost and buy?
| Stage | What it does | Cost scales with | What it improves | Skip it when |
| Retrieve (ANN) | Finds approximate nearest neighbours in the index | Corpus size, sublinearly | Recall | Never; this is the funnel mouth |
| Filter | Drops candidates failing metadata predicates | Candidates, or index selectivity | Correctness, cost of later stages | You have no metadata worth filtering on |
| Rerank | Re-scores with a cross-encoder | Candidates, linearly, and it is the expensive one | Precision at the top | Your first-stage ordering is already good enough to ship |
| Dedupe / group | Collapses near-identical results | Candidates | Perceived quality | Your corpus has no near-duplicates |
| Enrich | Attaches data the ranking did not need | Final result count | Usefulness of the payload | The client can fetch it lazily |
How many candidates should each stage pass on?
There is no universal number, but there is a correct way to choose one. Reranking 100 candidates costs roughly ten times reranking 10, and the recall you can recover is capped by what the first stage retrieved: a document the ANN search never returned cannot be rescued by any reranker downstream.
So the question is not "what is a good k" but "at what k does the first stage already contain the right answer". Measure that directly. Take a set of queries with known-correct results, retrieve at k of 50, 100, 200, 500, and find where recall stops improving. That knee is your candidate count. Everything past it is latency you are paying for nothing.
Two failure modes sit either side of it. Too few candidates and the reranker is polishing a shortlist that never contained the answer, which looks like a reranker problem and is a retrieval problem. Too many and you pay cross-encoder latency on documents that were never going to place.
Should I filter before or after the vector search?
This is the decision that most often gets made by accident, and both answers are right in different regimes.
Post-filtering runs the vector search first, then discards candidates failing the predicate. It is simple and it silently under-returns: ask for 10 results with a filter matching 1% of the corpus and your 200 candidates may contain two survivors. The search succeeded, the filter was applied correctly, and the user sees an almost-empty page.
Pre-filtering restricts the search to documents matching the predicate. It returns the right count, and it can be much slower, because a highly selective filter fights the index structure: ANN indexes are built for traversing the whole space, and constraining them to a small subset can degrade toward a scan.
The practical rule is selectivity. A filter that keeps most of the corpus is fine post-hoc. A filter that keeps a small fraction wants pre-filtering, or an index partitioned on that field so the filter picks a partition instead of fighting the graph. The unhappy middle, filters keeping a few percent, is where you should measure rather than assume.
A filter on a field with no index is a third case and always wrong: it forces a scan whatever order you apply it in.
How do I set a latency budget across the stages?
Work backwards from the number the product needs, and give each stage a share before you build it. A budget assigned afterwards is a post-mortem.
Most of the variance lives in the reranker, because it is the only stage whose cost you control directly through candidate count. If the budget is tight, cut candidates before you cut stages: a pipeline that reranks 50 candidates is almost always better than one that skips reranking to afford 500.
Bound each stage individually AND bound the total. Per-stage ceilings compose: a pipeline where every stage is individually acceptable can still add up to something no user will wait for, and the failure mode you want is a partial result with a warning, not a timeout that returns nothing.
Does this work the same over video, audio and documents?
The architecture is identical; the unit changes. Text pipelines retrieve documents or chunks. Multimodal pipelines retrieve segments: a scene in a video, a passage on a page, a span of audio. That distinction matters more than it sounds, because the thing a user wants back is a timestamp or a region, not the file that contains it.
It also changes the dedupe stage from optional to essential. Adjacent video frames are near-identical, so a naive top-k returns ten views of one moment unless something collapses them. Grouping by the parent object and taking the best segment per group is the usual fix, and it belongs in the pipeline rather than in the client.
One more asymmetry: for multimodal content the filter stage often carries more weight than the reranker, because metadata like duration, speaker, brand or rights status eliminates candidates no embedding similarity would separate.
What usually goes wrong
Frequently Asked Questions
Is multi-stage retrieval the same as RAG?
No. RAG describes using retrieved context to ground a generated answer. Multi-stage retrieval describes how the retrieval half is structured. A RAG system usually contains a multi-stage retrieval pipeline, and plenty of multi-stage pipelines feed a ranked list to a human rather than a model.
How much does a reranker actually improve results?
Enough to be worth the latency in most systems, and the honest answer is that it depends on how good your first stage already is. The gain is largest when queries and documents are worded differently, because a cross-encoder reads them together and a bi-encoder never does. If your first-stage ordering already puts the right answer in the top 3, a reranker mostly reorders things nobody looks at.
Can I skip the vector search and rerank everything?
Only on a small corpus. Cross-encoder cost is linear in documents scored with no precomputation to amortise, so "rerank everything" is a full scan with an expensive scorer attached. At a few thousand documents that can be fine. At a million it is not.
Where should business logic go?
After ranking, as its own stage. Folding rules like "boost in-stock items" into the scoring function makes relevance and merchandising impossible to debug separately, and you will want to change them on different schedules.
What is the minimum useful pipeline?
Retrieve and filter. Add reranking when you can show the top-k ordering is the thing limiting quality, and add dedupe as soon as your corpus has near-duplicates, which for video is immediately.