How Reranking Works: Recall Wide, Then Score Precisely

Your vector search returns the top 10 by cosine similarity. Fast, scalable, and approximate. The problem is that "approximate" means those 10 aren't always the real top 10.

Reranking fixes it with two stages.

First, a bi-encoder pulls 1,000 candidates out of a million documents in milliseconds. It embeds the query and each document separately, which is what makes it fast: you precompute every document vector ahead of time. The downside is it never looks at the query and document together, so it misses the subtle stuff.

Then a cross-encoder re-scores those 1,000. It reads each query and document as one input and sees both at once, which catches the token-level interactions the bi-encoder can't. That runs about 100ms per pair, so you only run it on the shortlist, never the full corpus.

What you get back is score redistribution. A document sitting at #47 by vector similarity can jump to #2 after the cross-encoder. The scores don't just reshuffle, they spread apart. The genuinely relevant results pull ahead and the false positives fall away.

So why not cross-encode everything? At 100ms a pair, scoring a million documents takes 28 hours. The bi-encoder narrows the field first.

Two variants worth knowing. MMR penalizes redundancy, so your top results stay diverse instead of five takes on the same thing. Learned reranking trains on your click and conversion data to fit your specific use case.

If you're building RAG or product search, or anything where the top 10 is what actually drives the outcome, reranking is where the precision comes from.