The Short Answer
MUVERA turns multi-vector retrieval into single-vector retrieval. It encodes an entire set of token embeddings, the kind ColBERT and ColPali produce, into one fixed-length vector called a Fixed Dimensional Encoding, or FDE. The encoding is built so that an ordinary inner product between a query FDE and a document FDE approximates the MaxSim score the multi-vector model would have computed over all the token pairs. Since the output is a single vector, you can serve it from whatever maximum inner product search index you already run, then rescore the surviving handful of candidates with exact MaxSim.
The paper is "MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings" by Laxman Dhulipala, Majid Hadian, Rajesh Jayaram, Jason Lee and Vahab Mirrokni (arXiv:2405.19504). It reports the same recall as earlier heuristics while retrieving 2x to 5x fewer candidates, and an average of 10% higher recall at 90% lower latency across the BEIR datasets compared to the best prior implementations.
Why Multi-Vector Retrieval Is Expensive In The First Place
A dense retriever gives you one vector per document. That is what makes vector search cheap: one vector goes into an approximate nearest neighbor index, and a query is one lookup.
Late interaction models keep a vector for every token instead. A 500-token document becomes 500 vectors. Scoring is no longer a dot product between two vectors, it is the Chamfer similarity, usually called MaxSim:
Chamfer(Q, P) = sum over q in Q of ( max over p in P of <q, p> )
For each query token, find its best-matching document token, then add those maxima up. That scoring rule is why late interaction wins on entity-heavy and exact-match queries, and it is also why it is awkward to serve. The score is not an inner product between two points, so there is no single vector you can hand to a MIPS index. Systems have historically worked around this by indexing all the individual token vectors and running pruning heuristics to assemble candidate documents, which means a lot of postings to touch and a lot of partial scores to track.
MUVERA attacks the shape of the problem rather than the constant factors. If you can build one vector per document whose inner product behaves like MaxSim, the whole serving problem collapses back to the thing the industry already knows how to do fast.
Start with a partition function that maps any embedding to one of B buckets. MUVERA uses SimHash. Sample k random Gaussian vectors, take the sign of the inner product with each one, and read the resulting bit string as a bucket index. That gives B = 2^k buckets, and vectors pointing in similar directions tend to land in the same bucket.
This asymmetry is the part that makes the approximation work, and it is easy to miss.
For a query, each bucket holds the sum of the query token vectors that fell into it.
For a document, each bucket holds the centroid, the mean of the document token vectors that fell into it.
Averaging on the document side is what stops overcounting. If five document tokens land in the same bucket as one query token, summing them would inflate that bucket's contribution fivefold, when MaxSim only ever wanted the single best match. Taking the mean keeps the bucket's contribution on the scale of one token.
A document has far fewer tokens than there are buckets, so most buckets are empty. Leaving them as zeros throws away information about documents whose tokens sit just outside a bucket a query token landed in. MUVERA fills an empty bucket k with the document token whose own bucket is closest to k in Hamming distance. Query FDEs skip this step, which is another piece of the asymmetry.
One random partition is noisy. The construction is repeated several times with independent partitions and independent Gaussian projections, and the blocks are concatenated. The final dimension works out to:
MUVERA attacks the shape of the problem rather than the constant factors. If you can build one vector per document whose inner product behaves like MaxSim, the whole serving problem collapses back to the thing the industry already knows how to do fast.
What A Fixed Dimensional Encoding Actually Is
Partition the space with SimHash
Start with a partition function that maps any embedding to one of B buckets. MUVERA uses SimHash. Sample k random Gaussian vectors, take the sign of the inner product with each one, and read the resulting bit string as a bucket index. That gives B = 2^k buckets, and vectors pointing in similar directions tend to land in the same bucket.
Aggregate queries and documents differently
This asymmetry is the part that makes the approximation work, and it is easy to miss.
For a query, each bucket holds the sum of the query token vectors that fell into it.
For a document, each bucket holds the centroid, the mean of the document token vectors that fell into it.
Averaging on the document side is what stops overcounting. If five document tokens land in the same bucket as one query token, summing them would inflate that bucket's contribution fivefold, when MaxSim only ever wanted the single best match. Taking the mean keeps the bucket's contribution on the scale of one token.
Fill the empty document buckets
A document has far fewer tokens than there are buckets, so most buckets are empty. Leaving them as zeros throws away information about documents whose tokens sit just outside a bucket a query token landed in. MUVERA fills an empty bucket k with the document token whose own bucket is closest to k in Hamming distance. Query FDEs skip this step, which is another piece of the asymmetry.
Repeat, project, concatenate
One random partition is noisy. The construction is repeated several times with independent partitions and independent Gaussian projections, and the blocks are concatenated. The final dimension works out to:
d_FDE = B * d_proj * R_reps
where B is the bucket count, d_proj the projected width of each block, and R_reps the number of repetitions. Those knobs are the whole quality-versus-size dial. An optional final projection compresses the result further.
Take one bucket. The query side holds the sum of query tokens that fell there, the document side holds the mean of document tokens that fell there. Multiply them out and you get the average inner product between those query tokens and those document tokens.
Now think about what a good partition does. A query token and its true best-matching document token point in similar directions, so SimHash tends to drop them in the same bucket. When that happens, the bucket recovers roughly the value MaxSim wanted for that query token. Summing across buckets approximates summing the per-token maxima, which is Chamfer similarity.
The paper proves this is an epsilon-approximation with high probability, which is the genuinely new part. Earlier ways of squashing multi-vector sets into one vector were heuristics with no bound on how wrong they could be. MUVERA is described in the abstract as the first single-vector proxy for multi-vector similarity carrying theoretical guarantees.
Note the direction of the error. FDE similarity is an approximation, so the ranking it produces is approximate. That is fine for deciding which few hundred documents deserve a closer look, and not fine as a final score.
Retrieval runs in two stages, and the second one is not optional:
1. Query a single-vector index with the query FDE and pull the top candidates. The paper uses DiskANN. 2. Rescore those candidates with the original Chamfer similarity over the real token vectors, and return that ranking.
So you still store the token vectors. The FDE buys you a cheap, index-friendly way to shrink the candidate set from the whole corpus down to something you can afford to score exactly. Anyone planning to drop the token vectors after computing FDEs has misread what the encoding does.
The last row is worth sitting with, because it is what most teams actually run today and it is the closest competitor. A dense vector plus a late interaction reranker is simple and often good. Its ceiling is recall: the reranker can only reorder what the first stage handed it, so a document the dense retriever missed is gone. MUVERA's first stage is derived from the same token vectors the final score uses, which is why its candidate sets are better aligned with the final ranking.
Reach for it when you are already committed to late interaction, your corpus is large enough that token-level serving hurts, and you want to keep using vector infrastructure you already operate. It is also attractive when your retrieval quality is bounded by first-stage recall rather than by reranking, since that is the specific failure MUVERA addresses.
Skip it when a single dense vector already answers your queries well. Late interaction earns its cost on entity-rich, exact-match and long-document queries, and if yours are broad semantic questions you may be buying storage and complexity for no measurable gain. Skip it too when your corpus is small enough to score exactly, because nothing beats the real thing when you can afford it.
The honest constraint: MUVERA adds a vector per document on top of token storage you are still paying for. It trades index size and build-time complexity for query latency and recall. That trade is good at scale and pointless below it. Our vector database cost comparison walks through the storage arithmetic if you want to price it before committing.
Mixpeek does token-level multimodal indexing and retrieval over object storage. An FDE is a vector like any other, so if you are generating them yourself, Mixpeek Vector Store will hold them on your own S3, GCS or B2 bucket and serve the first stage, while the token vectors sit alongside for the exact rescoring pass. That is the bring-your-own-vectors path, and it keeps the encoding entirely in your control.
If you would rather not run the encoder at all, the managed pipeline indexes objects and exposes late interaction retrieval through multi-stage retrievers, where a cheap first stage feeds an exact scoring stage. The shape is the same idea MUVERA formalizes: retrieve approximately, score exactly, and keep the two clearly separated. See pricing for what each path costs, or the docs for the retriever stage reference.
No. MUVERA replaces the candidate generation stage, not the scoring stage. The FDE inner product is an approximation of Chamfer similarity, and the published pipeline rescores retrieved candidates with the original Chamfer similarity before returning results. You can still add a cross-encoder on top of that if your quality budget justifies a third pass.
It is set by the construction rather than by the model. The final dimension is the bucket count multiplied by the projected block width multiplied by the number of repetitions, and an optional final random projection can compress it further. More buckets and more repetitions buy a better approximation and cost more space, so the dimension is a tuning decision rather than a fixed number.
No, and the difference is the point. Mean pooling collapses every token into one average with no guarantee about how far the result drifts from MaxSim. MUVERA partitions the space first, aggregates within each bucket, treats queries and documents asymmetrically, and comes with a proof that the inner product approximates Chamfer similarity within a bounded error.
Because MaxSim takes one best match per query token, not a total over all matching document tokens. If several document tokens land in the same bucket as a single query token, summing them would count that document several times over for one query term. Averaging on the document side keeps each bucket's contribution at the scale of a single token match.
The first stage needs maximum inner product search over a fixed-dimension vector, which almost every vector index supports, so that half is portable. The rescoring stage is the constraint: you need the original token vectors stored somewhere you can fetch them quickly for the candidate set, and you need somewhere to run the exact Chamfer computation. Databases that cannot store or retrieve the per-token vectors will handle stage one and leave you to build stage two.
The algorithm cares about sets of vectors and not about what produced them, so a model emitting per-patch or per-frame embeddings fits the same construction. ColPali and ColQwen produce exactly that shape for document images. The practical caveat is set size: video can emit far more vectors per item than a text passage, which pushes up both the storage you are trying to tame and the cost of the exact rescoring pass.
Late Interaction Retrieval -- ColBERT, ColPali and ColQwen, the models that produce the vectors MUVERA encodes
Budget-Aware Multi-Vector Retrieval -- deciding how many vectors per item you can afford
Multi-Stage Retrieval -- the retrieve-then-rescore pattern MUVERA slots into
Cross-Encoder Reranking -- the optional pass after exact MaxSim scoring
Hybrid Search Fusion -- combining an FDE stage with lexical retrieval
BM25 Inverted Index Internals -- the lexical side of the same pipeline
Vector Database on Object Storage -- where to keep FDEs and token vectors
Best Vector Databases -- which indexes support the MIPS first stage
Models -- browse ColPali, ColQwen and other late interaction models
Research: MUVERA -- the short version, if you want the idea without the construction
Diagram: How a multi-vector set becomes one fixed vector
Diagram: Query and document asymmetry in FDEs
Diagram: Two-tier retrieval, cheap FDE recall then exact rerank
Why The Inner Product Lands Near MaxSim
Take one bucket. The query side holds the sum of query tokens that fell there, the document side holds the mean of document tokens that fell there. Multiply them out and you get the average inner product between those query tokens and those document tokens.
Now think about what a good partition does. A query token and its true best-matching document token point in similar directions, so SimHash tends to drop them in the same bucket. When that happens, the bucket recovers roughly the value MaxSim wanted for that query token. Summing across buckets approximates summing the per-token maxima, which is Chamfer similarity.
The paper proves this is an epsilon-approximation with high probability, which is the genuinely new part. Earlier ways of squashing multi-vector sets into one vector were heuristics with no bound on how wrong they could be. MUVERA is described in the abstract as the first single-vector proxy for multi-vector similarity carrying theoretical guarantees.
Note the direction of the error. FDE similarity is an approximation, so the ranking it produces is approximate. That is fine for deciding which few hundred documents deserve a closer look, and not fine as a final score.
The FDE Is A Candidate Generator, Not A Final Scorer
Retrieval runs in two stages, and the second one is not optional:
1. Query a single-vector index with the query FDE and pull the top candidates. The paper uses DiskANN. 2. Rescore those candidates with the original Chamfer similarity over the real token vectors, and return that ranking.
So you still store the token vectors. The FDE buys you a cheap, index-friendly way to shrink the candidate set from the whole corpus down to something you can afford to score exactly. Anyone planning to drop the token vectors after computing FDEs has misread what the encoding does.
How MUVERA Compares To The Other Ways To Serve Late Interaction
| Approach | What you index | Index type | Scoring quality | Main cost |
| Exact MaxSim over the corpus | every token vector | none, brute force | exact | unusable beyond small corpora |
| Token-level index with pruning heuristics | every token vector | inverted or ANN over tokens | near exact after rescoring | many postings touched per query, complex serving |
| Mean-pool tokens into one vector | one pooled vector | ordinary MIPS | lossy with no error bound | cheap, but discards the token detail you paid for |
| MUVERA FDE plus rescoring | one FDE per document, token vectors kept for rescoring | ordinary MIPS | approximate first stage, exact final scores | FDE dimension, and you still store token vectors |
| Dense first stage, MaxSim rerank | one dense vector, token vectors kept | ordinary MIPS | bounded by what the dense stage recalls | misses anything the dense retriever never surfaced |
When This Is Worth Building
Reach for it when you are already committed to late interaction, your corpus is large enough that token-level serving hurts, and you want to keep using vector infrastructure you already operate. It is also attractive when your retrieval quality is bounded by first-stage recall rather than by reranking, since that is the specific failure MUVERA addresses.
Skip it when a single dense vector already answers your queries well. Late interaction earns its cost on entity-rich, exact-match and long-document queries, and if yours are broad semantic questions you may be buying storage and complexity for no measurable gain. Skip it too when your corpus is small enough to score exactly, because nothing beats the real thing when you can afford it.
The honest constraint: MUVERA adds a vector per document on top of token storage you are still paying for. It trades index size and build-time complexity for query latency and recall. That trade is good at scale and pointless below it. Our vector database cost comparison walks through the storage arithmetic if you want to price it before committing.
Where Mixpeek Fits
Mixpeek does token-level multimodal indexing and retrieval over object storage. An FDE is a vector like any other, so if you are generating them yourself, Mixpeek Vector Store will hold them on your own S3, GCS or B2 bucket and serve the first stage, while the token vectors sit alongside for the exact rescoring pass. That is the bring-your-own-vectors path, and it keeps the encoding entirely in your control.
If you would rather not run the encoder at all, the managed pipeline indexes objects and exposes late interaction retrieval through multi-stage retrievers, where a cheap first stage feeds an exact scoring stage. The shape is the same idea MUVERA formalizes: retrieve approximately, score exactly, and keep the two clearly separated. See pricing for what each path costs, or the docs for the retriever stage reference.
Frequently Asked Questions
Does MUVERA replace reranking?
No. MUVERA replaces the candidate generation stage, not the scoring stage. The FDE inner product is an approximation of Chamfer similarity, and the published pipeline rescores retrieved candidates with the original Chamfer similarity before returning results. You can still add a cross-encoder on top of that if your quality budget justifies a third pass.
How large is an FDE?
It is set by the construction rather than by the model. The final dimension is the bucket count multiplied by the projected block width multiplied by the number of repetitions, and an optional final random projection can compress it further. More buckets and more repetitions buy a better approximation and cost more space, so the dimension is a tuning decision rather than a fixed number.
Is this just mean-pooling ColBERT vectors into one vector?
No, and the difference is the point. Mean pooling collapses every token into one average with no guarantee about how far the result drifts from MaxSim. MUVERA partitions the space first, aggregates within each bucket, treats queries and documents asymmetrically, and comes with a proof that the inner product approximates Chamfer similarity within a bounded error.
Why do queries use a sum while documents use an average?
Because MaxSim takes one best match per query token, not a total over all matching document tokens. If several document tokens land in the same bucket as a single query token, summing them would count that document several times over for one query term. Averaging on the document side keeps each bucket's contribution at the scale of a single token match.
Can I use MUVERA with any vector database?
The first stage needs maximum inner product search over a fixed-dimension vector, which almost every vector index supports, so that half is portable. The rescoring stage is the constraint: you need the original token vectors stored somewhere you can fetch them quickly for the candidate set, and you need somewhere to run the exact Chamfer computation. Databases that cannot store or retrieve the per-token vectors will handle stage one and leave you to build stage two.
Does this work for images and video, not just text?
The algorithm cares about sets of vectors and not about what produced them, so a model emitting per-patch or per-frame embeddings fits the same construction. ColPali and ColQwen produce exactly that shape for document images. The practical caveat is set size: video can emit far more vectors per item than a text passage, which pushes up both the storage you are trying to tame and the cost of the exact rescoring pass.