Late Interaction: How ColBERT and ColPali Search Documents as Images
Late interaction keeps one embedding per token (ColBERT) or per image patch (ColPali) and scores query–document pairs with MaxSim, recovering fine-grained matching that single-vector embeddings flatten away — ColPali retrieves documents as images, no OCR.
Where this fits in Mixpeek
This is the family Mixpeek's multi-vector search implements. Late interaction is why Mixpeek can match a query against the parts of a document — a specific paragraph, a chart, a region of a page — rather than one averaged vector, and ColPali's document-as-image approach is how visually rich PDFs get retrieved without a brittle OCR step. MUVERA (also in this hub) is the trick that makes serving it affordable.
About this research
Bi-encoders squash query and document into single vectors (fast, lossy); cross-encoders read them jointly (accurate, unscalable). Late interaction (ColBERT, 2020) sits between: every token keeps its own embedding and they interact only at scoring, where each query word takes its best-matching token (MaxSim). ColPali (2024) extends this to visual documents: PaliGemma-3B embeds raw page images as patch vectors (D=128), queries hunt patches, and the OCR pipeline disappears from indexing. ViDoRe: 81.3 avg nDCG@5 vs 67.0 (captioning) and 66.1 (OCR + BGE-M3). Explore: mixpeek.com/visual-document-retrieval
Frequently asked questions
Go deeper
Put the research to work
Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API — the retrieval stack these papers describe.
Search your own data, free
