1.  [](/)
2.  [Research](/research)
3.  Late Interaction: How ColBERT and ColPali Search Documents as Images

[Back to Research](/research)

Stanford & Illuin (ColPali)[arXiv:2407.01449](https://mxp.co/e/colpali)

# Late Interaction: How ColBERT and ColPali Search Documents as Images

Late interaction keeps one embedding per token (ColBERT) or per image patch (ColPali) and scores query-document pairs with MaxSim, recovering fine-grained matching that single-vector embeddings flatten away: ColPali retrieves documents as images, no OCR.

## Where this fits in Mixpeek

This is the family Mixpeek's multi-vector search implements. Late interaction is why Mixpeek can match a query against the parts of a document (a specific paragraph, a chart, a region of a page) rather than one averaged vector, and ColPali's document-as-image approach is how visually rich PDFs get retrieved without a brittle OCR step. MUVERA (also in this hub) is the trick that makes serving it affordable.

## About this research

Bi-encoders squash query and document into single vectors (fast, lossy); cross-encoders read them jointly (accurate, unscalable). Late interaction (ColBERT, 2020) sits between: every token keeps its own embedding and they interact only at scoring, where each query word takes its best-matching token (MaxSim). ColPali (2024) extends this to visual documents: PaliGemma-3B embeds raw page images as patch vectors (D=128), queries hunt patches, and the OCR pipeline disappears from indexing. ViDoRe: 81.3 avg nDCG@5 vs 67.0 (captioning) and 66.1 (OCR + BGE-M3). Explore: mixpeek.com/visual-document-retrieval

colbertcolpalilate-interactionmulti-vectorretrievalvision-language

## Frequently asked questions

What is late interaction in retrieval?+

How does ColPali search documents without OCR?+

## Go deeper

*   [Guide: Late-Interaction Retrieval (ColBERT, ColPali, MUVERA) →](/guides/late-interaction-retrieval)
*   [Diagram: Multi-vector search →](/diagrams/multi-vector-search)
*   [Research: MUVERA →](/research/muvera)

Put the research to work

Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API: the retrieval stack these papers describe.

[Search your own data](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=research_detail&utm_campaign=mvs_onramp)

## Related research

[

![Google's MUVERA: The One-Vector Trick Behind Fast Multi-Vector Search](https://cdn.mixpeek.com/videos/thumbs/78-muvera.jpg)

Google Research

Google's MUVERA: The One-Vector Trick Behind Fast Multi-Vector Search



](/research/muvera)[

![How Meta Cut Its Ads Model Serving Cost From 24.6x to 3.9x (HILL, Explained)](https://cdn.mixpeek.com/videos/thumbs/63-meta-hill.jpg)

Meta

How Meta Cut Its Ads Model Serving Cost From 24.6x to 3.9x (HILL, Explained)



](/research/meta-hill-ads)[

![Composed Image Retrieval With No Vision Model: 44% R@1 Zero-Shot](https://cdn.mixpeek.com/videos/thumbs/73-vfcir.jpg)

University of Tokyo

Composed Image Retrieval With No Vision Model: 44% R@1 Zero-Shot



](/research/vision-free-cir)