Late Interaction: How ColBERT and ColPali Search Documents as Images
Summary
Bi-encoders squash query and document into single vectors (fast, lossy); cross-encoders read them jointly (accurate, unscalable). Late interaction (ColBERT, 2020) sits between: every token keeps its own embedding and they interact only at scoring, where each query word takes its best-matching token (MaxSim). ColPali (2024) extends this to visual documents: PaliGemma-3B embeds raw page images as patch vectors (D=128), queries hunt patches, and the OCR pipeline disappears from indexing. ViDoRe: 81.3 avg nDCG@5 vs 67.0 (captioning) and 66.1 (OCR + BGE-M3). Explore: mixpeek.com/visual-document-retrieval
About this video
Bi-encoders squash query and document into single vectors (fast, lossy); cross-encoders read them jointly (accurate, unscalable). Late interaction (ColBERT, 2020) sits between: every token keeps its own embedding and they interact only at scoring, where each query word takes its best-matching token (MaxSim). ColPali (2024) extends this to visual documents: PaliGemma-3B embeds raw page images as patch vectors (D=128), queries hunt patches, and the OCR pipeline disappears from indexing. ViDoRe: 81.3 avg nDCG@5 vs 67.0 (captioning) and 66.1 (OCR + BGE-M3). Explore: mixpeek.com/visual-document-retrieval