Multi-Vector Search vs. a Single Embedding

Most vector search stores one embedding per document. That works for text paragraphs. It doesn't work when your document is a 60-second video with speech, faces, scene transitions, and on-screen text.

Multi-vector search stores N vectors per document — one per visual frame, transcript segment, detected face, OCR region. The document isn't a point in vector space. It's a set of points.

The scoring mechanism: late interaction (MaxSim). Every query vector is scored against every document vector. Take the max similarity per query vector. Sum them. That's your relevance score.

Why this matters for multimodal:

→ Cross-modal matching — a text query matches visual, audio, and text vectors independently

→ No information loss — every frame and segment keeps its own embedding

→ Fine-grained relevance — partial matches contribute at the token-to-vector level

Single-vector search collapses your data into one point and hopes for the best. Multi-vector search keeps full fidelity and scores at the granularity that actually matters.

mixpeek.com