Multimodal Decomposition: One File, Many Signals

You don't search for a video. You search inside it.

Traditional search stores a video as a single record: one document, one embedding. You can find the video, but not the moment at 2:34 where someone says "quarterly revenue" next to a chart on the whiteboard.

Decomposition changes that. It breaks a file into segments, split on scene boundaries, silence gaps, or fixed time intervals, and each segment gets its own set of embeddings.

Visual: keyframes run through CLIP or SigLIP for per-frame vectors.

Audio: speech transcribed with Whisper, then embedded with E5.

Faces: detected and encoded with ArcFace, one vector each.

Text: on-screen text pulled with OCR, then embedded.

Now every segment is its own searchable document carrying several vectors. A 3 minute video can turn into 15 documents with 5 or more embeddings apiece.

The same idea carries across every content type. A PDF splits into pages and then paragraphs, each with text embeddings. An audio file splits on silence into transcript segments. An image gives you visual, OCR, and description vectors.

The payoff is that you can search by what was said, what was shown, what was written, or who showed up, at the segment level instead of the file level.

That's the real gap between multimodal search and dropping a video title into a vector database.