NEWVectors or files. Pick a path.Start →
    Back to DiagramsIR Foundations

    Pixels in, signals out

    Extraction turns a raw frame into structured, searchable signals — the ceiling for every downstream search.

    Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row — face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.
    Extraction turns a raw frame into structured, searchable signals — the ceiling for every downstream search.

    Here's the unglamorous step that decides whether any of this works: extraction.

    A frame of video is pixels. Nothing about it is searchable. Extraction runs it through models that each pull out one kind of signal: a face and who it belongs to, the scene type, the transcript of what's being said, the text visible on screen, an embedding of the overall visual.

    Now the frame isn't pixels anymore. It's a row: timestamp, faces, scene, words spoken, words shown. Structured data, born from unstructured data.

    Most teams obsess over which LLM to use and treat extraction as plumbing. I think that's backwards. If a signal was never extracted, no model downstream can reason about it. Garbage in isn't even the right phrase. It's nothing in.

    Mixpeek is an extraction pipeline with a warehouse attached. That ordering is deliberate.

    The model gets the credit. The pipeline does the work.

    Where this diagram appears

    Run this on your own data

    Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

    Search your own data, free

    Related diagrams

    The unstructured-data problem: the large majority of enterprise data is unstructured — video, images, audio, and documents — that keyword systems and relational databases cannot search by meaning until it is turned into embeddings and indexed. The raw files are stored but not answerable.

    99% of your data has no rows

    An embedding turns meaning into coordinates: a model reads 'dog', 'puppy', and 'invoice' and outputs vectors, so 'dog' and 'puppy' land near each other while 'invoice' lands far away. Text, images, audio clips, video frames, and faces all map into one shared coordinate space where distance measures difference in meaning.

    An embedding is meaning turned into coordinates

    Semantic search as nearest-neighbor lookup: embed the query into the same vector space as your content, measure distance to every indexed point, and return the closest top-K results with no keyword matching. Exact brute-force compares against everything and is accurate but slow, while approximate nearest-neighbor (ANN) indexes like HNSW and IVF check a smart fraction of the space and return 95–99% of true answers in milliseconds.

    Search by meaning = nearest neighbors