NEWVectors or files. Pick a path.Start →
    Back to DiagramsIR Foundations

    Matryoshka Representation Learning: One Vector, Five Resolutions

    Why an agent can search a corpus coarsely and still rank the winners precisely, without re-embedding anything.

    An ordinary embedding model spreads information across all dimensions with no ordering guarantee, so dimension 900 may carry as much signal as dimension 5 and truncating the vector discards random pieces of meaning. Matryoshka Representation Learning changes the objective so the model packs the most general information into the front of the vector and progressively finer detail toward the back, which makes every prefix a complete usable coordinate. The nesting is drawn as five bars that all start at dimension 0 and grow: the first 64 dimensions carry coarse semantics, enough to tell a cat photo from a car photo; 128 is the prefix the retrieval cascade indexes the whole corpus at; 256 gives mid resolution, enough to separate breeds of cat or a refund request from a refund confirmation; and the full 1024 gives fine resolution for ranking near-duplicates. The mechanism is a modified loss rather than a special architecture: one forward pass produces one d-dimensional output, the training loss is computed independently at each prefix size in the set 64, 128, 256, 512, 1024, and those losses are summed, so the front of the vector has to work on its own. What it buys is a retrieve-coarse, rerank-fine cascade: index 10,000,000 items at a 128-dimension truncation costing 5.1 GB instead of the full 41 GB, retrieve a top-200 shortlist with the truncated query, fetch the full 1024-dimension vectors for those 200 candidates only, and return a final top 10.
    Why an agent can search a corpus coarsely and still rank the winners precisely, without re-embedding anything.

    An AI agent searching unstructured content faces a constant tension. To cover a corpus of millions of images, video frames, audio spans or document chunks it wants embeddings small enough to store cheaply and scan fast. To decide which few results to act on it wants embeddings expressive enough to separate near-duplicates. Those needs pull in opposite directions, and the naive fix is to pick one dimensionality and live with the compromise.

    Matryoshka Representation Learning removes the forced choice. It trains a single model so that the first k dimensions of every embedding are themselves a complete, usable embedding. Named after Russian nesting dolls, an MRL vector contains smaller vectors inside it: slice the first 64, 128, 256 or 512 dimensions off a 1024-dimensional vector and each prefix still works as a coordinate in a meaningful space.

    The mechanism is a modified loss function, not a special architecture. Take an ordinary encoder that outputs a d-dimensional vector, choose a set of nested prefix sizes, and during training compute the loss independently at each prefix and sum them. A single forward pass is scored five times, so the front of the vector is under pressure to work alone rather than only as part of the whole.

    The mental model that makes it click: a normal embedding is a photograph at one fixed resolution, and an MRL embedding is a progressive JPEG. The first bytes already give a recognizable thumbnail, and every additional byte sharpens it. You decide how much to load based on what the moment requires.

    The payoff is the retrieve-coarse, rerank-fine cascade. Index the whole corpus at a small prefix, retrieve a shortlist with the truncated query, then fetch full-dimension vectors for the shortlist alone and re-score. At 10,000,000 items embedded at 1024 dimensions in float32 the full vectors cost about 41 GB while a 128-dimension prefix index costs about 5.1 GB, an 8x reduction in the index that has to live in fast memory. The agent pays full-dimension cost on 200 items rather than 10,000,000, and recall survives because the coarse pass is good enough to keep the truly relevant items inside the net.

    Where this diagram appears

    Run this on your own data

    Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

    Search your own data

    Related diagrams

    Diagram placing BM25, SPLADE and dense embeddings on a single axis measuring how much vocabulary survives compression, with what each one stores and why production systems run more than one.

    Sparse and Dense Are One Spectrum, Not Two Camps

    A 3072-dimensional embedding encodes everything about a video and distinguishes nothing: lighting, camera angle, dominant color, who is on screen, the room and the vibe of the room are all smeared across the same floats at equal weight, with no name you can filter on. That is why a search for a person holding a coffee cup returns videos with people, videos with cups, and a surprising number with neither, and why reranking and hybrid BM25 each help only a little. Decomposition is where the dimension reduction actually lives: a shot detector turns a thirty-second video into eight scenes with timestamps, a face identity model turns a frame into bounding boxes and 512-dimension face embeddings, a multimodal extractor turns a scene into a Gemini embedding plus OCR output plus dominant colors, and a custom plugin does the same for a proprietary file type. Each extractor's output becomes its own collection with its own schema and its own embedding space, so the features are named and filterable where a smear is not. Reduction alone leaves a smarter vector database; the leverage is placement. A new ad is placed against a brand taxonomy at ingest, and because the taxonomy that locates it is the same retriever a user invokes to ask for ads in that brand, enrichment and search become one traversal in two directions. Six primitives compose in one direction: bucket, extractor, collection, retriever, taxonomy, cluster.

    The 3072-Dimension Problem: Reduce, Then Place

    Contrastive learning trains a model to answer one question: are these two things related? You give it a batch of N anchor-positive pairs (an image and its own caption for CLIP, an audio clip and its description for CLAP) and nobody ever labels how similar a pair is, because the pairing itself is the entire supervision signal. Inside one batch each anchor has one positive and N-1 negatives, since every other item is by definition a non-match. CLIP scores the batch as an N by N grid of cosine similarities where the diagonal holds the true pairings, and InfoNCE pulls the diagonal together while pushing everything off it apart. Batch size is therefore a first-class hyperparameter rather than a throughput knob: CLIP trained at 32,768 so each anchor faced 32,767 simultaneous negatives and could not win on coarse cues. CLIP uses a softmax over the batch, SigLIP replaces it with a pairwise sigmoid that needs no global normalization and so trains stably at large scale.

    Contrastive Learning: How Two Encoders Learn One Space