NEWVectors or files. Pick a path.Start →
    Back to DiagramsRetrieval Mechanics

    Sparse and Dense Are One Spectrum, Not Two Camps

    BM25, SPLADE and dense embeddings sit on one axis. What moves along it is how much vocabulary survives compression, which is why a rare identifier ranks below three passages that never mention it.

    Diagram placing BM25, SPLADE and dense embeddings on a single axis measuring how much vocabulary survives compression, with what each one stores and why production systems run more than one.
    BM25, SPLADE and dense embeddings sit on one axis. What moves along it is how much vocabulary survives compression, which is why a rare identifier ranks below three passages that never mention it.

    Ask a dense retriever for "error code E-4042 on the XR-9 valve" and it happily returns passages about valves, errors and hardware faults. Semantically adjacent, lexically wrong. The one document containing the literal string can rank below three that never mention it.

    That's not a bug in the model. Compress a passage into a single 1024-dimensional vector and you keep its gist. You lose the guarantee that any specific token survives.

    Sparse and dense get talked about as two camps. They're one axis, and what moves along it is how much vocabulary survives compression.

    BM25 sits at one end. One dimension per vocabulary term, no expansion at all. It scores by summing term frequency, inverse document frequency and a length normalisation over each query term. The formula matters less than the property: the literal term has to appear.

    SPLADE sits in the middle, and this is the part most people skip. It's still a sparse vector over the same roughly 30,000-term vocabulary, almost all implicitly zero. The difference is that a model assigns the weights and adds expansion terms the document never wrote. So "valve" can be weighted into a document that only said "spool". You pay a model forward pass at index and query time, and longer postings lists than raw BM25.

    Dense sits at the far end. One fixed-length vector, nothing in it corresponding to a term. That's exactly what lets it match paraphrase and cross-language, and exactly what loses the rare identifier.

    Published hybrid systems routinely report recall gains in the 15 to 30 percent range over dense-only on mixed workloads. The gain isn't spread evenly. It concentrates on rare-term queries, which is the same place dense search fumbles.

    Before deciding, pull a hundred of your real queries and count how many contain a token that has to match exactly. That number tells you where on this axis you need to sit. A benchmark won't.

    mxp.co/d/sparse-vs-dense

    [image: sparse-vs-dense.png]

    Where this diagram appears

    Run this on your own data

    Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

    Search your own data

    Related diagrams

    Contrastive learning trains a model to answer one question: are these two things related? You give it a batch of N anchor-positive pairs (an image and its own caption for CLIP, an audio clip and its description for CLAP) and nobody ever labels how similar a pair is, because the pairing itself is the entire supervision signal. Inside one batch each anchor has one positive and N-1 negatives, since every other item is by definition a non-match. CLIP scores the batch as an N by N grid of cosine similarities where the diagonal holds the true pairings, and InfoNCE pulls the diagonal together while pushing everything off it apart. Batch size is therefore a first-class hyperparameter rather than a throughput knob: CLIP trained at 32,768 so each anchor faced 32,767 simultaneous negatives and could not win on coarse cues. CLIP uses a softmax over the batch, SigLIP replaces it with a pairwise sigmoid that needs no global normalization and so trains stably at large scale.

    Contrastive Learning: How Two Encoders Learn One Space

    Train a dual encoder like CLIP and you would expect a matched image and its caption to land at nearly the same point in the shared space. They do not. All image embeddings occupy one narrow cone of the hypersphere and all text embeddings occupy a different, well-separated cone, and the two clouds barely overlap. A matched pair is closer to each other than to mismatched pairs, but both still sit inside their own modality's region, separated by a roughly constant offset vector. Three forces create and preserve it: a dual encoder runs each modality through a separate network with no shared weights, InfoNCE only requires a matched pair to outscore in-batch negatives which is satisfied with every image in cone A and every caption in cone B, and at initialization each encoder's outputs already occupy a small cone while a low softmax temperature sharpens matched against unmatched pairs without pulling the cones together. The model was trained to rank, not to register.

    The Modality Gap: A Shared Space Is Really Two Cones