The Modality Gap: A Shared Space Is Really Two Cones

Train a dual encoder like CLIP and you would naively expect a matched image and its caption to land at nearly the same point in the shared space. They do not. All image embeddings occupy one narrow region, a cone, of the hypersphere, and all text embeddings occupy a different and well-separated region. The two clouds barely overlap.

A matched pair is still closer to each other than to mismatched pairs, which is why retrieval works at all. But both sit inside their own modality's cone, separated by an offset that is roughly constant in direction. Two facts make this counterintuitive. The gap is large relative to the within-modality spread, so raw cosine similarity is dominated by which-modality-is-this before it ever gets to what-does-this-mean. And because the offset is consistent, it is fixable.

Three forces create it and then keep it there. A dual encoder runs images through one network and text through another with no shared weights, so nothing forces the two into the same region of the output space. The contrastive objective only rewards a matched pair for scoring above the in-batch negatives, and that is perfectly satisfied with every image in cone A and every caption in cone B. And at initialization each encoder's outputs already occupy a small cone, while a low softmax temperature sharpens matched against unmatched pairs without ever pulling the cones together.

The cleanest framing: the model was trained to rank, not to register. Ranking is preserved by the gap; alignment is not, and nothing in the loss ever asked for alignment.

This is also why an absolute similarity threshold cannot transfer across modality pairs. A 0.3 text-to-image score can be as strong a match as a 0.7 image-to-image score, because the two are measured across different distances in the same space.