Contrastive Learning: How Two Encoders Learn One Space

Contrastive learning trains a model to answer exactly one question: are these two things related or not. You hand it a batch of N (anchor, positive) pairs, an image and its own caption for CLIP, an audio clip and its own description for CLAP, a query and a relevant document for a text model. Nobody ever labels how similar a pair is. The pairing is the entire supervision signal, which is why this scales to hundreds of millions of examples scraped off the internet.

Inside one batch, each anchor has exactly one positive and N-1 negatives, because every other item in the batch is by definition a non-match. CLIP scores the whole batch as an N by N grid of cosine similarities where the diagonal is the set of true pairings. Training pulls the diagonal together and pushes everything off it apart. That is InfoNCE: a softmax cross-entropy where the correct class is the matching positive and the wrong classes are the rest of the batch.

The consequence people miss is that batch size becomes a first-class hyperparameter rather than a throughput knob. CLIP trained at 32,768, so every anchor was contrasted against 32,767 negatives at once and could not win by learning coarse features like this-is-a-photo versus this-is-text. At batch 32 the task is easy enough to pass on superficial cues, which is why small-batch fine-tuning often produces embeddings less discriminative than the model you started from. The common floor is 256, and 1024 or more is better.

SigLIP changes one thing and it matters: it replaces the softmax with a pairwise sigmoid loss. A softmax has to normalize across the whole batch, which couples every example to every other and makes very large batches expensive to coordinate. A sigmoid scores each pair independently, so it trains stably at scale without that global normalization. Same idea, cheaper arithmetic.

What you get at inference is the part that pays for all of it: two encoders that map into one space, so a text query can retrieve an image it has never been paired with. The model was trained to rank pairs, and that is precisely what retrieval asks it to do.