MUVERA: Why Anisotropic Embeddings Break FDEs (and Mean-Centering Fixes It)

MUVERA's fixed encoding leans entirely on SimHash, which only works when the token vectors point in spread-out directions — a random hyperplane can only separate tokens that actually disagree in direction. This stress test (nfcorpus, ColBERTv2) induces anisotropy on purpose by adding a shared direction (scaled by alpha) to every embedding, driving the mean pairwise cosine from 0.26 up toward 1.0. As the vectors crowd onto one dominant component, raw FDE Recall@100 collapses from 0.73 at alpha=0 to 0.04 at alpha=8: tokens all fall into a handful of SimHash buckets, and the fixed encoding can no longer tell documents apart. Subtracting the corpus mean from every embedding before encoding restores it fully — the mean-centered line stays flat at about 0.725 across the entire sweep — at the cost of one extra pass over the data to estimate that mean. The lesson for deploying MUVERA on real, often anisotropic production embeddings: center first, or the FDE silently degrades.