Mixture of Experts (MoE) Explained: Why a 671B Model Only Uses 37B Parameters
Summary
Dense language models wake every parameter for every token. Mixture of experts adds a tiny learned router that sends each token to the two specialists most likely to help, so Mixtral runs 46.7B total parameters with about 12.9B active, and DeepSeek-V3 runs 671B with 37B active. The video covers the router, expert collapse, and why load balancing is required, then shows how the same routing bet powers multimodal retrieval pipelines.
About this video
Dense language models wake every parameter for every token. Mixture of experts adds a tiny learned router that sends each token to the two specialists most likely to help, so Mixtral runs 46.7B total parameters with about 12.9B active, and DeepSeek-V3 runs 671B with 37B active. The video covers the router, expert collapse, and why load balancing is required, then shows how the same routing bet powers multimodal retrieval pipelines.