Thompson Sampling: Why Your A/B Test Wastes Half Its Traffic
Summary
A fixed 50/50 A/B test sends half of every cohort to the inferior variant until the test ends. Multi-armed bandits reframe the decision as allocation under uncertainty, and Thompson sampling (1933) solves it in one loop: keep a belief distribution per option, sample once from each every round, play the highest sample, update with the result. Uncertain options explore themselves, winners take over automatically, and regret is provably near-optimal (Agrawal & Goyal 2012). Netflix picks homescreen artwork with contextual bandits; ranking under uncertain relevance is the same math. Explore: mixpeek.com/glossary/relevance-ranking
About this video
A fixed 50/50 A/B test sends half of every cohort to the inferior variant until the test ends. Multi-armed bandits reframe the decision as allocation under uncertainty, and Thompson sampling (1933) solves it in one loop: keep a belief distribution per option, sample once from each every round, play the highest sample, update with the result. Uncertain options explore themselves, winners take over automatically, and regret is provably near-optimal (Agrawal & Goyal 2012). Netflix picks homescreen artwork with contextual bandits; ranking under uncertain relevance is the same math. Explore: mixpeek.com/glossary/relevance-ranking