RL Changes 1-3% of Tokens, and They Were Already in the Top 5
Summary
A new arXiv paper (2605.06241) makes a precise mechanical claim about why reinforcement learning improves LLM reasoning: it is sparse policy selection, not capability learning. After RL training, only 1-3% of token positions change, and at those positions the promoted token always lies within the base model's top-5 alternatives. The model already contained the solution; training moved probability onto it. The authors' RL-free method, ReasonMaxxer, applies a contrastive loss only at high-entropy fork positions where the model hesitates, trains on tens of problems in minutes of single-GPU time, and matches RL outcomes at roughly three orders of magnitude lower training cost, holding across three model families, six scales, and six math reasoning benchmarks. If selection is this cheap, the expensive part of RL was the search, and knowing where the forks are is most of the work. Paper: arxiv.org/abs/2605.06241. More research we track: mixpeek.com/research
About this video
A new arXiv paper (2605.06241) makes a precise mechanical claim about why reinforcement learning improves LLM reasoning: it is sparse policy selection, not capability learning. After RL training, only 1-3% of token positions change, and at those positions the promoted token always lies within the base model's top-5 alternatives. The model already contained the solution; training moved probability onto it. The authors' RL-free method, ReasonMaxxer, applies a contrastive loss only at high-entropy fork positions where the model hesitates, trains on tens of problems in minutes of single-GPU time, and matches RL outcomes at roughly three orders of magnitude lower training cost, holding across three model families, six scales, and six math reasoning benchmarks. If selection is this cheap, the expensive part of RL was the search, and knowing where the forks are is most of the work. Paper: arxiv.org/abs/2605.06241. More research we track: mixpeek.com/research