The most expensive bug in LLM reinforcement learning may be a sampler that is only slightly different from the trainer. Not a broken reward function. Not a bad prompt. A different precision mode, a different reduction order, or a checkpoint that is 64 steps old can push policy-gradient training toward the sampler instead of toward the reward signal.

A new paper, "Score Centering Stabilizes Off-policy Reinforcement Learning," argues that this happens because training-inference mismatch creates a drift term inside the gradient. The authors propose a small additive correction called score centering. It does not make the sampler and trainer identical. It removes the part of the update that comes from their disagreement rather than from which sampled answers earned better rewards.

Training accuracy under sampler mismatch

That distinction matters if you run RL with a quantized inference engine, asynchronous rollouts, or a serving stack that cannot use exactly the same kernels as the trainer. Those choices improve utilization. They also create the conditions that can make RL unstable.

What actually breaks

Policy-gradient RL needs two views of the same model. The inference engine samples a response. The training engine then scores the sampled tokens and calculates gradients. In a perfect on-policy setup, both engines describe the same probability distribution. In a real system, they rarely do.

The mismatch can come from reduced precision, separate kernels, different batch shapes, or stale weights. Autoregressive sampling and parallel training do not always perform floating-point operations in the same order. At low precision, that matters. A sampler may also keep serving while the trainer takes updates, so its weights lag behind the current checkpoint.

The paper's useful move is to split the expected policy update into two pieces. One piece is the reward-sensitive signal. The other is drift, which is the sampler's expected score multiplied by the average reward. Drift does not tell the model which sampled token led to success. It nudges the trainer toward the sampler. If the sampler is a quantized or stale copy, the trainer keeps being pulled toward an imperfect target, then copied back to that target at the next synchronization point.

That explains why ordinary RL can fail even when the reward function looks sensible. Importance sampling tries to correct the distribution mismatch by weighting samples with a probability ratio. But those ratios can become large, which raises gradient variance. Clipping the ratios makes training safer, but adds bias. Score centering takes a different route: subtract the sampler-expected score so the correction has zero expectation under the sampling distribution.

The method was tested across models ranging from 0.6B to 30B parameters. The experiments covered synthetic weight noise, reduced precision, and deliberately stale samplers. They used Qwen3-0.6B on Countdown and Qwen3-30B-A3B-Base on the math subset of INTELLECT-2, so this is not a claim that every RL workload will behave the same way.

The quantization result is the one infrastructure teams should care about. In the severe FP4 KV-cache setting, uncorrected policy gradient collapses after roughly 200 steps. Score centering reaches about 52% training accuracy, compared with about 51% for truncated importance sampling in the reported comparison, a 52% vs 51% result in the paper's shorthand. With an FP8 sampler, ordinary policy gradient stayed near 58% in the same summary. The gap is small in the reported accuracy, but the stability difference is the point: one run keeps learning while the other falls apart.

When to use score centering

Use it first when the failure lines up with a sampler mismatch that you cannot remove without giving up useful throughput. Examples include a quantized sampler paired with a bf16 trainer, an inference engine updated less often than the trainer, or a disaggregated setup where rollout generation and optimization run on separate machines.

The paper's severe staleness test updated the inference engine every 64 steps. Under that condition, combining score centering with an importance-sampling correction performed better than pure importance sampling. That result suggests the two methods address different problems. Importance sampling corrects distribution weighting. Score centering removes the mismatch-induced drift. If staleness is the dominant problem, using both may be more sensible than endlessly tuning a clipping threshold.

Do not use score centering as an excuse to stop checking the boundary between sampler and trainer. If you can run matching precision, matching kernels, and frequent synchronization at a reasonable cost, that remains the cleaner system. A correction is useful when mismatch is an intentional tradeoff, not when it hides an accidental bug.

The implementation is available in the authors' GitHub repository. The code is the right place to start if you want to reproduce the correction, but reproduction is not cheap. The reported experiments total approximately 6,180 H100-hours. That number should change how you read the graphs. The evidence is substantial for a paper-scale study, yet most teams cannot casually repeat every sweep before making a production decision.

There are other limits. The 30B results use few seeds in some panels because the compute cost was high. The quantized sampler and bf16 trainer arrangement is deliberately designed to expose mismatch. Real systems may have a milder or differently shaped error. The experiments also do not prove that score centering fixes reward-model problems, bad data, environment bugs, or a policy that is simply learning the wrong behavior.

A sensible rollout is therefore diagnostic. First run a short baseline with the exact sampler and trainer pair you intend to use. Record reward, held-out accuracy, sampler precision, checkpoint age, and engine synchronization interval. Then add score centering without changing the rest of the stack. If the corrected run removes collapse while the baseline drifts, you have evidence that TIM is involved. If both runs fail, spend the next GPU budget on the reward, data, or environment instead.

The practical conclusion is narrower than the paper's title sounds. Score centering is not a new RL recipe that makes infrastructure irrelevant. It is a pressure-release valve for a very specific failure mode. That is still useful. In large asynchronous training systems, eliminating every mismatch can cost more than tolerating one. The trick is to know which mismatch you are paying for.

Sources