A 20-billion-parameter model where only 2 billion are active at any given time just won gold at both the International Mathematical Olympiad and the International Physics Olympiad. No calculators, no internet searches, no tool-use tricks. Just pure reasoning through a looped Transformer architecture that keeps reusing its layers instead of stacking new ones. The results are hard to ignore: 35 points on IMO 2025 problems and 20.3 points on IPhO 2025, both above the gold-medal thresholds.

Layer-loop architecture diagram

The model is called Loopie, and it comes from a simple question that the Transformer-scaling community has been circling for years. What if, instead of making the model wider or deeper (the two dominant scaling axes of the past half-decade), you just ran the same layers more times? The idea has been around since the Universal Transformer in 2018, but nobody has made it work at this scale before.

The loop that changed the math

Standard Transformers stack layers one after another. Layer 1 feeds into Layer 2, which feeds into Layer 3, and so on for 30, 60, or even 120 layers. Each layer has its own weights, so the total parameter count grows linearly with depth. A looped Transformer does something stranger: it applies the same layer multiple times before moving to the next one.

The Loopie team calls this the "layer-loop" pattern, and it beats the earlier "model-loop" approach where you would repeat the entire stack of layers as one unit. With layer-looping, Layer 1 runs twice, then Layer 2 runs twice, and so on. The parameters stay local to the hardware cache, the gradients do not need to travel as far, and the shared parameters handle more similar functional demands than they would across widely separated depths.

This matters more than it sounds. The Loopie paper shows that a model with effectively 2 active layers (each run twice) outperforms a vanilla 30-layer model trained with the same compute budget. The numbers bear this out: after 600 billion tokens of training, Loopie-20B-A2B consistently surpasses the Qwen3-30B-A3B baseline. The throughput gap is significant at 261.53 TFLOPS/s against 189.65 TFLOPS/s, because the recurrent model has fewer unique parameters to move around the hardware.

The compute-matching trick

The hard part of comparing looped and non-looped models is that looping a model N times multiplies the training compute by N. If you test a 10-layer looped model against a 30-layer vanilla model, you are not comparing the same training budget. Earlier papers avoided this problem or handled it poorly.

The Loopie Recipe solves it with a three-step process. First, halve the stored layers and set recurrence depth R to 2, keeping the total block executions constant. This frees up activation memory because fewer unique layers need their states stored. Second, reinvest that memory into larger microbatches, which pushes hardware utilization higher. Third, take the efficiency gains and pump them back into capacity (slightly larger hidden dimensions or an extra stored layer) until the per-step training time matches the non-looped reference model.

The result is a fair benchmark. And the result of that benchmark is that looped Transformers, when measured correctly, beat vanilla Transformers at the same compute budget. Not match. Beat.

Olympiad-level reasoning

What makes Loopie especially interesting is the post-training pipeline. After initial pre-training on 3.5 trillion tokens, the model goes through what the authors call Supervised Pre-Training (SPT). Think of it as Supervised Fine-Tuning, but scaled to pre-training levels: 2 trillion additional tokens of high-quality reasoning, coding, and mathematics data, processed at pre-training batch sizes and sequence lengths. This lets the model acquire advanced reasoning without the catastrophic forgetting that smaller-scale SFT tends to cause.

The reinforcement learning stage afterward pushes it further. The model learns to allocate its looped computation adaptively, spending more recurrent steps on hard problems and fewer on easy ones. This test-time scaling is what lets a 2B-active-parameter model land gold at both the IMO and the IPhO, matching or exceeding dedicated math solvers with far more parameters.

On general benchmarks, Loopie-20B-A2B scores 81.28 on MMLU and 82.28 on BBH, roughly matching the Nemotron 3 Nano and Cascade 2 30B-A3B models while using only two-thirds of their pre-training compute.

What this changes

The conventional wisdom has been that frontier reasoning requires either massive scale (hundreds of billions of parameters) or specialized architectures (dedicated math models with tool use). Loopie challenges both assumptions. If you can get gold-medal olympiad performance from a model where 90% of parameters are dormant at any moment, the scaling conversation shifts from "how many parameters" to "how many times you reuse them."

The repo is already public under megatron-loopie and vllm-loopie, built on the familiar Megatron and vLLM frameworks. That matters because it means the looped architecture is implementable with existing infrastructure, not a research curiosity that requires custom hardware. The open-source community can start experimenting with recurrence depth as a first-class scaling axis.

There is a catch, and the paper is honest about it. Optimal recurrence depth depends on the task. Deep recurrence (R=16 or more) helps for iterative reasoning problems but offers marginal returns for general language modeling. The sweet spot for frontier training seems to be R=2, enough to get the compute efficiency gains without overfitting the recurrent pattern. Finding the right recurrence for the right task is still an open problem.

But that is a much more interesting open problem than "how to get more GPUs." Looped Transformers, done right, mean you trade GPU-seconds for architectural creativity. And that trade, for the first time, looks like a winning one.

Sources