A $4,400 budget can buy a surprisingly serious language-model pretraining run. It can also disappear into a very expensive science project if you mistake a paper's compute ledger for a shopping list.

That tension is the useful part of Puro-2B, a 2-billion-parameter model released by the PACMAN Group at Tsinghua University. The authors trained it from scratch on consumer RTX 5090 GPUs, released the code, data artifacts, checkpoints, and weights, then published the accounting behind the run. The headline is easy to repeat. The decision is harder: should you try to reproduce this, rent the hardware, or download the weights and spend your time elsewhere?

Puro-2B reported reproduction cost

The short answer is that Puro-2B makes small-scale pretraining more inspectable, not effortless. The paper reports a uniform-data checkpoint at about $4.37K that exceeded Qwen2-1.5B on its evaluation panel. Its canonical run cost about $6.89K and came within 4.02 points of Qwen2.5-1.5B on the paper's four-task mathematics and code average. Those are useful reference points. They are not a promise that one RTX 5090 in a spare room will finish the same run next week.

The $5,090 ceiling is not the real budget

The first mistake is treating $5,090 as the price of the whole project. The authors' accounting boundary is much narrower. Their figures cover active training GPU-hours converted to a normalized RTX 5090 rental-equivalent rate. The ledger excludes data acquisition and preprocessing, proxy experiments, scaling and ablation work, failed or exploratory runs, evaluation, post-training, and checkpoint averaging.

That distinction changes the decision immediately. If you already have a cluster, clean data shards, storage, networking, and someone who can recover a distributed run at 3 a.m., the published number is a useful compute proxy. If you are buying hardware, the number is only one line in the budget. You still need enough VRAM, power delivery, cooling, fast local storage, spare capacity for failed jobs, and time to make the data pipeline behave.

The paper's own run schedule also makes the word consumer sound a little misleading. Phase 1 used 24 GPUs for 438.84 billion tokens and took 10.43 days. Phase 2 used 96 GPUs for either 480 billion tokens in the roughly $4.4K run or 959.99 billion tokens in the roughly $6.9K canonical run. The full canonical schedule processed 1.4 trillion tokens across its scheduled phases, consumed 22,514 active-training GPU-hours, and took 17.6 elapsed days.

In other words, this is consumer GPU hardware operated as a cluster. The paper does not describe a lone desktop quietly training a model beside a web browser. That matters more than the model's parameter count. It is the training-side version of the difference between open weights and runnable hardware.

The reported cost scaling law is still valuable. It estimates that about $4.4K of compute can reach the Puro recipe's Qwen2-1.5B performance target. But the authors explicitly frame the relationship as recipe-specific, not a universal law for every model family. Use it as a planning curve. Do not use it as a quote for what any 2B model must cost.

There is another reason to be cautious with the comparison. The paper reports an unweighted average of four generative benchmarks for mathematics and code. Puro-2B scored 43.50 on that average, versus 40.29 for Qwen2-1.5B and 47.52 for Qwen2.5-1.5B. On a separate eleven-benchmark reasoning and knowledge average, Puro-2B scored 63.02, compared with 60.54 for Qwen2-1.5B and 65.53 for Qwen2.5-1.5B. Those numbers say the model is competitive under the stated protocol. They do not say it is a drop-in replacement for every compact model or that benchmark parity means equal product quality.

A repeatable training recipe

A sensible reproduction attempt should start with the paper's smallest useful checkpoint, not the canonical headline. The uniform Puro-2B run is the controlled starting point because it costs about $4.37K in the authors' ledger and removes some of the extra Phase 2 choices. Once that path works, you can decide whether the curriculum and checkpoint averaging in the $6.89K run are worth the additional compute.

First, freeze the accounting boundary. Write down whether your estimate includes hardware purchase, rental, electricity, storage, data preparation, evaluation, and failed runs. If your spreadsheet only contains GPU-hours, label it compute cost. That single label prevents the most misleading comparison in this entire topic.

Second, use the released repository and configuration as the source of truth. Puro-Megatron is a fork of Megatron-LM adapted for the project. The paper describes a dense decoder-only Transformer with a Qwen3-1.7B-style configuration and untied input embeddings and output language-model head, producing about 2 billion parameters. Do not quietly swap in a different tokenizer, sequence length, optimizer, or data mixture and then call the result a reproduction.

Third, make the data pipeline boring before making it large. The project releases data manifests and materialized components, but the paper notes that upstream dataset terms remain component-specific. Check each component's license before redistributing shards. Build a small validation run that checks token counts, ordering, shard readability, and restart behavior. A corrupted shard found after a week of distributed training is not an optimization problem.

Fourth, reproduce the numerical choices that carry much of the cost advantage. Puro uses blockwise FP8 mixed precision, the MuonH optimizer with hyperball constraints, a carefully designed learning-rate schedule, and Curriculum Model Averaging for the canonical run. The paper reports that blockwise FP8 improved the throughput proxy while retaining a comparable quality fraction to BF16. It estimates a 1.34 times net speedup at matched quality in its ladder-derived comparison, but labels the full-horizon BF16 figure as a counterfactual rather than a measured run. Keep that qualifier in your notes.

Fifth, treat networking as part of the model. The run uses 24 GPUs in Phase 1 and 96 in Phase 2. At that point, inter-node communication, load balance, micro-batch choices, checkpoint writes, and restart handling can erase the savings from inexpensive cards. A smaller run with reliable communication is more useful than a larger run that spends half its time waiting at a barrier.

Finally, stop at a decision checkpoint. At roughly $4.4K, ask whether the uniform checkpoint already answers your research question. If the answer is yes, continuing to the $6.9K canonical run may only buy a nicer curve. If you need to study curriculum effects or want the strongest released checkpoint, the extra phase has a reason. If you only need a compact base model for an application, using the released Puro-2B weights costs less time and gives you a cleaner comparison.

That last option is easy to underrate. The collection contains ten model and checkpoint variants, including thu-pacman/Puro-2B-Base and intermediate phase artifacts. Downloading a checkpoint lets you test tokenizer behavior, inference speed, fine-tuning quality, and data fit before committing thousands of dollars to pretraining. It is the right first experiment for most individual developers.

The practical decision looks like this:

  • Use the released weights when your goal is an application, fine-tune, or compact-model comparison.
  • Attempt the uniform recipe when you need control over data, training dynamics, or a genuinely reproducible research result.
  • Pay for the canonical run when curriculum model averaging is part of the question, not simply because the larger number sounds more complete.
  • Rent before buying when you have no proven distributed configuration and no reason to keep the GPUs after the experiment.

Puro-2B lowers the entry fee for open pretraining, but it does not remove the engineering. The best reading of the $4,400 result is not "anyone can train a language model cheaply." It is "a transparent team showed exactly which assumptions make a 2B run approach that price." That is enough to change the conversation. It gives the next researcher a ledger, a codebase, and a stopping point instead of another mysterious model card.

Sources