A lot of AI software has the same awkward shape: a tiny decision gets shipped across the network to a huge model, then the same decision gets made again five thousand times. “Does this email need a signature?” “Should this question return a link or a paragraph?” “Turn this sentence into the style our team uses.” These jobs are fuzzy enough to resist a pile of if statements, but small enough that paying a remote model on every request feels wasteful.

A new University of Waterloo and Harvard paper called Compile by Training proposes a different split. Use large models while building the function, then run the finished function locally. The result is a small PAW program, not a general chatbot. It is closer to compiling a narrow behavior into a reusable software artifact.

PAW helper workflow from the Compile by Training paper

The distinction matters because the paper is easy to misread as “a 0.6B model replaces a frontier model.” That is not what happened. The large teachers still do the expensive thinking during compilation. The useful trick is that they do not have to do it again for every future input.

The decision boundary

Compile by Training starts with a natural-language specification. You describe the behavior, such as extracting only arXiv PDF identifiers, routing a website question to the right content pack, or translating plain English into a particular writing style. Teacher models synthesize input and output examples from that description. Those examples then train a LoRA adapter for a shared, quantized Qwen3-0.6B interpreter.

The finished artifact packages the adapter, the runtime scaffold, the original specification, and interpreter metadata. Future inputs run through the local interpreter and adapter. The paper says those programs can be stored, versioned, cached, and composed like ordinary software. That is the part worth paying attention to. The model is being used as a build tool, not treated as a dependency in every request path.

The paper reports 83.6% semantic accuracy on FuzzyBench-Hard. This is a deliberately difficult slice: the earlier fast PAW compiler produced no exact matches on these tasks. Compile by Training improved mean LLM Exact Match from 0.224 to 0.836, an absolute gain of 0.612. That sounds like a strong win until you read the comparison carefully. It is a hard subset, not a broad claim about all language tasks, and the score comes from a GPT-5.5 judge rather than a deterministic test for every output.

The build cost is the other half of the result. A representative compile took 50.9 seconds on a B300, compared with 3.5 seconds for the fast amortized compiler. The paper also measured 68.2 seconds on an H200 and 99.2 seconds on an RTX GPU. After that build, the compiled function runs locally without the teacher models. If a task repeats thousands of times, a minute of compilation can be a sensible trade. If it runs twelve times a day, the remote call may still be the simpler and cheaper choice.

Here is the practical filter I would use.

  • Use ordinary code when the rule is exact. Dates, permissions, schemas, arithmetic, database queries, and anything with a hard compliance requirement should not be delegated to a fuzzy neural function.
  • Use a remote model when the task changes often, needs broad context, or has too few repetitions to amortize compilation. A build artifact is overhead if the specification is still moving every afternoon.
  • Use a compiled PAW function when the behavior is narrow, repeated, tolerant of more than one valid answer, and expensive or inconvenient to send to a provider on every call.
  • Keep deterministic checks around the compiled function when the output controls an external action. A local adapter can classify an email, but ordinary code should still enforce who can send it, where it goes, and what data it may contain.

That last point is the difference between a useful deployment pattern and a magic-model story. Compile by Training handles fuzzy judgment. It does not make fuzzy judgment safe by itself.

The examples in the paper show why composition is more interesting than a standalone demo. The Paw-helper system uses 30 compiled programs, with 28 in live routing, across four websites. Ordinary code handles retrieval, caching, and branch control. The compiled functions handle questions such as whether a student wants a link or a written answer, or which retrieved answer should be returned. That division lets developers keep exact operations in normal software while moving only the language-shaped decisions into small local programs.

There is a similar boundary in the Avatar Director demo. A user says “jump twice, then dance.” The compiled program turns that instruction into a small action DSL. The browser, not the model, validates and executes the actions. On 44 hand-authored validation instructions, the system produced the expected action structure in 43 cases. That is a much more credible design than asking a model to emit arbitrary JavaScript and hoping the browser behaves.

Where the design breaks

The hosted build step is a real constraint. The paper says the specification is sent to the PAW service and teacher APIs during compilation, while later local execution does not send future inputs to PAW or the teachers. That means “local” describes runtime behavior, not the full lifecycle. If your specification contains customer data, private policy text, or examples that reveal internal workflows, the build boundary needs the same review as any other external model call.

Synthetic supervision can also copy teacher mistakes. The authors explicitly say applications that need guaranteed correctness should validate outputs or retain deterministic control paths. I would go further: treat the compiled artifact like generated code. Version it, test it against a fixed corpus, record which teacher configuration built it, and reject it when a new version changes behavior outside the allowed range.

The benchmark has another caveat. FuzzyBench-Hard was selected because the fast compiler failed to produce exact matches. That makes it useful for measuring improvement, but it is not a neutral sample of every workload. A score of 83.6% semantic accuracy does not mean 83.6% of your production messages will be correct. It means this particular compiler, task slice, judge, and supervision recipe produced that result.

There is also a privacy asymmetry between compile time and run time. A local runtime can reduce ongoing exposure and latency, but it does not erase the data sent during build. Teams that need fully offline training would need local teachers or an existing labeled dataset, which is a different project than using the public compiler service.

So the clean deployment pattern is not “replace the API.” It is: identify one stable fuzzy function, build it with representative synthetic examples, test it against real edge cases, keep exact operations outside the adapter, and measure whether the repeated calls justify the build. Start with routing, extraction, normalization, or style conversion. Do not start with a function that can approve payments, change permissions, or make an irreversible decision without a second control layer.

Compile by Training is interesting because it gives neural behavior a software lifecycle. The useful unit is no longer a prompt that gets pasted into an API call. It is a small artifact with a specification, an adapter, a runtime, tests, and a version history. That will not eliminate remote models. It does give developers a better answer for the narrow class of jobs that are too fuzzy for rules and too repetitive for a frontier call.

Sources