Micro-Agent routing architecture

Sakana Fugu made headlines last week for being a "model" that is actually five models in a trench coat. Reverse engineering showed it uses Gemini workers behind the scenes, with token consumption running 5x to 12x higher than what the user sees. The market treated this like a novel trick. It is not. The vLLM Semantic Router project has been building the same idea as open-source infrastructure, and yesterday they dropped a blog post showing their version actually beats Fugu Ultra on coding benchmarks.

The core insight is simple but the implications are not. A single API call does not have to mean a single model. The router sitting in front of your models can decide to fan out to multiple workers, collect votes, verify disagreement, and synthesize one clean response. The caller never knows. The response looks identical to a normal OpenAI-compatible completion. Under the hood, it was a team.

How the Looper actually works

The execution runtime is called the Looper. It manages five distinct collaboration patterns, each designed for a different kind of problem. Confidence mode starts with a cheap model and escalates only when confidence scores drop below a threshold. Think of it as a bouncer who only calls the manager when something looks wrong. Ratings mode launches multiple candidates in parallel and aggregates results, like a panel of judges. ReMoM (Repeated Mixture-of-Model Reasoning) runs multiple reasoning attempts, waits for a quorum, and uses a synthesis model to merge evidence. Fusion takes independent responses and feeds them to a judge that resolves contradictions. Workflows execute a planner-worker-verifier loop with strict constraints on steps and parallelism.

The router picks which recipe to use based on signals it extracts from the request before generation begins. Difficulty, risk, latency tolerance, cost budget. A simple "what is the capital of France" goes to the cheap local model. A complex legal analysis triggers Fusion with expensive backend models. A coding task with hidden test cases gets the code-specific workflow recipe. The user sends one request. The router decides how many models need to be involved and which recipe fits.

This is not theoretical. The benchmarks are concrete. On LiveCodeBench, VSR scored 92.6, beating Fugu Ultra at 92.0 and GPT-5.5 at 90.7. On GPQA-Diamond, VSR hit 96.0 against Fugu Ultra's 95.5 and Gemini 3.1 Pro's 94.3. On Humanity's Last Exam, it matched Fugu Ultra at 50.0. These are not marginal wins. The coding benchmark in particular is notable because LiveCodeBench evaluates on problems released each month, which makes contamination unlikely.

The Fugu comparison is the real story

Sakana Fugu Ultra and vLLM Semantic Router are solving the same problem from opposite directions. Fugu is a commercial product that presents a multi-agent system as a single model. You call sakana/fugu-ultra, get one response, and never see the orchestration tokens. Requesty's reverse engineering found that every query carries a fixed 1,260-token system prompt overhead, and complex queries consume 8x to 12x the visible output in hidden orchestration tokens. The latency for complex tasks hits 30 to 160 seconds.

VSR takes the open-source path. You run it yourself. You see every routing decision. The dashboard shows which signals triggered which recipe, which models were called, and how the final answer was synthesized. The cost is transparent because you are paying for the actual model calls, not a bundled orchestration fee. For teams already running vLLM, adding the router is a configuration change, not a vendor commitment.

The benchmark comparison between VSR and Fugu Ultra is interesting but the real difference is architectural. Fugu Ultra is a black box where the orchestration is hidden. VSR is a glass box where every decision is inspectable. For production systems where you need to debug why a particular answer was given, that transparency matters more than a half-point benchmark difference.

What this changes for inference infrastructure

The vLLM Semantic Router project has been evolving since September 2025. The initial version was a simple classifier that routed queries into 14 MMLU categories. By January 2026, version 0.1 Iris introduced the Signal-Decision Driven Plugin Chain Architecture. By March 2026, version 0.2 Athena added ClawOS for orchestrating multiple model teams. The micro-agent post yesterday represents the latest evolution: the router is no longer just selecting models, it is constructing capabilities.

This is a meaningful shift in how inference infrastructure works. The traditional model is: user sends request, system picks one model, model generates response. The new model is: user sends request, router classifies difficulty and risk, router selects a collaboration recipe, multiple models work together under infrastructure-level constraints, router synthesizes and returns one response. The serving stack is no longer passive. It is an active participant that manages retries, timeouts, and synthesis.

The cost implications are significant. Instead of routing everything to GPT-5.5 at premium prices, the router can use a cheap local model for 70% of requests and escalate to frontier models only when the recipe calls for it. The "Hybrid" variant of VSR mixes open and closed models, using stronger models specifically for high-risk tasks like judging or repair. This is not just cost optimization. It is a different architecture for how AI systems handle workloads.

The practical question is whether this actually works in production, not just on benchmarks. The Red Hat deployment guide shows VSR running on AMD Developer Cloud with a Qwen 3.5 122B backend, routing between local and cloud models based on privacy and complexity signals. The system includes a Playground for testing routing tiers and a Brain Topology visualization showing how decisions connect. For teams deploying on-premise, the privacy-first governance model is compelling: sensitive data stays local, only non-sensitive tasks escalate to cloud APIs.

What strikes me about this moment is that the most interesting AI infrastructure work is happening in the layer between you and the model, not in the model itself. Everyone is watching for GPT-6 or Claude 5 or whatever comes next. Meanwhile, the router sitting in front of your models just learned how to turn a single API call into a disciplined team. The next frontier model might not be a model at all. It might be the system that decides which models to use, how many, and when to stop.

Sources