A power reduction that reaches 90% of its target in about 10 seconds sounds like a grid operator's dream. The new arXiv paper Creating Power Flexibility in Datacenters via Query Routing reports exactly that in a simulation. It also reports as much as 16% power flexibility for one modeled workload. The useful part is the mechanism: send requests that can tolerate a smaller model to that model when the site needs to draw less power, while keeping latency-sensitive traffic on the larger pool.

Diagram of query routing shifting flexible AI inference requests between model pools to adjust datacenter power

That is a more interesting control knob than simply turning down GPU clocks, but the result is narrower than the headline number suggests. The authors simulate a 10,000-H100 datacenter using an Azure chat trace. They do not report a live grid dispatch test, nor do they claim every query can move to a weaker model. Their 16% figure assumes every query in the modeled workload is eligible for either model pool. With half the workload eligible, the reported reduction is 0.6 MW, about 8% of the simulation's mean facility load. That is still useful, and much less magical.

The research is fresh, posted November 3, 2026, and its important contribution is not a new router product. It is an operational framing: estimate how much request traffic is actually movable, then measure whether the destination pool has capacity to absorb it. If either condition fails, routing cannot create power headroom without creating a service problem.

When query routing is worth a pilot

Start with the requests, not the GPUs. Split traffic into three buckets: requests that require the large model, requests that the small model already handles adequately, and requests where either model meets the quality target. The third bucket is the flexible share. The paper's controller changes the destination only for new requests; it does not interrupt responses already in flight.

Now check capacity at both ends. The large-model pool must have enough room for requests that cannot move. The small-model pool needs enough spare throughput for the flexible requests you plan to shift. The paper calculates a feasible interval for the share routed to the small pool. If that interval is empty, the current fleet cannot serve the load under any routing split. No routing policy can compensate for missing capacity.

That capacity check matters because the power curve is not a simple line. A model instance draws idle power even when it has no active requests. Waking another instance has a larger cost than adding work to one that is already busy. Consolidating flexible requests onto fewer small-model instances can therefore reduce total draw, provided concurrency limits and response-time targets remain intact. Sending everything to the smaller model is not the optimization. Finding the feasible allocation is.

The paper's small-scale experiments help explain the boundary. In one four-instance setup, three Llama 3.1 8B instances and one Llama 3.2 1B instance served ShareGPT-style queries. As arrival rate climbed, the feasible routing fraction narrowed: it covered roughly 0.01–1 at 66 requests per second and 0.45–0.58 at 119 requests per second. At higher load, the small pool has less room to accept traffic. The specific rates belong to that setup, not a sizing rule for your hardware.

For the large simulation, the authors report up to 16% reduction in mean facility power, with 90% of the requested reduction reached in roughly 10 seconds. The power-flexibility estimates had 2.8% mean absolute percentage error in the reported evaluation. Read that as a promising model fit, not a guarantee that a production facility will land within 2.8% of its grid target. The 10-second result is a simulated transition between operating points, and the test depends on which prompts are judged flexible, the model mix, serving headroom, and how long existing responses take to finish.

A sensible pilot would replay a week of your own request logs. Label quality-eligible traffic using an evaluation set, profile throughput and board power for each pool under the actual serving configuration, then calculate the feasible routing interval at each arrival rate. Next, replay target power reductions against that interval and check latency, quality, queue depth, and the duration of the response. Start in shadow mode. If the router's prediction and observed power diverge, do not connect it to a utility dispatch signal.

The arithmetic should be redone whenever models, batching, concurrency, hardware, or request lengths change. The paper's model uses calibrated per-instance profiles; token counts alone do not translate into a universal power estimate. Its facility calculation also includes a modeled 300 W per GPU for non-GPU loads and a PUE of 1.2. Those assumptions belong in any comparison with your own meter data.

When another control is safer

If few queries pass the quality test for the smaller model, there is little flexible traffic to move. If the small pool is already near capacity, shifting work there risks queue growth and SLO violations. If your dispatch target exceeds the power range between feasible allocations, routing alone cannot meet it. The paper explicitly notes that enough GPU unavailability eventually breaks capacity feasibility and can cause queues or service denial.

Those cases call for a portfolio rather than a cleverer routing score. Delay-tolerant batch jobs can be rescheduled; GPU power limits can trim draw with different performance tradeoffs; storage can cover a short grid event. Google's own account of demand response describes shifting non-urgent compute and says its ML workload flexibility is still being developed, with reliability and location constraints. That is a useful reality check: grid response is a site-level operational commitment, not just an inference middleware feature.

Use this as a decision screen:

  • Pilot routing when a measured portion of traffic can use either model, the smaller pool has headroom, and quality and latency tests support the substitution.
  • Combine routing with other controls when the eligible share is real but does not cover the requested reduction.
  • Prefer workload shifting, power caps, or storage when the routing interval is narrow, unstable, or empty.
  • Keep the system in shadow mode until measured facility power tracks the requested change and service metrics stay within limits.

The paper is a preprint and its headline fleet is simulated, not an installed 10,000-GPU control system tested with a utility. Still, it gives operators a sharper question than “Can AI be flexible?” Ask how much of today's traffic can change models without hurting users, and whether the spare pool can take it. If you cannot answer both with measurements, the reported 16% is not your number.

Sources