Real-Time TTS Still Breaks at Six Requests Per Second

Nari Labs published a result that looks like a model win but is really a serving-system warning. Its Qwen3-TTS 1.7B CustomVoice stack holds sub-50 ms p95 time-to-first-audio through 10 requests per second on one NVIDIA H100 SXM. That is fast enough for a spoken assistant. The awkward part is what happens to the alternatives: after tuning, VoxServe is 49.3 ms at 1 RPS and 363.2 ms at 6 RPS. A demo can feel instant while a modest queue makes it sound broken.

This matters because "real time" is often reduced to one latency number. Speech has a stricter contract. The first audible sample must arrive quickly, every later chunk must arrive before the previous chunk finishes, and the system must keep doing that as concurrent requests rise. A server that wins TTFA at one request per second but underruns at six is not a conversational voice server. It is a nice benchmark screenshot.

The six request per second cliff

Nari's test is more useful than a casual throughput claim because it measures the part users hear. The team ran each engine for five minutes under Poisson open-loop traffic, streamed audio while the request was active, detected audible TTFA rather than merely the first network byte, reconstructed playback from PCM, and checked completed speech with Deepgram STT. That is closer to a production workload than sending one request, timing the response, and calling it streaming.

The upstream defaults show why configuration matters. At 1 RPS, vLLM-Omni reached 277.883 ms p95 audible TTFA and reported underruns on every request in Nari's table. SGLang-Omni reached 1,140.69 ms. VoxServe reached 315.064 ms, with no underruns, while M* reached 1,159.956 ms. These are not small differences that a nicer client can hide.

Nari then tuned the engines for compatibility, leading-silence removal, frame accumulation, continuity, and capacity. The tuned table is more revealing:

The lesson is not that VoxServe is bad. It is that a single-load result tells you almost nothing about the queueing behavior you will hear. VoxServe is the quickest option in this table at 1 RPS, then loses its conversational edge by 6 RPS. Nari's implementation keeps sub-50 ms p95 TTFA through 10 RPS and below 100 ms at 20 RPS, according to the post. That is a different operating envelope.

There is a second trap in Nari's headline number. TTFA is not total response latency. Once playback starts, the scheduler has a deadline: produce the next audio chunk before the current chunk ends. Sending a chunk earlier than that does not make the user hear it earlier. Sending it late creates an underrun, which is the audible click, gap, or chopped word that makes a voice assistant feel defective.

Nari trims leading silence from the first PCM output with short RMS windows. The change improves TTFA by about 80 ms, but it does not make inference faster. That distinction is worth copying into your own test harness. Measure network first-byte time, audible TTFA, leading silence, underruns, and real-time factor separately. Otherwise you may optimize a number that the listener never experiences.

Why the scheduler beats a faster-looking engine

Qwen3-TTS is a 1.7B model with three moving parts: a Talker, a Code Predictor, and a causal Codec. The Code Predictor generates 15 remaining codebook tokens for each audio frame, while the Codec turns those tokens into waveform samples. Those stages do not have identical compute or urgency. Treating the whole request as one indivisible job is convenient, but it lets a long stage block a codec job that is about to run out of playback buffer.

Nari exposes all three modules as independently schedulable tasks on one scheduling surface. The scheduler can prioritize a request that has not produced its first audio, then give an established stream attention only when its playback deadline is close. It can fill the rest of the batch with compatible work instead of running every urgent request alone and destroying GPU utilization.

That policy explains why the result is more than a faster model wrapper. The system uses small initial chunks to start speech quickly, then larger chunks later to improve batching. It preallocates the Code Predictor's KV cache, captures its fixed 15-step frame loop as a CUDA graph, and uses a Triton attention kernel for its bounded context. The Codec keeps transformer and convolutional state between chunks, so later decoding does not replay the full audio history.

These are practical changes, not magic settings. They also explain why copying a model name into a generic inference server will not reproduce the number. The first chunk and sustained playback want different policies. A useful load test should vary initial chunk size, later chunk size, batch limits, scheduler priority, and the point at which the codec state cache takes over. It should record the worst tail, not just the average.

Nari's implementation and benchmark repositories are public, which gives a team a reasonable starting point. The published result still belongs to Nari's hardware, software revisions, request lengths, and traffic model. The post says the H100 cost estimate uses $4.29 per hour and works out to about $2 per 1M characters at full utilization. It explicitly excludes networking, idle capacity, and operational overhead. Treat that as a lower-bound compute estimate, not a cloud invoice.

A deployment decision that survives load

Use the Nari implementation first if your target is interactive speech and you can provision an H100-class GPU. The reported envelope, sub-50 ms p95 TTFA through 10 RPS and roughly 630 characters per second at 10 RPS, is the only result here that clearly leaves room for a real conversational queue. Start with the public code and reproduce the benchmark with your own utterance lengths before committing to a second GPU.

Tune an existing engine first if your traffic is light, your team already operates vLLM-Omni, or your latency target is closer to 100 ms than 50 ms. The tuned vLLM-Omni result, 56.815 ms at 1 RPS and 93.451 ms at 6 RPS, is not far away. The operational cost of adopting a separate scheduler may exceed the benefit when the workload is small. Do not choose from the 1 RPS row alone. Run at the concurrency you actually expect, plus a burst test.

Do not deploy any of these results as a production promise until four checks pass:

• p95 and p99 audible TTFA stay inside the product target at expected RPS.
• Underruns remain at zero during sustained streams, including the first chunk transition.
• Speech quality stays intelligible after chunking, caching, and input streaming changes.
• The cost model includes idle GPU time, network transfer, logging, retries, and peak capacity.

A simple acceptance test is five minutes of open-loop traffic with a request mix taken from your product logs. Record the first audible sample, every playback gap, completed characters, GPU memory, and queue depth. Repeat it after changing chunk controls. If your result looks excellent at 1 RPS and collapses at 6, you have found the decision, not a failure of the test.

The interesting part of Nari's result is therefore not the phrase "sub-50 ms." It is the boundary around that phrase. Real-time TTS is a deadline scheduler attached to a neural codec. Once the queue grows, the scheduler matters more than the model card. That is where teams should spend their next engineering day.

Sources

• Nari Labs Qwen3-TTS serving report: latency, throughput, cost assumptions, methodology, and scheduler design
• Nari Qwen3-TTS implementation: open serving implementation referenced by the report
• Nari serving benchmarks: engine comparisons and tuned p95 TTFA tables
• Qwen3-TTS 1.7B CustomVoice model card: model identity and licensing context
• Lambda H100 pricing: hourly GPU price used in Nari's compute estimate