The long-context arms race has spent years pretending that a larger window solves the problem. It does not. Dense attention still makes every token compare itself with every earlier token, while the KV cache grows with the entire conversation. At some point, "one million tokens" stops sounding useful and starts sounding like a GPU invoice.

HiLS-Attention's hierarchical sparse attention design

Tencent Hunyuan's HiLS-Attention takes a more practical route. It keeps the model's local view, then learns which distant chunks deserve attention. The paper reports comparable or better results than full attention at tested context lengths, plus length extrapolation that dense attention does not handle gracefully. A 345M model trained at 8K reached 4M-token evaluation while staying above 90% on needle-in-a-haystack retrieval. That is a 512x jump, not a marketing-friendly context-window setting.

The 64x claim gets most of the attention because it is easier to repeat. The more interesting detail is that the authors released a 7B checkpoint continued-trained from an OLMo3-style backbone, along with code and an SGLang backend. This is an attempt to make sparse attention a model property and a serving property at the same time, rather than another research trick that disappears after the PDF.

The routing problem hiding inside sparse attention

Most chunked attention methods divide the history into blocks and pick the blocks that look relevant. The catch is that relevance is hard to estimate without doing the expensive work you were trying to avoid. Mean-pooling the keys can miss one decisive token. Using the maximum can overreact to a single noisy match. In both cases, the router can throw away the paragraph that contains the answer.

HiLS adds a landmark token to each chunk. Its learned query produces a compact summary key and an entropy term. The query scores those summaries, selects the top K distant chunks, and then runs ordinary token-level attention only inside the chosen chunks plus the local sliding window. The paper describes this as a hierarchical softmax: first allocate attention mass across chunks, then distribute that mass across tokens within each selected chunk.

That second step matters. The routing score remains in the forward attention calculation, so the language-modeling loss can teach the router which chunks help predict the next token. The selector is not trained as a separate retrieval model and then bolted onto the transformer. It gets punished when it routes badly.

The cost is not magically linear. HiLS still scans chunk summaries, and the routing term carries an O(N²/S) component for sequence length N and chunk size S. But the expensive token-to-token work is restricted to a constant number of selected chunks. That is the trade the paper is making: spend cheap computation deciding where to look, then avoid multiplying every token by every token.

What the numbers actually say

The paper evaluates 345M, 1.4B, and 7B models across perplexity, RULER, LongBench, short-context tasks, and needle retrieval. At 345M, HiLS stays close to full attention on perplexity at in-domain lengths. On variable tracking with a 256K training context, the paper reports improvements over full attention of up to 50 percent. Those are the results that make the approach more than a cache-saving exercise. A model that saves memory but loses track of a variable is not useful for long documents.

The small-model extrapolation is even stranger. With 8K training context, the 345M system reaches 4M context and maintains more than 90 percent needle retrieval accuracy. The 1.4B results show the same split between dense and sparse behavior: a full-attention RoPE model collapses at longer tested lengths, while the HiLS variant continues producing usable perplexity and retrieval scores at 32K, 128K, and 512K in the reported table.

For the 7B conversion, the team continued training for 50B tokens. The goal was to inherit the base model's short-context behavior while giving it a sparse long-context path. HiLS reportedly beats the full-attention baseline on LongBench and the YaRN-extended version of the base model. The important qualifier is continued training. You cannot drop the attention module into an untouched checkpoint and expect the router to know what to keep.

The released serving path also has sharp edges. The SGLang backend requires a page size equal to the 64-token chunk size. It currently favors single-sequence prefill, and the README says overlap scheduling must be disabled. Those constraints make sense for a research backend, but they matter to anyone translating the benchmark into an actual API service. A headline about long context is cheap. A stable multi-tenant server is the part that takes engineering.

The catch is still retrieval

HiLS does not make lost information recoverable. If the learned summary ranks the wrong chunk, the token-level attention never sees the answer. That is the same basic failure mode as every retrieval system, only embedded inside the attention layer. The claimed 90 percent retrieval figure is strong, but 10 percent misses across long agent traces is enough to create very strange failures.

The paper's comparisons are also mostly controlled research evaluations. LongBench and needle tests tell us something useful, but they do not settle whether a sparse model can preserve the exact evidence trail needed for legal discovery, codebase navigation, or a long-running agent that revisits its own notes. Those workloads contain repeated references, misleading near-matches, and documents whose useful token is not semantically obvious in isolation.

Still, HiLS gets the architecture question pointed in the right direction. Long context should not mean making every token equally responsible for every other token. A model should spend its expensive attention where its own learned routing says the evidence is. That sounds obvious after the fact, which is usually a sign that the implementation problem was the hard part.

The project is worth watching for a less glamorous reason too. It ships a checkpoint, an evaluation recipe, and an inference backend instead of stopping at a diagram. If the kernels mature and the router survives messier workloads, sparse attention may become the default way open models stretch context. If it does not, the failure will probably tell us that chunk selection is the real bottleneck, not the attention formula.

Sources