Long-running agents often pay twice for the same history: they keep loading old key-value (KV) state, then pay again when a summarizer rewrites that history and the server has to prefill the surviving text. A new Tsinghua-led paper proposes a lower-level alternative. AKV lets an agent drop selected KV entries while keeping the conversation transcript intact, then reuse retained cache rather than rebuilding it.

AKV paper figure showing attention paid to tool responses across an agent run

The headline result is a 27.7% average inference-cost reduction across three models and four agent benchmarks, with average task score up 1.7 points against the paper’s standard context-management baseline. That is promising, not a free win for every chatbot. The result is most relevant to self-hosted, multi-turn agents whose tool history grows long and whose serving stack can implement custom cache operations.

The distinction matters. Ordinary prompt or prefix caching reuses a matching prefix. AKV changes which KV states are active for attention, while preserving the full logical message history. Its operations are append, drop, and reposition. Drop removes selected cache entries from later attention without deleting their text. Reposition compacts the positional indices of retained keys, which matters because a short active cache can otherwise still inherit a long sequence position.

That creates a different engineering problem from trimming text. If two requests contain the same messages but applied different drop/reposition histories, their internal cache states are not interchangeable. AKV’s serving layer therefore tracks cache-operation history as well as tokens. It uses an extended radix tree for state reuse, drop-aware eviction, and segment-batch attention to restore missing states in one prefill pass. This is why the paper is about both an agent policy and a serving system, not a prompt trick you can paste into a hosted API call.

When AKV is worth a pilot

A good first candidate is an agent that repeatedly calls tools, accumulates large tool outputs, and revisits a task over many turns. The paper’s rolling-drop policy retains reasoning and recent tool responses while dropping older tool-response cache. On its four benchmark suite, that policy averaged 27.7% lower inference cost and 1.7 points higher benchmark score than standard threshold-triggered summarization. The comparison is also favorable to a lower summarization threshold: AKV reports 21.9% lower average cost than its text-engineering baseline.

Those numbers support a pilot if your own trace data shows that old tool responses dominate the active context. They do not establish a general savings rate for every model, provider, or traffic pattern. The paper’s comparison uses three model families and four agent benchmarks. Its published project page gives operational detail for the serving test: GPT-OSS-120B in BF16 on eight A800-80GB GPUs, split four GPUs for prefill and four for decode. That is a serious serving setup, not a laptop experiment.

Before changing production traffic, measure your own agent on a fixed task set. Compare success rate, input/cache-read cost, prefill work, p95 request latency, and throughput at the same concurrency. Log which message roles and ages are being dropped. Start with older tool responses, not assistant reasoning or user instructions; the paper’s results do not license an indiscriminate “delete old context” rule. A context policy that saves tokens but breaks a rare dependency is a regression, even if average cost falls.

What to measure before adopting it

The available implementation is an early systems project. Its API remains OpenAI-compatible at the outer request level, but it adds AKV-specific cache operations and uses an AKV-extended SGLang server. The repository documents current support for Qwen-family architectures, GPT-OSS, and MiniMax-M2-family models. It also calls out constraints: Drop/Repos requires page size = 1, and the serving system compiles CPU kernels during startup, so a working C++20 compiler and writable build cache are required. The system README lists unsupported serving modes, including pipeline/context/data parallelism, speculative decoding, LoRA, HiCache, external/session caches, and beam search.

That should shape the go/no-go decision. If you depend on unsupported serving features, use the paper as a design reference and wait for compatibility rather than forcing AKV into the stack. If your model and SGLang path match, run a shadow or offline replay first. The public repository provides a demo and a payload-processing integration point, but its own instructions still require the full message history to be sent; do not remove messages from the request because the cache operation is separate from transcript editing.

There is also an important benchmark caveat. The paper’s cost figures are API-equivalent estimates based on active input and generated output lengths, with named provider rates; they are not a universal bill from a live commercial API. The headline serving speedups come from separate workloads and concurrency sweeps, so do not combine them as if one controlled run delivered every metric simultaneously. On the project page, the largest p95 and throughput ratios compare AKV against Standard at concurrency 16. The same page explains that high-concurrency Standard runs suffer cache pressure and re-prefill. That is precisely the regime where this design is meant to help, but it is not evidence that a small agent will be 7.5 times faster.

Where the headline savings stop

AKV does not make cache policy safe by itself. A dropped tool result may contain an identifier or constraint needed much later. The authors retain reasoning and recent tool responses in their main policy, but real applications need task-specific evaluation for stale references, recovery behavior, privacy boundaries, and tool-result dependencies. Keep an untouched transcript or an auditable event log, and make the active-cache policy reversible in testing.

The more fundamental limitation is deployment coupling. Standard hosted model APIs generally do not expose arbitrary KV deletion and RoPE repositioning. AKV requires the modified serving path, compatible model architecture, and agent harness integration. That narrows the immediate audience to teams operating their own inference stack. It also means the 27.7% figure should be treated as a reason to measure, not a promise to put in a budget.

The decision is simple: if your agent is short-lived or mostly calls a hosted API, stick with normal prompt caching and careful context limits. If you operate a long-horizon self-hosted agent, have a compatible model, and can replay real traces, AKV is worth a bounded benchmark pilot. The paper’s strongest contribution is the idea that agent memory can be managed below the transcript layer. The engineering burden is real, but so is the waste caused by repeatedly rewriting and recomputing the same history.

For background on the higher-level alternative, see why long-running agents need editable context, not just more tokens. AKV addresses a separate layer: it leaves that logical history alone and manages the model’s active cache beneath it.

Sources