A research agent can burn most of its budget rereading the same instructions, tool schemas, and reference material before it gets to the new question. Prompt caching fixes that, but only when the part you cache stays byte-for-byte stable. Put a timestamp, user-specific note, or changing tool result in the wrong place and the discount disappears.

Prompt caching cost and latency results from a 500-session agent evaluation

The useful question is not "Does my provider support prompt caching?" The useful question is "Which exact prefix will still be identical on the next request?" That is the boundary you should cache.

The strongest evidence comes from a recent evaluation of long-horizon agents across OpenAI, Anthropic, and Google. The study ran more than 500 agent sessions with 10,000-token system prompts and measured both API cost and time to first token. Caching cut API cost by 41-80% depending on the model and strategy. The same study found that time to first token improved by 13% to 31% in its best conditions. Those numbers are large enough to change an architecture decision, but they do not mean "turn caching on" is a complete plan.

The prefix audit

Start with the request as the provider sees it, not the clean prompt template in your application code. Draw a line through the rendered input and label every block as stable or dynamic.

Stable material usually includes the system instructions, tool definitions, policy text, few-shot examples, and a shared reference document. Dynamic material includes the current user request, timestamps, tenant-specific data, search results, file contents, and tool output from this particular run.

Place stable content first. Put dynamic content after it. Then put the cache boundary at the end of the stable region.

That sounds obvious. It is also where many agent implementations fail. A team may keep the system prompt unchanged while rebuilding its tool list on every request. A harmless-looking description change, reordered JSON property, or different tool availability changes the prefix. The provider sees a different sequence of tokens and starts over.

OpenAI's current documentation describes this as exact rendered-prefix reuse. For GPT-5.6 and later, a visible prefix needs at least 1,024 tokens before it becomes eligible for caching. Cached reads cost 0.1 times the ordinary input rate, while cache writes cost 1.25 times the ordinary input rate. OpenAI also exposes cached_tokens and cache_write_tokens, which means you can measure the result instead of guessing.

Anthropic gives you a more explicit control surface. Its request can place cache_control on a block, and the cache covers the tools, system content, and messages before that breakpoint. The default lifetime is 5 minutes. A 1-hour lifetime is available when requests are more spread out, but the write cost is higher. Anthropic reports cache_read_input_tokens, cache_creation_input_tokens, and the uncached input count separately. If you only watch the last number, you can badly underestimate the total context you sent.

A practical audit takes five minutes:

  • Log the rendered prompt hash before every model call.
  • Log the model, tool schema version, cache key, cache read tokens, cache write tokens, and uncached tokens.
  • Compare hashes for requests that you expected to share a cache.
  • Record the first block where the hashes diverge.
  • Move that changing block after the cache boundary, or accept that the workload is not cacheable.

For a multi-tenant application, include the tenant or permission scope in the cache key. A shared prefix is useful only when it is safe to share. A cache hit must never cause one customer's policy or private reference material to appear in another customer's request.

When caching pays

The arithmetic is simple enough to do before you add any provider-specific code. Let L be the reusable prefix length, N the number of requests, r the cached-read multiplier, and w the cache-write multiplier. Without caching, the repeated prefix costs roughly N × L in ordinary input-token equivalents. With one write and N - 1 reads, it costs L × (w + (N - 1) × r).

Using OpenAI's current GPT-5.6 multipliers, a 1,024-token prefix costs 1.25 times on the first write and 0.1 times on later reads. Across ten requests, that is 2.15 times the ordinary cost of those 1,024 tokens instead of 10 times. The write premium is recovered quickly when the same prefix is reused.

The threshold still matters. If your stable prefix is only 200 tokens and the provider requires 1,024, padding it with random prose is a bad optimization. It increases every request's context and may add no useful cache hit. The OpenAI guide's break-even example shows why reuse frequency changes the answer: expanding a short prompt to the 1,024-token minimum can pay off after enough requests, but it can lose money when the prefix is rarely reused. Add meaningful stable instructions, examples, or reference material only when the model benefits from them anyway.

For Anthropic, choose the TTL from the traffic pattern. A busy interactive agent that calls the model several times in a few minutes can use the default 5-minute cache. A batch worker that pauses between stages may need the 1-hour option. Do not choose the longer TTL because it sounds safer. The higher write price only makes sense when it prevents repeated writes after the shorter cache expires.

The evaluation's most useful result was not the headline savings. It was the comparison between cache boundaries. Caching only the stable system prompt gave more consistent cost and latency behavior than blindly caching the full conversation. Dynamic tool results often belong to one session and are unlikely to be reused by the next session. Putting them inside the shared prefix can trigger cache writes for text that will never produce another hit.

That is also why you should separate two goals. Cache the system prompt to reduce repeated setup work across sessions. Preserve the current conversation when it helps the same session. Treat tool results as session data unless you have a specific reason to reuse them. The research found cost savings of 79.6% for GPT-5.2 when excluding tool results, 78.5% for Claude Sonnet 4.5 when caching the system prompt, and 41.4% for Gemini 2.5 Pro in its best tested condition. These are experimental results, not a promise for your workload, but they point to the same design: stable instructions first, changing evidence later.

What breaks first

The first failure is usually a low hit rate caused by an unstable prefix. Print the exact serialized tool definitions and compare them across calls. A changing tool description is enough to invalidate everything after it. The same applies to a request ID, current date, rotating experiment flag, or permission text inserted near the top.

The second failure is a cache that works but saves less than expected. Check the provider's token fields, not just a boolean hit flag. A request may read 4,000 cached tokens, write 2,000 new tokens, and send another 8,000 uncached tokens after the breakpoint. The word "cached" does not mean the whole request was cheap.

The third failure is a latency regression caused by caching the wrong material. The long-horizon study found that naive full-context caching could increase latency in some cases because dynamic tool calls and results caused extra writes or poor reuse. If your goal is faster first output, benchmark time to first token separately from total completion time. If your goal is lower spend, calculate input and output costs separately. Those goals can choose different cache boundaries.

The fourth failure is assuming the provider's defaults are portable. OpenAI uses automatic prefix matching on supported models and now offers explicit breakpoints on newer models. Anthropic uses explicit or automatic cache controls with up to four breakpoints and different TTL prices. Minimum token lengths, retention, routing, and usage fields differ. Keep a provider adapter in your code rather than scattering cache assumptions through the agent loop.

The smallest reliable implementation is a stable prompt builder plus a cache report. Version the stable instructions and tool schema. Put the version in the cache key. Append the request, current files, search results, and tool output after the breakpoint. On every response, record the cache read and write counters. After a day of real traffic, calculate the hit rate by workflow, tenant, and model. A single global average will hide the agent that is quietly missing every time. For adjacent infrastructure tradeoffs, see the NestFrontier analysis of MCP hosting costs.

Try this on one workflow before changing your whole stack. Pick an agent with a long system prompt and at least five model calls per run. Freeze the tool schema, move timestamps and run-specific context to the end, enable the provider's cache controls, and compare seven days of input cost and time to first token. If the prefix hash keeps changing, fix that before tuning TTLs. If the hit rate is high but the bill barely moves, inspect the uncached tail and output tokens. The cache may be doing its job while the expensive part of the workload lives somewhere else.

Sources