A prompt cache can look broken even when the provider says it is working. The usual culprit is less dramatic: one timestamp, reordered tool schema, or rewritten message appears before the reusable prefix ends. One small difference, and the request no longer matches the cached path you expected.

Diagram showing a stable prompt prefix before the cache boundary and changing user data after it

That is an API prompt-prefix cache, not the GPU KV-memory problem discussed in our long-agent KV cost analysis. The two optimizations live at different layers. Here, the practical question is whether repeated API requests share enough of the same eligible prefix to receive cached-input pricing, and whether the response proves that happened.

Check eligibility before tuning

Start with the exact provider and model. There is no safe universal cache threshold. OpenAI's current guide says GPT-5.6 and later need at least 1,024 visible input tokens in an eligible prefix; other models have different rules. Hidden system tokens do not count toward that threshold. The threshold is a gate, not a promise of a hit. A 1,024-token prefix that changes between requests is still a miss. A separate OpenAI cookbook experiment ran 2,300 prompts and found its cached 1,024-token requests 7% faster. Treat that as one measured setup, not a latency promise for your model or traffic.

Anthropic exposes a different control surface. Its Messages API can add automatic caching with a top-level cache_control field, or place an explicit breakpoint on the last block that remains identical. Automatic mode is convenient for a growing conversation. Explicit breakpoints are easier to reason about when the request has fixed reference material followed by a user-specific tail. Put a breakpoint on the changing tail and you can keep writing cache entries without reading the one you meant to reuse.

Before editing code, log the fully serialized request shape for two calls that should share a cache. Compare the model, ordered messages, tool definitions, structured-output schema, and any provider cache key or breakpoint. Do not log secrets or private user content; hash sensitive blocks or compare them locally. Look for volatile material near the start: current time, request IDs, per-user instructions, randomly ordered tools, changing retrieval snippets, and conversation history that your application rewrites instead of appending.

A useful layout is simple: put stable instructions, shared reference text, and unchanged tool definitions first. Put the incoming question and request-specific context later. Keep the order and serialization deterministic. If the user-specific material genuinely needs to appear earlier for model behavior, do not contort the prompt just to chase cache hits. Measure the trade-off; output quality and correct tool behavior matter more than a dashboard percentage.

Run a controlled cache-hit test

Use a single model, region, account, and request template. Send one request to create or warm the prefix. Then send a second request with exactly the same reusable part and a different final question. Do not change the model, tool list, schema, or message structure between the pair. If the provider has a documented cache-key control for that model, use a stable key for requests in the same workload, not a fresh random key on every call.

Read the usage object returned by the API. OpenAI's Responses API reports input_tokens_details.cached_tokens; Chat Completions reports prompt_tokens_details.cached_tokens. Anthropic reports cache_read_input_tokens and cache_creation_input_tokens in its usage data. Field names differ, so instrument the response you actually receive rather than assuming your SDK maps them to a common name. Track ordinary input, cached input, cache writes, request latency, and the model version alongside the hit counter.

For a quick check, calculate the share of eligible input served from cache as cached input tokens divided by total input tokens. Keep the provider's definition in mind: OpenAI rounds reported cached tokens to 128-token increments, and the reported value can omit hidden tokens. A zero is useful evidence, but it does not by itself tell you whether the prefix was too short, structurally different, outside its retention window, or routed elsewhere.

Now change one thing at a time. First move a timestamp out of the prefix and place it after the boundary. Repeat the same pair. Then, if needed, test stable tool ordering, appended conversation turns, and the documented cache key or breakpoint behavior. Capture a before/after record for each change. This turns "caching feels flaky" into a traceable diagnosis: which request changed, what the provider read, and whether the cached-token count moved.

Do not treat cached-token count as the savings figure. Cached input is billed at a provider- and model-specific rate; cache writes may have their own price, and output tokens are unaffected by input caching. For a workload estimate, use the current pricing page and your observed mix of cache reads, writes, and misses. A high hit share can still be a bad trade if requests rarely repeat or the workload pays to write large prefixes that expire unused.

Also separate hit rate from latency. A cache read can reduce prefill work, but network time, queueing, generation length, and tool calls still shape end-to-end response time. Measure time to first token and total latency separately. Run the test more than once, keep the prompts representative, and compare the same model and service tier. One warm second request is evidence that reuse can happen, not a service-level guarantee.

A clean deployment checklist is short: record model and request settings; count the visible stable prefix using the provider's tokenization; move volatile fields after the reusable boundary; keep tools and schemas deterministic; test with paired requests; inspect provider usage fields; and estimate cost from current rates. If the cache stays cold after that, inspect the provider's boundary and routing rules before rewriting the prompt again. For OpenAI models before GPT-5.6, the current guide says a stable prompt_cache_key can help route related requests; it does not pin every request to a machine or guarantee a hit. Newer models have different lookup behavior, so do not cargo-cult the older key advice.

Prompt caching rewards boring request construction. Stable prefixes, predictable ordering, and usage-level checks beat clever prompt tricks. If the prompt must change, let it change. Just make the cache boundary an intentional part of the request design, then verify the bill and latency before calling the optimization real.

Sources