Why Caching Beats Blindly Shortening Prompts
Long system prompts, tool schemas, and conversation history are expensive -- but cutting them can wreck quality. Prompt caching keeps the useful context while charging a discounted rate (and often lower latency) when the same prefix appears again.
Both OpenAI and Anthropic support prompt caching on modern models. The APIs differ; the engineering principles do not.
---
How the Major APIs Behave
OpenAI
Anthropic
Exact multipliers change by model and date. Treat the ratios above as orientation and verify on the provider pricing page for the model you ship.
---
Structure Prompts for Hits
Caching matches prefixes. Tiny early changes invalidate reuse.
Do:
1. Put durable instructions, tool definitions, and large reference material first
2. Append conversation turns instead of rewriting history
3. Keep tool schemas stable across requests in a session
4. Move timestamps, user IDs, and per-request variables to the end (or into later messages)
5. For agents, consider a breakpoint after the static system block and another after prior history so the latest user turn stays outside the cached region when needed
Don't:
A useful mental model from OpenAI's guidance: preserving growing conversation history often saves more than caching only the initial instructions, because later turns share a longer prefix.
---
Prove Savings With Metrics
Instrument before celebrating:
If hit rate is low, your prefix is unstable. Fix ordering before buying a bigger model discount tier.
---
Patterns That Pay Off
RAG assistants
Cache the system policy + tool schemas. Keep retrieved chunks and the latest question outside or after the breakpoint so documents can change without thrashing the entire cache.
Coding / tool agents
Static developer instructions and tool list stay fixed. Append tool results and user messages. Avoid regenerating the full system prompt each step.
Multi-tenant SaaS
Use cache keys / separate accounting so Customer A's giant policy docs do not collide with Customer B's routing. Still keep each tenant's own prefix stable.
Eval suites
Reuse the same system prompt across hundreds of cases so CI judge and candidate calls benefit too -- but pin versions so evals remain comparable.
---
Common Failure Modes
---
Who Should Prioritize This
Perfect for: apps with large stable instructions, multi-turn agents, or repeated tool schemas.
Lower urgency: one-shot short prompts under the provider's caching threshold.
---
Bottom Line
Prompt caching is one of the highest-leverage cost wins available to API builders in 2026. Structure prefixes deliberately, measure hit rate, and keep dynamic content at the end. You keep quality -- and stop paying full price to re-read the same instructions on every turn.