Back to articles
Guide

Prompt Caching for Developers: Cut LLM API Costs Without Gutting Quality

Prompt caching reuses stable prefixes so you pay less and wait less on repeated context. Here is how OpenAI and Anthropic caching work, how to structure prompts for hits, and which metrics prove you are actually saving money.

9 min read

Why Caching Beats Blindly Shortening Prompts

Long system prompts, tool schemas, and conversation history are expensive -- but cutting them can wreck quality. Prompt caching keeps the useful context while charging a discounted rate (and often lower latency) when the same prefix appears again.

Both OpenAI and Anthropic support prompt caching on modern models. The APIs differ; the engineering principles do not.

---

How the Major APIs Behave

OpenAI

  • Caching is automatic for supported models once prompts are long enough / eligible
  • Reused prefixes bill at a reduced cached-input rate (vendor docs cite discounts up to ~90–95% on reads depending on model)
  • Monitor `usage.input_tokens_details.cached_tokens` (and write counters on newer models)
  • Keep a stable prefix; optional `prompt_cache_key` helps routing stickiness for multi-tenant apps
  • On newer GPT generations, explicit breakpoint controls and cache lifetime options exist -- read the current Prompt Caching guide before hard-coding lifetimes
  • Anthropic

  • Two modes: automatic top-level `cache_control`, or explicit breakpoints on content blocks
  • Default cache lifetime is short (commonly five minutes) and refreshes on use
  • Cache writes cost a premium (often ~1.25× input for the short TTL); cache reads are heavily discounted (often ~0.1×)
  • Place breakpoints on the last block that stays identical across requests
  • Exact multipliers change by model and date. Treat the ratios above as orientation and verify on the provider pricing page for the model you ship.

    ---

    Structure Prompts for Hits

    Caching matches prefixes. Tiny early changes invalidate reuse.

    Do:

    1. Put durable instructions, tool definitions, and large reference material first

    2. Append conversation turns instead of rewriting history

    3. Keep tool schemas stable across requests in a session

    4. Move timestamps, user IDs, and per-request variables to the end (or into later messages)

    5. For agents, consider a breakpoint after the static system block and another after prior history so the latest user turn stays outside the cached region when needed

    Don't:

  • Inject the current clock into the first system line on every call
  • Shuffle tool order between turns
  • Compact/summarize mid-session unless you accept a cache reset
  • Assume a write is free -- a write premium that never gets a read is a loss
  • A useful mental model from OpenAI's guidance: preserving growing conversation history often saves more than caching only the initial instructions, because later turns share a longer prefix.

    ---

    Prove Savings With Metrics

    Instrument before celebrating:

  • Cache hit rate = cached tokens / total input tokens (slice by route, customer, or model)
  • Write vs read token counts
  • Realized $/1K requests before and after restructuring
  • Time-to-first-token for agent loops
  • If hit rate is low, your prefix is unstable. Fix ordering before buying a bigger model discount tier.

    ---

    Patterns That Pay Off

    RAG assistants

    Cache the system policy + tool schemas. Keep retrieved chunks and the latest question outside or after the breakpoint so documents can change without thrashing the entire cache.

    Coding / tool agents

    Static developer instructions and tool list stay fixed. Append tool results and user messages. Avoid regenerating the full system prompt each step.

    Multi-tenant SaaS

    Use cache keys / separate accounting so Customer A's giant policy docs do not collide with Customer B's routing. Still keep each tenant's own prefix stable.

    Eval suites

    Reuse the same system prompt across hundreds of cases so CI judge and candidate calls benefit too -- but pin versions so evals remain comparable.

    ---

    Common Failure Modes

  • "We enabled caching" but every request starts with a unique UUID in the system prompt
  • Summarizing history every N turns and wondering why hits vanished
  • Adding a new tool mid-conversation and invalidating the whole prefix
  • Looking only at output tokens while input tokens dominate the bill
  • ---

    Who Should Prioritize This

    Perfect for: apps with large stable instructions, multi-turn agents, or repeated tool schemas.

    Lower urgency: one-shot short prompts under the provider's caching threshold.

    ---

    Bottom Line

    Prompt caching is one of the highest-leverage cost wins available to API builders in 2026. Structure prefixes deliberately, measure hit rate, and keep dynamic content at the end. You keep quality -- and stop paying full price to re-read the same instructions on every turn.

    Found this helpful?Share this article with your network to help others discover useful AI insights.

    Share your thoughts on X or Facebook