Prompt caching is the rare optimisation that reduces both latency and cost without touching output quality — when it hits. The gap between teams who benefit and teams who see nothing is almost entirely about prompt structure, not volume.
The prefix rule
Provider-side caching works on a prefix: the model reuses computation for a leading span of tokens that is byte-identical to a previous request. Everything up to the first difference can be reused; everything after it cannot.
That single rule explains most caching outcomes. If your prompt begins with a long, stable system message and a fixed tool schema, and the user's variable content comes last, you get hits. If you interpolate a timestamp, a request id, a shuffled list of retrieved documents, or the user's name into the top of the prompt, you have changed byte one and there is nothing to reuse.
The practical instruction is therefore simple, and it is a prompt-engineering decision rather than an infrastructure one: stable content first, variable content last. Teams that reorganise their prompts around this typically see a large share of their input tokens become cached; teams that do not see essentially zero.
What caching does not do
Three misunderstandings are worth separating out.
It does not cache the answer. Provider prompt caching reuses the encoded prefix, not the completion. The model still generates fresh output, and output tokens are still billed at the full rate. Caching helps most on prompt-heavy, completion-light workloads: long documents, big tool schemas, extensive few-shot examples.
It is not free. Cache writes are often billed at a premium over ordinary input tokens, and entries expire — typically in minutes unless refreshed. A workload with low request frequency against a given prefix can pay to populate a cache it never reads. Caching a huge prefix that is used twice an hour can cost more than not caching at all.
It is not the same as your own response cache. Storing "this exact question got this exact answer" in your own layer is a different mechanism with different risks — chiefly that answers become stale, and that two users with the same question but different permissions can receive each other's cached content. If you build one, key it by everything that affects the answer, including the identity of the caller.
Ways caching quietly changes behaviour
Two failure modes are worth naming, because they are hard to debug after the fact.
First, metrics drift. Cached input tokens are usually reported in a separate field from ordinary input tokens. If your accounting sums only the ordinary field, cache adoption looks like a mysterious drop in usage; if it sums both at the same price, your cost model overstates spend. Track them separately — see the note on token fields in cost attribution.
Second, prompt churn. Once caching is load-bearing, an innocuous edit to the system prompt invalidates every cached prefix at once. Latency and cost jump, and nobody connects it to the one-word copy change deployed that morning. Treat the stable prefix as a versioned artefact: change it deliberately, and expect a warm-up period after you do.
Deciding whether it is worth it
A quick screen, in order:
- Is your input-to-output token ratio high? If completions dominate, caching will not move your bill much.
- Do many requests share a long, identical prefix? If every request is bespoke, there is nothing to reuse.
- Is the request rate against that prefix high enough that entries stay warm?
Three yeses means restructure prompts and measure. Any no means spend the effort on the other levers — model selection and output length usually pay more.