Every HTTP client library has a retry knob, and for most APIs the default (three attempts with exponential backoff) is fine, because most API calls are cheap and idempotent. LLM calls are neither. A retried request re-sends the full prompt at full input price, re-generates output you may already have paid for once, and lands on a provider at the exact moment it is least able to serve it. The right mental model is not "how many times should I retry" but "how much am I willing to spend on requests that have already failed once": a budget, not a count.
Why retry counts go wrong for model traffic
Three properties of LLM requests break the standard advice:
- Requests are expensive and slow. A retried call is not a few milliseconds and a fraction of a cent; it can be thirty seconds and a meaningful fraction of the original cost. Retrying a large-context request three times quadruples its worst-case bill.
- Failures cluster. Rate limits and overload errors do not arrive uniformly; they arrive in bursts, provider-wide. When every client retries three times into the same burst, offered load triples at the worst moment. That is retry amplification, and it is how a provider's bad five minutes becomes your bad hour.
- Streams fail halfway. A streamed response that dies after 800 tokens already cost you those 800 output tokens on most providers. Retrying from scratch pays for them again. A retry policy that ignores partial output undercounts its own cost by the exact amount that hurts.
Sort errors before you spend on them
A budget is only spent on failures that a retry can actually fix. That sorting is the most valuable part of the whole exercise:
- Retry, with backoff: 429 with a
retry-afterhint (honour the hint, not your own schedule), 500/502/503, connection resets, and timeouts where nothing was received. These are transient by definition. - Do not retry: 400 (the request is malformed (it will be malformed again), 401/403 (credentials will not heal), 404 on a model id (it is gone; see deprecations), and content-policy refusals (deterministic on the same input) a retry is a second bill for the same no).
- Retry somewhere else: persistent 429s and overload responses on one provider while your error budget drains are not a retry problem, they are a routing problem. The correct "retry" is the same request against a different provider or model — failover — and it should draw from the same budget, because from the caller's point of view it is the same spend on the same failed intent.
What a budget looks like in practice
Two limits, one global and one local:
- Per-request: at most one or two extra attempts, ever, and a ceiling on total wall-clock time. The user has stopped waiting; there is no request so important that the fourth attempt rescues it.
- Global: retries as a percentage of traffic — if more than a few percent of requests system-wide are retries, stop retrying and start failing fast or failing over. This is the circuit-breaker view: the budget protects you from your own policy during an incident, which is precisely when the per-request view looks locally reasonable and is globally ruinous.
The global limit is the one almost nobody implements, because it needs a place where all LLM traffic is visible at once. Per-service retry configuration cannot see that the fleet collectively tripled its offered load; a shared choke point can. That is the operational argument for routing model traffic through one layer — whether that is a gateway or something you built — and giving that layer, not each client, the retry policy.
The accounting question
However you implement it, make retries visible in cost attribution. A request that succeeded on attempt three should carry the cost of all three attempts, including the partial output of any stream that died: otherwise your per-feature cost numbers silently exclude exactly the traffic that is most expensive per successful answer. When an incident review asks "what did that outage cost us", the difference between billed tokens and tokens-per-successful-request is the answer, and you can only compute it if the ledger kept attempts distinct from successes.
None of this is exotic to build, but all of it has to live in one place to work: the timeout chain, the error sorting, the failover targets and the budget are one policy seen from four angles. Splitting them across client libraries is how each piece ends up locally correct and the system ends up retrying a content refusal four times during a rate-limit storm — at full price.