Rate limits are the most predictable failure in LLM infrastructure and still the one that takes most products down first. The reason is that they do not behave like the rate limits engineers are used to.
You are usually limited on three axes at once
Most providers enforce some combination of:
- Requests per minute (RPM) — the familiar one.
- Tokens per minute (TPM) — where it gets interesting, because your token consumption depends on user input length, which you do not control.
- Concurrency — how many requests may be in flight simultaneously, which interacts badly with long generations.
A workload can sit comfortably under its RPM ceiling and still get throttled all day because a handful of long-context requests exhausted TPM. Teams that monitor only request counts see 429s that appear to come from nowhere.
Worse, TPM accounting usually includes the tokens you are about to generate, estimated from max_tokens. Setting max_tokens to a comfortable 4096 "just in case" on every request can reserve far more budget than you use, and throttle you well below your real consumption. Sizing max_tokens to the actual task is one of the cheapest throughput wins available.
Bursts, not averages, cause 429s
Limits are typically enforced over short windows against a token bucket. An average of 60% utilisation over a minute tells you nothing if your traffic arrives in three-second spikes — a cron job, a batch import, a retry storm from your own client. The bucket empties, and everything queued behind it fails.
The fix is a shaper on your side: a queue with a controlled drain rate so bursts are absorbed rather than forwarded. It costs a little latency at peak and removes a whole class of errors. Note that the queue only helps if it has a bound and a policy for what happens when it is full — an unbounded queue converts a rate-limit problem into a memory and latency problem.
Read the headers before you guess
Most providers return remaining-quota and reset headers alongside 429s, and many return them on successful responses too. That is a live signal of how close you are to the ceiling, available before anything breaks. Two things worth doing with it:
- Emit remaining quota as a metric, per key and per model. Sudden drops predict incidents.
- Honour
Retry-Afterwhen present. Backing off for the interval the provider asked for is strictly better than a backoff you invented.
Backoff rules that hold up
Exponential backoff with full jitter — randomising the entire wait window, not adding a small random offset — is the pattern to use. Without jitter, all the clients that failed at the same moment retry at the same moment, and the synchronised spike re-triggers the limit.
Pair it with a retry budget: retries may consume no more than some fraction of your request volume in a window, say 10%. When the budget is exhausted, fail fast. Nothing else prevents a partial rate-limit event from amplifying into a full outage, and almost nobody implements it until after their first one.
Not all requests deserve the same treatment
A user waiting on a chat response and a nightly batch summarisation should not compete for the same quota on equal terms. Once you have a queue, class-based scheduling is nearly free: interactive traffic gets priority and a short timeout, batch traffic gets the leftovers and a long one. Under pressure, the batch job slows down and the product stays responsive — which is the outcome you would have chosen manually anyway.
Spreading load across providers
Rate limits are per provider, per account. Two accounts with the same provider double your ceiling; a second provider does the same and adds independence from the first one's outages. This is where a gateway earns its place: the routing, the per-key budgets and the shaping all belong in one component rather than in every service that calls a model. It is also where teams discover that failover and rate limiting are the same problem viewed from two angles — both are about deciding where a request should go when the obvious target is not available.
If you are building this yourself, put the queue, the budget and the header telemetry in one place from the start. Retrofitting them into six services after an incident is the expensive path — it is the path most teams take.