Every HTTP client ships with a default timeout, and every piece of advice about it assumes a request that either answers in under a second or has failed. Model traffic breaks that assumption: a legitimate response can take a minute, a failed one can hang for the same minute, and the two are indistinguishable until one of them ends.
Getting this wrong produces the most confusing class of incident there is: the one where your dashboard says the provider is fine, the provider's status page says the provider is fine, and your users are staring at spinners.
Four timeouts, not one
"The timeout" is almost always four separate things, and conflating them is the root of most of the trouble.
- Connect timeout: how long to wait for a TCP and TLS handshake. This should be short, a couple of seconds at most. A slow handshake means a network or DNS problem, and waiting longer never fixes it.
- Time to first byte: how long to wait for the response to start. On a streaming request this is the one that matters, and it is the only one that distinguishes "the model is thinking" from "nothing is coming".
- Inter-token (or read) timeout: once a stream has started, how long a gap between chunks is acceptable before you give up. Almost nobody sets this, and it is what catches a stream that dies mid-flight without closing.
- Total request timeout: the ceiling on the whole thing. This is the one everybody sets and the one that should usually be the loosest.
Most client libraries expose a single number and apply it as the total. That single number is then set low enough to catch hangs, which means it also kills legitimate long generations, which shows up as sporadic failures on exactly your most valuable requests: the long, complex ones.
Streaming changes which number matters
On a non-streamed request you cannot distinguish slow from stuck, so the total timeout is all you have and it has to accommodate your longest legitimate response.
On a streamed request you can. First byte tells you the request was accepted and generation started; the inter-token gap tells you it is still alive. That means you can set a tight first-byte timeout, a few seconds, and a tight inter-token gap, while allowing the total to run long. A response that takes ninety seconds but produces tokens steadily is healthy. A response that produces nothing for ten seconds is not, and you no longer have to wait ninety seconds to find out.
This is the strongest practical argument for streaming even when the interface does not show tokens as they arrive: you get a failure signal you cannot otherwise have.
Stacked timeouts are where outages come from
A request typically crosses several hops, each with its own timeout: browser, your API, your gateway, the provider. The rule that keeps this sane is that timeouts must decrease as you move outward from the user: the outermost timeout is the longest, each inner one shorter.
When they are ordered the other way, the outer layer gives up while the inner one is still working. The inner request completes, having consumed the tokens and the money, and delivers its response to a caller that stopped listening. You pay for output nobody received, your error rate reports failures the provider never saw, and retries pile a second request on top of a first that is still running.
Two rules follow:
- Write the numbers down as a chain, not as four independent settings. If the browser waits 60s, your API should wait less, and whatever it calls should wait less again. If you cannot state the chain, you do not have one.
- Budget the retries inside the ceiling. A 30-second timeout with two retries is a 90-second worst case at the layer above, which needs to know that. This is the same reasoning as a retry budget, applied to time rather than to volume.
What a timeout should do
Not "retry immediately with identical parameters". A timeout means something is slow, and the most common reason is load, so an immediate identical retry adds load to a system that is already struggling. That is how a slow provider becomes a down one.
Better, in order of preference:
- Retry elsewhere. If another provider serves the same model or an acceptable alternative, that is a different failure domain and worth trying first.
- Retry smaller. Shorter context, lower max tokens, a faster model. A degraded answer beats a spinner.
- Fail honestly and fast. "This is taking longer than usual — try again" returned in five seconds preserves more trust than a ninety-second hang that ends in a generic error.
What is worth avoiding in all three cases is silently paying twice. If the first request is still running upstream when you start the second, you will be billed for both. Cancelling the original — actually closing the connection, not just abandoning the promise — is the part people skip.
Numbers to start from
These are starting points to be replaced by your own measurements, not recommendations dressed as facts. Measure your own time to first token distribution per model and set from that.
- Connect: a couple of seconds. Longer never helps.
- First byte: a small multiple of your p99 TTFT for that model. If p99 is two seconds, waiting thirty is waiting for nothing.
- Inter-token gap: generous relative to normal token spacing but far below your total — a stream that pauses for many seconds mid-generation is rarely recovering.
- Total: above your longest legitimate generation, with headroom, and strictly below whatever the layer above you allows.
The one rule with no measurement behind it: the numbers must decrease outward-to-inward, and you should be able to write the chain on one line. Most timeout incidents are not a badly chosen number — they are four numbers nobody ever wrote down together.
Where this lands operationally: the timeout chain, the retry budget and the failover policy are the same decision viewed three ways, which is the argument for them living in one place rather than in four services' configuration files. That is what a gateway is for, and Runix Router is ours — but the chain is yours to define either way, and nobody else can tell you what your longest legitimate response looks like.