Failover before the first token is easy: nothing has been observed, so you can quietly try somewhere else. Failover after the first token is a product decision disguised as an infrastructure one, because the user has already seen output that you are now considering contradicting.
How streams fail
Four distinct modes, each needing a different response:
- Connection drop mid-stream. Bytes stop. Often indistinguishable at first from a model that is thinking.
- Stall. The connection is alive, tokens have stopped arriving. Without a token-rate floor, this can hang until an outer timeout fires minutes later.
- Truncation. The stream ends cleanly but early —
finish_reasonsayslength, or the content simply stops mid-sentence. The transport succeeded; the response is unusable. - Malformed frames. Rare, and brutal: a partial JSON chunk, a tool-call fragment that never completes. Strict parsers throw at exactly the wrong moment.
The detection primitive that covers most of these is a token-rate floor: if the stream produces fewer than N tokens in M seconds after having started, treat it as failed. It catches stalls and slow degradation that a single overall timeout misses entirely.
The restart problem
Suppose you detect a failure after 200 tokens have reached the user. Three options exist, and only three:
- Fail the request. Honest, simple, and the user loses the partial answer.
- Restart silently on another provider. The user sees the answer restart from scratch — or worse, sees the first attempt's text followed by a second attempt's, which reads as gibberish. Never do this without clearing what was already emitted, which most clients cannot do.
- Restart with continuation. Feed the partial output back as context and ask the new provider to continue. Plausible for prose, fragile for structured output, and it changes the token accounting because you now pay for the partial twice.
The right default for interactive traffic is to fail over invisibly only before the first token, and after that to surface the failure to the client and let the application decide. Which means the practical goal is to make the pre-first-token window as informative as possible: time-to-first-token is your best early health signal, and treating a slow TTFT as a routing failure lets you switch providers while it is still free.
What a proxy in the path must preserve
A gateway that terminates and re-emits streams can break things that clients depend on. The checklist:
- No buffering. Collecting the full response and forwarding it at the end technically works and destroys the reason for streaming. Any buffer that is not flushed per chunk is a bug.
- Frame fidelity. Tool-call deltas arrive in fragments that only make sense reassembled in order. A proxy that reorders, coalesces or drops empty deltas will break structured-output clients.
- Terminators. The final usage frame and the end-of-stream sentinel must survive. Dropping the usage frame is the most common way cost data ends up with a hole exactly where the expensive requests are.
- Cancellation propagation. When the client disconnects, the upstream request must be cancelled too — otherwise you keep paying for tokens nobody will read.
- Errors after 200. HTTP status is committed as soon as streaming starts, so a mid-stream failure cannot be a 500. It has to be an in-band error frame, and clients have to be written to expect one.
Degradation beats failure
When failover is not available, the interesting question is what the user gets instead of a spinner. Options that work in practice: switch to a smaller, faster model on the same provider; drop optional context to shorten the prompt and get under a stressed provider's limits; or return the partial answer with an explicit marker that it is incomplete. All three are better than a blank screen, and all three require the application to have been designed with a degraded mode — which is a decision nobody makes during an incident.
The infrastructure layer can only enable those choices, not make them. What it should guarantee is that a stream never silently becomes wrong: partial output is labelled partial, cancellations stop the meter, and a failure that happens after the first token is reported rather than papered over. That last property is the one to ask a vendor about, and it is the one we hold ourselves to in Runix Gateway.