EngineeringAugust 6, 2026·5 min read

Migrating to a gateway without a maintenance window

Putting a gateway in front of live model traffic is a routing change on the hot path of a product that is already earning money. The usual migration playbook applies (shadow, ramp, keep the rollback one config change away) but LLM traffic breaks three of its assumptions, and those are where migrations go wrong.

This is the sequence that works, and what to watch at each step.

Before you start: make the endpoint a variable

If the base URL is a string literal in your services, everything below requires a deploy, which means your rollback also requires a deploy, and a rollback you cannot perform in seconds is not a rollback. Move the base URL and the key into configuration you can change without shipping code, and confirm you can change it and see the effect. That check is the whole migration in miniature; if it does not work, nothing after this matters.

The same indirection is what lets you change model ids later without a redeploy, which is a separate problem with the same solution: covered in a post of its own.

Step 1: shadow, and compare what you can

Send a copy of real requests through the gateway while the original path continues to serve users. Discard the shadow response. You are not testing correctness yet — you are testing that the request shape survives the trip.

What to compare on the shadow path:

What not to compare: the text of the completions. Two calls to the same model with the same prompt do not produce the same output, so a diff of response bodies produces noise that looks like a problem and is not. Compare distributions later, on real traffic, with volume — not two responses side by side.

Step 2: a percentage, chosen by blast radius

Route a small share of production traffic through the gateway for real. Pick the share by what you can afford to have degraded, not by a round number — and pick the route deliberately. The best first candidate is high-volume and low-stakes: a summarisation endpoint, a classification step, something with a fallback that is not "the feature is broken". The worst first candidate is the one your demo uses.

Hold at each step long enough to see a full traffic cycle. LLM failure modes are often time-of-day shaped, because provider capacity is; a two-hour soak at 10% during your quiet period tells you very little about 10% at peak.

Step 3: the three things that are different about LLM traffic

Streaming is a second integration. A gateway that handles non-streamed requests perfectly can still break streaming: the framing, the terminator, tool-call deltas arriving as fragments, or an error mid-stream that reaches your client as a truncated response rather than an error. If any of your traffic streams, test it as a separate migration with its own ramp. Read what happens to a stream when a provider fails before you do.

Usage accounting can double-count during the overlap. While both paths are live you have two systems recording spend, and if you sum them your dashboard shows a cost spike that did not happen. Decide before you ramp which system is authoritative for the overlap window, and label the other one clearly. Finance discovering this on a monthly total is an unpleasant conversation you can avoid entirely.

Non-determinism hides regressions. Output varies run to run, so a quality regression does not announce itself the way a broken field does. Have at least one automated signal that does not depend on reading outputs: the parse rate on structured routes, the share of finish_reason: length, refusal-shaped response rate. These move before anyone files a ticket.

Step 4: cut over, and keep the way back open

At 100%, the old path should still be one config change away for at least a full billing cycle. Two reasons: the failure you have not seen yet is the one that only appears at full volume, and a rollback you have deleted is not a rollback.

Before you call it done, actually exercise the reversal. Flip back, confirm traffic serves, flip forward. A rollback path nobody has run is a hypothesis.

What to keep watching for a month

  1. Error rate by provider, not just overall — the aggregate hides one upstream degrading while the others compensate.
  2. Retry share. If the gateway is retrying more than you expect, something upstream is unhealthy and you are paying for it twice.
  3. Spend rate against its own recent norm, not against a fixed budget. A budget alarm tells you after the money is gone; a rate alarm tells you while it is happening.
  4. Time to first token as a distribution, per model. This is the number that tells you what the extra hop actually costs you — measure it yourself rather than accepting anyone's published figure, including ours.

The short version

Make the endpoint a variable. Shadow to check shape, not text. Ramp on a route you can afford to have degraded, through a full traffic cycle. Treat streaming as its own migration. Decide who owns usage accounting during the overlap. Keep the rollback live and prove it works. Then watch per-provider error rate, retry share, spend rate and TTFT for a month.

None of this is specific to any one gateway, which is the point — it is the same sequence whether you are adopting Runix Router, something you host yourself, or moving between two. If a vendor cannot describe how you would reverse the migration, that is the answer to a different and more important question.