A model swap is the one production change where the system keeps returning 200s and the thing that broke is the part no monitor watches. Latency is fine. Error rate is fine. The answers are just... worse, in a way nobody can point at for another three weeks, by which time nobody remembers the change.
This is not a hypothetical problem to plan for. Providers retire model versions on their own schedule, two months' notice is normal, so you will be moved off a model that works, on a date somebody else chose. The question is not whether to change models. It is how you will know if the change hurt.
Why the obvious approach does not work
The instinct is to run both models on the same prompts and compare the outputs. That fails for a specific reason: identical inputs do not produce identical outputs, so a diff of two responses shows differences on every single request. You end up reading them by hand, concluding "seems fine", and shipping on a sample of thirty.
Human review is not useless, it is the only thing that catches a subtly wrong tone or a misread instruction, but it does not scale and it cannot run continuously. It belongs at the end, on the cases the automated signals flag, not at the start on a random sample.
Four signals that do not require reading the output
Each of these is cheap, computable on live traffic, and moves before a human notices anything.
1. Structured-output parse rate. If any route asks for JSON, the fraction of responses that parse is a direct quality measurement with no judgement in it. A model that is slightly worse at instruction-following shows up here first and unambiguously. If you have one signal, have this one, and if no route currently asks for structured output, consider adding a small one that does purely as a canary.
2. Output length distribution, per route. Not the average: the shape.
A replacement model that is more verbose, more terse, or more prone to truncation moves the
distribution in a way an average hides. Watch particularly for a wall at the top of the
histogram: that is finish_reason: length, and it means answers are being cut off
mid-sentence. Track the share of length-finishes as its own number.
3. Refusal and hedge rate. Count responses matching a small set of refusal-shaped and hedge-shaped patterns — "I cannot", "I'm not able to", "As an AI", "it depends on". You are not trying to judge the content; you are watching a rate that should be stable. A jump means the new model draws a policy line somewhere the old one did not, which is the single most common way a model swap breaks a product without breaking anything.
4. Downstream acceptance, if you have it. The best signal is always the one from the next step: retry rate, edit rate, thumbs-down, "regenerate" clicks, the share of generated code that gets committed. If your product has any such action, it is worth more than the other three combined, because it measures what users did rather than what the response looked like.
Establish the baseline before you need it
All four are only useful against a before. Two weeks of the old model's numbers is worth more than any amount of pre-launch evaluation, and you cannot collect it retroactively.
The practical consequence: start recording these the day you learn a model is being retired, not the day you switch. Sixty days' notice is enough time to build a baseline; it is not enough time to build a baseline after you have already migrated.
Run both models, but not on the same request
Rather than shadowing every request through both models — which doubles your spend and still leaves you diffing non-deterministic text — split traffic. Send a slice to the new model and compare the four signals between the two populations rather than between two responses.
This is an A/B test on operational metrics rather than on outputs, and it has three advantages: it costs a percentage rather than a duplicate, the comparison is statistical rather than anecdotal, and it keeps running after you have stopped paying attention.
Give it a real sample. Refusal rate and parse rate are proportions, and a proportion measured over a few hundred requests moves around enough to hide a change worth acting on. If your volume is low, hold the split for longer rather than reading a small sample early.
What to do when a signal moves
- Confirm it is the model. Check whether the same signal moved on routes that did not change. If it moved everywhere, something else did: a prompt template, a context change, a seasonal shift in what users are asking.
- Pull the affected requests and read them. This is where human review earns its cost, because you now have twenty specific cases rather than a random sample.
- Try the prompt before you try another model. Most regressions after a model swap are instruction-following differences that a slightly more explicit prompt fixes. Changing models again is a bigger change with its own unknowns.
- Keep the option to move. If the pin lives in configuration rather than in code, reverting or trying a third model is a config change and a re-measure. If it lives in a binary, every hypothesis costs a deploy — which is the argument for keeping the model id out of your application in the first place.
The short version
You will be moved off a working model on someone else's schedule. Output diffs cannot tell you whether the replacement is worse, because output varies anyway. Parse rate, length distribution, refusal rate and downstream acceptance can — and they need a baseline that only exists if you started collecting it before the migration. Split traffic rather than shadowing, compare populations rather than responses, and keep the model id somewhere you can change without shipping code.
The observability side of this is covered in what to log for LLM traffic; the mechanics of moving traffic are in migrating without a maintenance window. If you would rather the routing layer hold the pin, that is what Runix Router does — but the four signals are yours to watch either way, and nobody else can tell you what "worse" means for your product.