Building a basic LLM gateway takes about a weekend: a request forwarder, a key store, a couple of provider adapters, some logging. That version works, and it is genuinely the right answer for a lot of teams. The disagreement is never about the weekend — it is about the eighteen months after it.
What the weekend version does not include
In rough order of when teams discover them:
- Retry budgets. Not retries — anyone writes retries. The budget that stops retries amplifying a provider brownout into your outage, which almost nobody adds before their first one.
- Streaming failover semantics. Deciding what happens when a stream dies at token 200, and implementing it without emitting duplicate text. Covered in streaming and failover.
- Silent degradation detection. Providers return 200 with truncated or malformed output. Catching this needs response-body distributions, not status codes.
- Token accounting that reconciles. Including cached-input fields, the usage frame at the end of a stream, and rate-at-time-of-call so history does not rewrite itself.
- Rate-limit shaping. Queues, class-based priority, header telemetry — see rate limits.
- Provider drift. New models, changed parameters, deprecated endpoints, subtly different tool-call semantics. This is not a project; it is a subscription paid in engineering time.
None of these are hard problems individually. Collectively they are a product, and the honest question is whether it is your product.
The cost that is easy to miss
Build-versus-buy spreadsheets usually compare vendor fees against a few weeks of engineering. The item that dominates in practice is neither: it is that a self-built gateway sits on the hot path of every AI feature you ship, so it acquires an on-call rotation, an upgrade cadence, and a bus factor. The question that predicts the outcome is not "can we build it" — you can — but "who owns it in six months when the person who wrote it is on another team". A component with no owner and a place in every request path is a slow-motion incident.
When building is clearly right
- Regulatory or contractual isolation. If traffic cannot leave your perimeter, the decision is made for you. Some managed vendors offer in-perimeter deployment; if not, build.
- You are the product. If routing quality is your differentiation, do not outsource your differentiation.
- Scale where margin matters. At sufficient volume a percentage fee exceeds a team's cost. That crossover is much further out than most teams estimate, but it is real.
- One provider, one team, modest volume. Then you do not need a gateway at all — you need a library, and that is a genuinely smaller thing.
When buying is clearly right
- You have a procurement review pending. The paperwork — entity, DPA, retention statements, security posture — is a real deliverable, and building it takes longer than building the proxy.
- Multiple teams share providers. The governance problem is the actual problem, and it is not fun to build.
- Your differentiation is upstack. If the product is the application, infrastructure that is table stakes should be bought.
The hybrid nobody mentions
Buy first, keep the option to leave. If the gateway speaks a standard API, migrating away later is a base-URL change — which means "buy now, build later if the economics turn" is a real path rather than a rationalisation, provided you avoid proprietary SDKs and keep your own copy of usage history. That constraint is worth writing into the contract while you still have leverage.
We would rather you ran this comparison honestly and concluded "build" than adopted Runix Gateway and resented it in a year.