An LLM gateway is a service that sits between your application and the model providers you call, so that requests, credentials, limits and cost accounting are handled in one place instead of in every service that happens to talk to a model.
That is the whole definition. The rest of this post is about what follows from it — because the interesting part is not what a gateway is, it is which problems disappear once you have one and which ones stubbornly do not.
What a gateway actually does
Four responsibilities show up in every serious implementation:
- One interface across providers. Your code speaks a single API — in practice usually the OpenAI schema, because that is what most SDKs and frameworks already emit — and the gateway translates to whatever the target provider expects. Switching a model becomes a configuration change rather than a code change.
- Credential custody. Provider keys live in the gateway. Application teams get gateway keys instead, which can be scoped, rate-limited and revoked individually. This is the difference between rotating one secret and hunting for it across nine repositories, four CI systems and someone's laptop.
- Traffic control. Retries, timeouts, failover between providers, per-key rate limits and concurrency caps. This is the part that decides whether a provider's bad hour becomes your incident.
- Accounting. Token counts and cost attached to every request, attributable to a key, a team, or a feature. Without this, an LLM bill is a single number nobody can explain.
What a gateway is not
Three things get confused with gateways often enough to be worth separating.
It is not a model. A gateway does not make answers better. If a model is wrong for your task, routing it through anything will not fix that.
It is not an agent framework. Orchestration, tool loops, memory and planning live in your application or in a framework you choose. A gateway that starts making those decisions for you becomes a dependency you cannot debug.
It is not a cache by default. Caching LLM responses is legitimate for some workloads and dangerous for others — see our post on cost control for where the line sits. Gateways can offer caching; conflating the two leads to surprising correctness bugs.
Proxy, gateway, router: the words people use
The vocabulary in this category is loose, and vendors use it inconsistently. A rough map:
- A proxy forwards requests, usually with minimal transformation. Useful for logging and key hiding, not much else.
- A router chooses between targets — often by price or availability. Routing is a feature, not a category.
- A gateway is the control plane: it includes routing, but the point is governance — who may call what, at what rate, with which credentials, at what cost, with what record.
The distinction matters when buying. If your problem is "we want the cheapest token", a router is enough. If your problem is "we cannot tell which team spent $40,000 last month and our provider keys are in six places", you want a gateway.
When a team actually needs one
Not immediately. A single service calling a single provider with one key does not need a control plane, and adding one early buys latency and an extra dependency for no benefit.
The signals that it is time are consistent:
- The second provider. The moment you support two, you are writing translation, retry and failover logic — and you will write it again in the next service.
- The second team. Once more than one team calls models, "whose spend is this" and "who can use which model" become real questions with no answer in the provider dashboard.
- The first incident. Provider degradation that takes your product down usually converts the argument on its own.
- The first procurement review. Somebody asks where keys are stored, what is logged, and how long it is retained. Answering per-service is painful; answering once is not.
Self-host or managed
Both are reasonable. Self-hosting keeps traffic inside your perimeter and gives you the source; it also means you own a component on the hot path of every AI feature you ship, including its upgrades, its failover behaviour and its 3am pages. A managed gateway trades that for a dependency you do not control.
The honest way to decide is to ask who will own the on-call rotation for it in six months. If the answer is "nobody has volunteered", self-hosting is a decision to accumulate risk rather than to save money.