Terms that come up while evaluating a gateway, defined plainly. Each one says what the thing is, how you would verify it yourself, and the mistake that is easy to make. Where the interesting part would be a number, this page tells you how to measure it rather than quoting one.
LLM gateway
A service that sits between your application and one or more model providers, presenting a single API while handling routing, retries, key custody, quotas and usage accounting.
How you would check it. Point a client at it and remove a provider key from your application: if the request still works and the spend still shows up attributed correctly, the gateway is doing its job.
Where people go wrong. Treating it as a proxy. A proxy forwards; a gateway takes responsibility for what happens when the thing it forwards to misbehaves.
What Is an LLM Gateway? A Plain Definition
LLM router
The part that decides which model or provider serves a given request, by cost, health, capability or explicit policy.
How you would check it. Send the same request twice under different routing policies and compare which upstream served it: the response metadata should tell you.
Where people go wrong. Assuming routing means cheapest. Routing on health matters more the day a provider degrades.
LLM Router vs LLM Gateway: What's the Difference?
Failover
Re-issuing a request to a healthy alternative when the first choice errors, rate-limits or times out.
How you would check it. Block one provider at the network level and watch whether requests still complete, and how the latency distribution changes when they do.
Where people go wrong. Counting a retry to the same failing provider as failover. If the alternative shares the failure domain, nothing failed over.
Designing for Model Failover: Lessons from Production LLM Traffic
Retry budget
A cap on how much of your traffic may be retries, so that a widespread failure cannot turn into self-inflicted load amplification.
How you would check it. Express it as a percentage of window traffic and alert when the retry share approaches it.
Where people go wrong. Setting per-request retry counts without a global budget: three retries each sounds modest until every request needs them at once.
Designing for Model Failover: Lessons from Production LLM Traffic
Circuit breaker
A switch that stops sending traffic to an upstream once its failure rate crosses a threshold, then probes cautiously before restoring it.
How you would check it. Watch the breaker state transitions in your logs during a real incident; a breaker that never opens is not tuned, it is decorative.
Where people go wrong. Keying the breaker on the provider alone. Providers fail per model far more often than they fail entirely.
Designing for Model Failover: Lessons from Production LLM Traffic
Streaming (SSE)
Delivering a response incrementally as it is generated, usually as server-sent events, so the reader sees output before the model has finished.
How you would check it. Measure time to first token separately from total time — streaming changes the first and barely moves the second.
Where people go wrong. Assuming failover still works mid-stream. Once bytes have been sent, recovering cleanly is a different and harder problem.
Streaming LLM Responses Without Breaking Failover
Time to first token (TTFT)
The delay between sending a request and receiving the first token of the response.
How you would check it. Measure at the client, on a streaming request, over a distribution — a single sample tells you nothing.
Where people go wrong. Reporting an average. Latency distributions are skewed, so quote p50, p95 and p99 or say nothing.
Prompt caching
Reusing the provider-side computation for a repeated prefix of a prompt, so the shared part is not paid for twice.
How you would check it. Compare billed input tokens against sent input tokens for a workload with a stable system prompt; the gap is the cache working.
Where people go wrong. Assuming any repetition caches. Caching keys on an exact prefix, so a timestamp near the top of the prompt quietly defeats it.
Prompt Caching: When It Pays and When It Bites
Rate limit (429)
A provider's cap on requests or tokens per interval, signalled by an HTTP 429 response.
How you would check it. Track the ratio of 429s to total requests per provider and per model, not just the absolute count.
Where people go wrong. Retrying a 429 immediately. Without backoff you consume the next window's allowance before it opens.
Surviving LLM Rate Limits: 429s, Budgets and Queues
Cost attribution
Assigning model spend to the team, customer or feature that caused it, rather than to one undifferentiated bill.
How you would check it. Issue separate keys per boundary you care about and check that the statement splits along the same lines.
Where people go wrong. Attributing after the fact from logs. If the boundary is not in the key, the attribution is a reconstruction and it will be argued with.
Per-Team LLM Cost Attribution That Survives Audit
Virtual key
A credential issued by the gateway rather than the provider, carrying its own quota, model allowlist and rate limits.
How you would check it. Revoke one and confirm the blast radius is exactly one consumer and no provider key had to be rotated.
Where people go wrong. Sharing one key across teams because it is easier. The day you need to revoke it, you find out what it cost.
LLM Provider Keys: Scope, Rotation and Blast Radius
Blast radius
How much breaks when one credential, provider or component fails.
How you would check it. Ask what stops working if this key leaks and has to be revoked in the next five minutes.
Where people go wrong. Measuring it only for outages. A leaked credential is the more common incident and the answer is usually worse.
LLM Provider Keys: Scope, Rotation and Blast Radius
Deduplication
Removing repeated records from a corpus before it is used for retrieval or fine-tuning — exact, near-duplicate and semantic.
How you would check it. Report how much was removed at each tier; a pipeline that cannot tell you what it dropped cannot be audited.
Where people go wrong. Stopping at exact matches. Near-duplicates are the ones that quietly skew a retrieval corpus.
From Raw to Model-Ready: A Practical Guide to AI Data Pipelines
PII masking
Detecting and redacting personal data before it reaches a model or a stored corpus.
How you would check it. Verify the failure mode: on an ambiguous field the pipeline should mask rather than pass through.
Where people go wrong. Treating recall as the only metric. A masker that fails open is a compliance incident waiting for a bad input.
12 Data Questions to Ask an AI Vendor
Observability (for LLM traffic)
Recording enough about each request — latency, tokens, model, outcome, request id — to answer questions after the fact, without hoarding content you should not keep.
How you would check it. Pick a past incident and check whether the logs you keep would have explained it.
Where people go wrong. Logging whole prompts and responses by default. That is a data-retention liability, and it is rarely the field you needed.
What to Log for LLM Traffic, and What Not To
Something missing?
If a term left you guessing, tell us and it belongs here — contact@runixcloud.io.