Terms that come up while evaluating a gateway, defined plainly. Each one says what the thing is, how you would verify it yourself, and the mistake that is easy to make. Where the interesting part would be a number, this page tells you how to measure it rather than quoting one.
LLM gateway
A service that sits between your application and one or more model providers, presenting a single API while handling routing, retries, key custody, quotas and usage accounting.
How you would check it. Point a client at it and remove a provider key from your application: if the request still works and the spend still shows up attributed correctly, the gateway is doing its job.
Where people go wrong. Treating it as a proxy. A proxy forwards; a gateway takes responsibility for what happens when the thing it forwards to misbehaves.
What Is an LLM Gateway? A Plain Definition
LLM router
The part that decides which model or provider serves a given request, by cost, health, capability or explicit policy.
How you would check it. Send the same request twice under different routing policies and compare which upstream served it: the response metadata should tell you.
Where people go wrong. Assuming routing means cheapest. Routing on health matters more the day a provider degrades.
LLM Router vs LLM Gateway: What's the Difference?
Failover
Re-issuing a request to a healthy alternative when the first choice errors, rate-limits or times out.
How you would check it. Block one provider at the network level and watch whether requests still complete, and how the latency distribution changes when they do.
Where people go wrong. Counting a retry to the same failing provider as failover. If the alternative shares the failure domain, nothing failed over.
Designing for Model Failover: Lessons from Production LLM Traffic
Retry budget
A cap on how much of your traffic may be retries, so that a widespread failure cannot turn into self-inflicted load amplification.
How you would check it. Express it as a percentage of window traffic and alert when the retry share approaches it.
Where people go wrong. Setting per-request retry counts without a global budget: three retries each sounds modest until every request needs them at once.
Designing for Model Failover: Lessons from Production LLM Traffic
Circuit breaker
A switch that stops sending traffic to an upstream once its failure rate crosses a threshold, then probes cautiously before restoring it.
How you would check it. Watch the breaker state transitions in your logs during a real incident; a breaker that never opens is not tuned, it is decorative.
Where people go wrong. Keying the breaker on the provider alone. Providers fail per model far more often than they fail entirely.
Designing for Model Failover: Lessons from Production LLM Traffic
Streaming (SSE)
Delivering a response incrementally as it is generated, usually as server-sent events, so the reader sees output before the model has finished.
How you would check it. Measure time to first token separately from total time — streaming changes the first and barely moves the second.
Where people go wrong. Assuming failover still works mid-stream. Once bytes have been sent, recovering cleanly is a different and harder problem.
Streaming LLM Responses Without Breaking Failover
Time to first token (TTFT)
The delay between sending a request and receiving the first token of the response.
How you would check it. Measure at the client, on a streaming request, over a distribution — a single sample tells you nothing.
Where people go wrong. Reporting an average. Latency distributions are skewed, so quote p50, p95 and p99 or say nothing.
Prompt caching
Reusing the provider-side computation for a repeated prefix of a prompt, so the shared part is not paid for twice.
How you would check it. Compare billed input tokens against sent input tokens for a workload with a stable system prompt; the gap is the cache working.
Where people go wrong. Assuming any repetition caches. Caching keys on an exact prefix, so a timestamp near the top of the prompt quietly defeats it.
Prompt Caching: When It Pays and When It Bites
Rate limit (429)
A provider's cap on requests or tokens per interval, signalled by an HTTP 429 response.
How you would check it. Track the ratio of 429s to total requests per provider and per model, not just the absolute count.
Where people go wrong. Retrying a 429 immediately. Without backoff you consume the next window's allowance before it opens.
Surviving LLM Rate Limits: 429s, Budgets and Queues
Cost attribution
Assigning model spend to the team, customer or feature that caused it, rather than to one undifferentiated bill.
How you would check it. Issue separate keys per boundary you care about and check that the statement splits along the same lines.
Where people go wrong. Attributing after the fact from logs. If the boundary is not in the key, the attribution is a reconstruction and it will be argued with.
Per-Team LLM Cost Attribution That Survives Audit
Virtual key
A credential issued by the gateway rather than the provider, carrying its own quota, model allowlist and rate limits.
How you would check it. Revoke one and confirm the blast radius is exactly one consumer and no provider key had to be rotated.
Where people go wrong. Sharing one key across teams because it is easier. The day you need to revoke it, you find out what it cost.
LLM Provider Keys: Scope, Rotation and Blast Radius
Blast radius
How much breaks when one credential, provider or component fails.
How you would check it. Ask what stops working if this key leaks and has to be revoked in the next five minutes.
Where people go wrong. Measuring it only for outages. A leaked credential is the more common incident and the answer is usually worse.
LLM Provider Keys: Scope, Rotation and Blast Radius
Deduplication
Removing repeated records from a corpus before it is used for retrieval or fine-tuning — exact, near-duplicate and semantic.
How you would check it. Report how much was removed at each tier; a pipeline that cannot tell you what it dropped cannot be audited.
Where people go wrong. Stopping at exact matches. Near-duplicates are the ones that quietly skew a retrieval corpus.
From Raw to Model-Ready: A Practical Guide to AI Data Pipelines
PII masking
Detecting and redacting personal data before it reaches a model or a stored corpus.
How you would check it. Verify the failure mode: on an ambiguous field the pipeline should mask rather than pass through.
Where people go wrong. Treating recall as the only metric. A masker that fails open is a compliance incident waiting for a bad input.
12 Data Questions to Ask an AI Vendor
Observability (for LLM traffic)
Recording enough about each request — latency, tokens, model, outcome, request id — to answer questions after the fact, without hoarding content you should not keep.
How you would check it. Pick a past incident and check whether the logs you keep would have explained it.
Where people go wrong. Logging whole prompts and responses by default. That is a data-retention liability, and it is rarely the field you needed.
What to Log for LLM Traffic, and What Not To
AI file system
A file system built for how AI workloads read data: many readers, large datasets and frequent checkpoints. It is usually layered over object storage with a cache in front, so data is read through a path instead of fetched object by object.
How you would check it. Point an existing data loader at the mount without changing its code, then compare the second epoch's read time with the first. The cache should make the difference visible.
Where people go wrong. Assuming every write is in the bucket the moment it returns. Some modes write to the file system first and synchronise to the bucket in the background; know which one each mount uses.
The AI file system for object storage
POSIX semantics
The file behaviour code written for a local disk expects: open, read, write, seek, atomic rename, directory listing and permissions.
How you would check it. Run an atomic rename and a git status against the mount. Both depend on semantics that plain object storage does not have.
Where people go wrong. Assuming every object-storage mount provides them. Many read-mostly mounts do not support in-place modification or atomic rename, which is fine for a dataset and fatal for an agent's workspace.
Multi-tier cache
A cache spread over memory, SSD and HDD on the worker nodes, which serves data from the fastest tier that holds it and falls back to the bucket for anything it does not hold.
How you would check it. Read a large file, give the cache a moment to fill in the background, then read it again. The second read should not reach the bucket at all, which the object store's own request metrics will show.
Where people go wrong. Sizing the cache to the whole dataset. It needs to hold the working set: what the job reads repeatedly.
The AI file system for object storage
CSI driver
The Kubernetes Container Storage Interface plugin that turns a PersistentVolumeClaim into a volume mounted inside a pod.
How you would check it. Create a few hundred claims at once and time how long they take to bind. Provisioning that calls a cloud API for every volume slows down and runs into rate limits.
Where people go wrong. Using block volumes for per-pod workspaces at high density. Nodes have attachment limits, and they are reached long before the CPU runs out.
KV cache offload
Moving an LLM's attention key-value cache out of GPU memory, into host memory or a shared storage tier, so a prompt prefix computed once can be reused instead of recomputed.
How you would check it. Send the same long prefix to two different replicas and compare time to first token. With a shared tier, the second request should skip most of the prefill.
Where people go wrong. Counting only the GPU's own memory. A prefix cache that dies with one replica does not help a router that sends the next request somewhere else.
The AI file system for object storage
Something missing?
If a term left you guessing, tell us and it belongs here — contact@runixcloud.io.