An LLM gateway holds a position of unusual trust: every prompt your company sends passes through it, and the provider keys it holds can spend your money and read your traffic. That is true whether the gateway is a product you bought or a service you wrote in a week. This is the review we think any gateway should survive: phrased as questions, because the point is that you can ask them of anyone, including us.
1. Who holds the provider keys, and how?
The whole premise of a gateway is key consolidation: your teams hold gateway keys, the gateway holds the provider keys. The questions that follow are mechanical. Where do provider keys live at rest: an encrypted store, or a config file? Who can read them: which staff, which processes, which logs? Can a gateway key holder ever extract a provider key through any API surface, error message or debug endpoint? The last one deserves an actual test, not an assurance: trigger errors and read what comes back.
2. What do gateway keys authorize, and how narrowly?
Per-key scoping is the security payoff of the whole arrangement. A key should be limitable to specific models and groups, carry its own quota so a leak has a bounded blast radius, and be revocable in seconds without touching any other key. If disabling one team's key requires rotating a shared credential, the gateway has recreated the problem it was bought to solve.
3. What is logged, and does the operator need it?
There is a spectrum. Billing needs token counts, model ids, timestamps, key ids: pure metadata. Debugging sometimes wants request bodies. Those are different retention classes and a review should get a straight answer per class: are prompt and completion bodies stored at all, for how long, and can body logging be disabled for your tenant? "We keep metadata for accounting and do not persist message content" is a coherent answer. "Logs are kept for quality purposes" is not an answer; it is a question that has not been answered yet.
4. How long does anything live, and can you make it shorter?
Whatever is stored, the follow-ups are the same: what is the retention period, is it enforced by machinery (a TTL, a partition drop) or by intention (someone means to delete it), and what happens on your deletion request? Ask specifically about backups and about trace or debug captures made during incident response: the ad-hoc copies are the ones that outlive every policy.
5. What separates you from the operator's other tenants?
Multi-tenancy is fine; undisclosed multi-tenancy is not. Can another tenant's key ever route to your dedicated upstream, read your logs, or exhaust the quota you paid for? If you have negotiated a dedicated provider account or region for compliance reasons, what — concretely, in configuration — prevents your traffic from ever using the shared pool, and can the operator show you that configuration's effect rather than describe it?
6. Where does traffic go, and where is it processed?
A gateway adds a hop, and the hop has a geography. Which jurisdictions do the gateway itself and its storage run in? Which upstream providers, and in which of their regions, can a given model id resolve to? Model routing is exactly the place where a data-residency promise dies silently — a failover target in another jurisdiction is a compliance event nobody scheduled. If residency matters to you, the answer must cover the failover path, not just the happy path.
7. What happens on the operator's worst day?
Assume the gateway operator is compromised. What does the attacker get — live traffic, stored bodies, provider keys, all three? Now assume merely a bad deploy: does the gateway fail closed (requests error) or open (requests bypass controls)? For billing controls specifically, ask whether quota enforcement reads from a source that can lag, because a quota check against stale state is an overdraft mechanism with extra steps.
8. Can you verify any of this from the outside?
The strongest answers are the ones you can test with a key and an afternoon: error responses that leak nothing, per-request token accounting you can recompute, a revocation that takes effect mid-session, a model list that matches the contract. A gateway that invites that testing is making a structural claim — that its security posture survives contact with a motivated customer — and that claim is worth more than any diagram.
We keep our own answers to these questions on the security page, and the parts you can test from outside, we would rather you test. The review is the product working as intended: the entire argument for putting a gateway in front of your LLM traffic is that trust concentrated in one place can be inspected in one place.