An LLM provider key is unusual among credentials: it is trivially easy to use, it costs money on every call, and it typically carries no scoping of its own. That combination makes it worth treating with more care than its convenience suggests.
The blast radius is larger than it looks
A leaked database password gets you data. A leaked provider key gets an attacker three things: your data in transit through the requests they make, your money at whatever rate they can sustain, and your reputation if they use it for content that traces back to your account. The third is the one teams forget — abuse under your key is abuse attributed to your organisation.
Because the key is a single string with no per-call authorisation, the only real controls are where it lives, what it can reach, and how fast you can revoke it.
Where keys actually leak
Rarely through a breach of the vault. In practice:
- Client-side code. A key shipped to a browser or mobile app is public, whatever the obfuscation. If your app calls a model provider directly, it is a matter of when.
- Notebooks and prototypes. A key pasted into a notebook, committed to a research repo, and forgotten.
- Logs and traces. An error handler that dumps the outgoing request headers, then ships them to an observability vendor.
- CI configuration. Especially in forks and pull-request builds, where secrets policies are easy to get subtly wrong.
- Screenshots. Not a joke — terminal screenshots in issue threads are a real source.
Two keys, not one
The structural fix is indirection: the provider key exists in exactly one place, and everything else uses an internal key issued to it. Then the properties you want become achievable.
- Scope. An internal key can be limited to specific models, a spend ceiling, a rate, an environment. Provider keys generally cannot.
- Revocation without redeploy. Killing one team's key should not require rotating a secret every other service also reads.
- Attribution. One key per service is what makes cost attribution possible at all.
- Containment. A leaked internal key exposes one scope with one budget, not the whole account.
This is the security half of what an LLM gateway is for, and it is available whether you buy one or build it.
Rotation that is actually performed
Rotation policies fail when rotation is disruptive, so make it boring:
- Support two live keys per provider at once, so you can cut over and then retire the old one instead of doing both at the same instant.
- Rotate on a schedule and on every departure of someone who had access — the second one is the trigger people skip.
- Record when each key was last rotated somewhere a human will see it. Unmeasured policies are aspirations.
Detection, since prevention is imperfect
Assume a key will eventually leak and instrument for it. The strongest signals are cheap: spend rate per key against its recent baseline, calls from unexpected regions or user agents, sudden use of models that key has never called, and a spike in unique source addresses. Any one of these firing on a key is worth an alert. Automatic suspension on a hard spend ceiling is a blunt instrument, but a blunt instrument that stops a $20,000 weekend is a good trade.
The uncomfortable question for vendors
If you hand provider keys to a third-party gateway or framework, you have extended your blast radius to their infrastructure and their staff. That may be entirely reasonable — but ask where the keys are stored, who can decrypt them, whether access is logged, and what happens to them if you terminate. A vendor who cannot answer crisply is telling you something. Ours are answered on the security page.