Standard HTTP observability assumes that a 200 means success. For LLM traffic that assumption is wrong often enough to be dangerous: the most expensive failures return 200, take normal time, and produce output that is subtly unusable.
The signals that earn their storage
Per request, cheap to record and repeatedly useful:
- Time to first token. The single best health signal for a provider. It is stable enough per model that a shift is meaningful, and it is what users actually experience.
- Inter-token latency. Total duration conflates a long answer with a slow one; token rate separates them.
- Token counts, split. Input, output and cached input as separate fields — they are priced differently and they move independently.
finish_reasondistribution. A rising share oflengthfinishes means truncation, which no status code will tell you.- Attempt count and final provider. Without these, retry storms are invisible and per-provider quality comparisons are wrong.
- Structured-output parse rate. For routes expecting JSON, the fraction that parses is a direct quality metric that degrades before anything else does.
Notice that none of these require storing prompt content.
What not to log
Prompts and completions are the highest-risk data your system handles, and they end up in the widest-access system you own if you are careless. Defaults worth adopting:
- Off by default. Content logging should be opt-in per route, time-boxed, and separately access-controlled — not a flag someone left on after a debugging session.
- Never in the general log stream. Application logs are shipped to vendors, read by everyone on call, and retained for months.
- Redact credentials. An error handler that dumps outgoing headers will eventually leak a provider key into your observability vendor.
- Beware inadvertent capture. Exception messages, request-body dumps and trace attributes often contain prompt fragments nobody intended to keep.
If content capture is genuinely needed — eval sets, abuse investigation — treat that store as a separate system with its own retention and access rules, and tell users it exists. That is the distinction our privacy policy draws between content and operational metadata, and it is the one auditors ask about.
Aggregates that catch silent degradation
Single requests prove nothing; distributions do. Three worth computing continuously, per model and per provider:
- Output-token histogram. Truncation appears as a wall in the distribution, often at a suspicious power of two.
- Refusal rate. A rising share of refusal-shaped responses usually means an upstream policy or model change, not a change in your users.
- Empty and near-empty completions. Cheap to count, and one of the earliest indicators that something upstream changed.
Alerts worth being woken by
Most LLM alerting is either absent or noise. A defensible starting set: error rate by provider over a short window; time-to-first-token at p95 against a rolling baseline rather than a fixed threshold; spend rate per key against its own recent norm; and parse-failure rate on structured routes. Four alerts, each mapping to an action.
Deliberately excluded: total latency (dominated by legitimate long generations) and total spend (arrives too late to act on — rate is the actionable form).
Where it belongs
All of this is per-request instrumentation on the path every model call already takes. Implemented per service, you get four partial versions and no shared view of provider health; implemented once at the gateway, you get one dataset that answers both "is the provider healthy" and "who spent this". That is the same argument as cost attribution, and it lands in the same place.