Engineering notes & announcements
What we learn running AI infrastructure, written for engineers who run it too.
All posts
One key can drain a shared quota pool
A shared quota pool with no per-key ceiling is first-come-first-served: one client's batch job empties it. Why runway beats balance as an alert threshold.
Your LLM traffic needs a retry budget, not a retry count
Retries multiply cost and load exactly when a provider is struggling. What a retry budget is, which errors deserve one, and how failover changes the arithmetic.
Why your LLM bill doesn't match the price list
Cache multipliers, injected prompts, tokenizer changes, tiered rates: the mechanics behind a surprising token bill, and how to reconcile one request by hand.
The security review an LLM gateway should survive
Eight questions a security review should ask any LLM gateway — key custody, logging, retention, tenant isolation, egress — and what good answers look like.
Your model id has an expiry date
Providers retire model versions on a published schedule. If the id is a string literal in your code, their calendar is your deploy calendar.
Where your prompts actually go
Every security review opens with this and most teams cannot answer it. How to trace the path text takes through an LLM integration, hop by hop.
What to settle before an LLM feature carries real traffic
Twelve decisions that are cheap before launch and expensive after: cost ceilings, timeouts, failover, what you log, and how you roll back.
Migrating to a gateway without a maintenance window
Shadow, then a percentage, then a cutover you can reverse in one config change, and the three things about LLM traffic the usual playbook does not cover.
How to tell whether a model change made things worse
Output varies run to run, so a quality regression does not announce itself. Four signals that move before anyone files a ticket.
How long should an LLM request be allowed to take?
Standard HTTP timeout advice is wrong for model traffic. Which timeout to set, where each one belongs, and why stacked timeouts cause the outage.
"OpenAI-compatible" is a surface, not a contract
Compatibility claims cover the endpoint shape, rarely the details that break you: ignored parameters, missing usage, streaming that ends wrong. How to test one.
Comic Drama vs Webtoon vs Motion Comic, Explained
Three overlapping formats for drawn stories: how webtoons, motion comics and comic dramas differ in production, distribution and audience — and which to make.
LLM Router vs LLM Gateway: What's the Difference?
The two names describe overlapping tools. What routing and gateway features actually do, when the difference matters, and how to choose on capabilities.
How to Make a Comic Drama with AI: Script to Screen
The full AI comic drama production line: script and characters, storyboards, styled panels, motion, voice and publishing — and where tools still fall short.
What to Log for LLM Traffic, and What Not To
Status codes and latency miss the failures that matter for LLM traffic. The signals worth recording, the fields to redact, and the alerts worth waking up for.
What Is an LLM Gateway? A Plain Definition
An LLM gateway is a control plane between your application and model providers. What it does, what it does not do, and when a team actually needs one.
Surviving LLM Rate Limits: 429s, Budgets and Queues
Why provider rate limits bite in bursts, how RPM, TPM and concurrency interact, and the queueing and backoff patterns that keep throughput stable under 429s.
Streaming LLM Responses Without Breaking Failover
Once tokens have reached the user, failover stops being free. Mid-stream failure modes, what a proxy must preserve, and how to degrade without duplicating text.
Prompt Caching: When It Pays and When It Bites
How provider-side prompt caching works, the prefix rule that decides whether you get hits, and the cases where caching quietly changes behaviour.
Per-Team LLM Cost Attribution That Survives Audit
Provider invoices arrive as one number. How to attach cost to a team, feature and request at call time, and why reconstructing it later never works.
OpenRouter Alternatives: A Buyer's Checklist
Evaluating LLM gateways beyond routing: the contract, data, failover and cost-attribution questions that decide whether a gateway survives procurement.
LLM Provider Keys: Scope, Rotation and Blast Radius
Provider keys spread through repos, CI and notebooks. How to scope and rotate them, and why a compromised LLM key is a billing incident as well as a data one.
Build or Buy an LLM Gateway: An Honest Comparison
The first version of a gateway is a weekend. The parts that take a year are retry budgets, streaming failover, cost accounting and being on call for all of it.
12 Data Questions to Ask an AI Vendor
The data-handling questions that decide an AI procurement review, what a good answer sounds like, and which vague replies should stop the process.
LLM Cost Control: Tokens, Caching, and Budgets Explained
Token pricing asymmetry, prompt caching, model routing, budgets and kill switches — and the waste patterns that quietly inflate an LLM bill.
From Raw to Model-Ready: A Practical Guide to AI Data Pipelines
A stage-by-stage guide to AI data pipelines: ingestion, cleaning, deduplication, LLM-powered extraction, PII masking and quality scoring.
Designing for Model Failover: Lessons from Production LLM Traffic
Keeping LLM-backed products up when providers degrade: failure taxonomy, retry budgets, circuit breakers, failover trade-offs and degradation tiers.
Choosing an LLM Gateway: Seven Things That Actually Matter
How to evaluate an LLM gateway: API compatibility, failover behaviour, rate limits, cost tracking, observability, security and operational burden.
Introducing Runix
Why we are building an OpenAI-compatible LLM gateway, managed data pipelines and ready-to-deploy solutions — and what comes next.