Blog
Engineering notes, by layer of the stack
What we learn running AI infrastructure, written for the engineers who run it too. 58 posts, grouped by the layer of the stack each one is about, newest first within each.
02 · Agents
Agents
Layer 05 · domain agents and the harnesses that run them. 7 posts, newest first. Agents on the stack.
What is a coding agent? From autocomplete to pull requests
A coding agent reads a repository, plans, edits files, runs tests and revises until they pass. What separates it from an assistant, and what it is not.
The harness: what actually runs an AI agent
An agent harness is the runtime around a model: tools, permissions, context, budgets and logs. Why it, not the model, decides if an agent is safe to run.
Reviewable by default: why an agent's changes should land as diffs
Agent output should land as a diff a person can review, not a silent write. Blast radius, attribution, tests as the gate, and how reviewing an agent differs.
Evaluate coding agents on your own repository, not on a leaderboard
Leaderboard scores do not transfer to your codebase. How to build a task set from closed issues with fail-to-pass tests, and what to measure beyond pass rate.
Domain-specific AI agents: why agents are built one profession at a time
Why AI agents are built per profession: the domain supplies the tools, the data rules, the evaluation and the review gate, and a generic agent lacks all four.
Comic Drama vs Webtoon vs Motion Comic, Explained
Three overlapping formats for drawn stories: how webtoons, motion comics and comic dramas differ in production, distribution and audience — and which to make.
How to Make a Comic Drama with AI: Script to Screen
The full AI comic drama production line: script and characters, storyboards, styled panels, motion, voice and publishing — and where tools still fall short.
03 · Gateway
Gateway
Layer 04 · one way in to every model. 27 posts, newest first. Gateway on the stack.
Turning an LLM bill into a policy: budgets per team and per feature
Turning an LLM bill into enforceable policy: per-key quotas, allowed models, hard stops versus alerts, what happens at the limit, and how retries draw on it.
How to read an LLM price list
Units, cached input, batch and tiered rates, context bands and media pricing: how to read a vendor's LLM price list and record a dated baseline to compare on.
Auto routing versus a pinned model id: choosing an LLM routing policy
Two routing policies behind one endpoint: auto, chosen by cost and health, or a pinned model id. When each is right, and what to log to keep both honest.
One key can drain a shared quota pool
A shared quota pool with no per-key ceiling is first-come-first-served — one client's batch job empties it. Why runway beats balance as an alert threshold.
Your LLM traffic needs a retry budget, not a retry count
Retries multiply cost and load exactly when a provider is struggling. What a retry budget is, which errors deserve one, and how failover changes the arithmetic.
Why your LLM bill doesn't match the price list
Cache multipliers, injected prompts, tokenizer changes, tiered rates: the mechanics behind a surprising token bill, and how to reconcile one request by hand.
The security review an LLM gateway should survive
Eight questions a security review should ask any LLM gateway — key custody, logging, retention, tenant isolation, egress — and what good answers look like.
Your model id has an expiry date
Providers retire model versions on a published schedule. If the id is a string literal in your code, their calendar is your deploy calendar.
Where your prompts actually go
Every security review opens with this and most teams cannot answer it. How to trace the path text takes through an LLM integration, hop by hop.
What to settle before an LLM feature carries real traffic
Twelve decisions that are cheap before launch and expensive after: cost ceilings, timeouts, failover, what you log, and how you roll back.
Migrating to a gateway without a maintenance window
Shadow, then a percentage, then a cutover you can reverse in one config change — and the three things about LLM traffic the usual playbook does not cover.
How to tell whether a model change made things worse
Output varies run to run, so a quality regression does not announce itself. Four signals that move before anyone files a ticket.
How long should an LLM request be allowed to take?
Standard HTTP timeout advice is wrong for model traffic. Which timeout to set, where each one belongs, and why stacked timeouts cause the outage.
"OpenAI-compatible" is a surface, not a contract
Compatibility claims cover the endpoint shape, rarely the details that break you: ignored parameters, missing usage, streaming that ends wrong. How to test one.
LLM Router vs LLM Gateway: What's the Difference?
The two names describe overlapping tools. What routing and gateway features actually do, when the difference matters, and how to choose on capabilities.
What to Log for LLM Traffic, and What Not To
Status codes and latency miss the failures that matter for LLM traffic. The signals worth recording, the fields to redact, and the alerts worth waking up for.
What Is an LLM Gateway? A Plain Definition
An LLM gateway is a control plane between your application and model providers. What it does, what it does not do, and when a team actually needs one.
Surviving LLM Rate Limits: 429s, Budgets and Queues
Why provider rate limits bite in bursts, how RPM, TPM and concurrency interact, and the queueing and backoff patterns that keep throughput stable under 429s.
Streaming LLM Responses Without Breaking Failover
Once tokens have reached the user, failover stops being free. Mid-stream failure modes, what a proxy must preserve, and how to degrade without duplicating text.
Prompt Caching: When It Pays and When It Bites
How provider-side prompt caching works, the prefix rule that decides whether you get hits, and the cases where caching quietly changes behaviour.
Per-Team LLM Cost Attribution That Survives Audit
Provider invoices arrive as one number. How to attach cost to a team, feature and request at call time, and why reconstructing it later never works.
OpenRouter Alternatives: A Buyer's Checklist
Evaluating LLM gateways beyond routing: the contract, data, failover and cost-attribution questions that decide whether a gateway survives procurement.
LLM Provider Keys: Scope, Rotation and Blast Radius
Provider keys spread through repos, CI and notebooks. How to scope and rotate them, and why a compromised LLM key is a billing incident as well as a data one.
Build or Buy an LLM Gateway: An Honest Comparison
The first version of a gateway is a weekend. The parts that take a year are retry budgets, streaming failover, cost accounting and being on call for all of it.
LLM Cost Control: Tokens, Caching, and Budgets Explained
Token pricing asymmetry, prompt caching, model routing, budgets and kill switches — and the waste patterns that quietly inflate an LLM bill.
Designing for Model Failover: Lessons from Production LLM Traffic
Keeping LLM-backed products up when providers degrade: failure taxonomy, retry budgets, circuit breakers, failover trade-offs and degradation tiers.
Choosing an LLM Gateway: Seven Things That Actually Matter
How to evaluate an LLM gateway: API compatibility, failover behaviour, rate limits, cost tracking, observability, security and operational burden.
04 · Models
Models
Layer 03 · open-weight models, tuned and served. 6 posts, newest first. Models on the stack.
Single-tenant model serving explained: what a dedicated LLM inference endpoint changes
What single-tenant serving means beside a shared endpoint: isolation, latency under your own load, capacity planning, quantised variants and operations.
LoRA vs QLoRA vs full fine-tuning: what each one changes in the weights
What LoRA and QLoRA change in the weights, the memory and quality trade-offs, merging versus adapter serving, and when full fine-tuning is still warranted.
How to read an open-weight model licence before you fine-tune it
The clauses that decide whether you can fine-tune and ship an open-weight model: use restrictions, attribution, user thresholds, redistribution and patents.
How to evaluate a fine-tuned LLM against the model you run today
Held-out sets, task metrics versus generic benchmarks, regression checks on what you did not tune, contamination, and when a tuned model is ready to ship.
Fine-tuning vs RAG vs prompting: a decision guide by failure mode
When a prompt is enough, when retrieval fixes a knowledge or freshness gap, when fine-tuning is the right tool, and how the failure mode tells you which one.
DPO fine-tuning explained: what preference tuning changes and what it cannot
How DPO fine-tuning learns from chosen and rejected pairs without a reward model, where preference tuning helps and where it cannot, and the data pitfalls.
05 · Data
Data
Layer 02 · domain data, cleaned for training and evaluation. 8 posts, newest first. Data on the stack.
What a data quality report should contain, and how to read one sceptically
What a data quality report should state about coverage, duplication, extraction confidence, drops and licences, and how a buyer reads one sceptically.
Training data provenance: what every record should carry, and why a README is not proof
What a provenance entry should carry per record, from source and retrieval date to an SPDX licence identifier, and why a detected licence beats a README.
Split by source, not by row: how benchmark contamination reaches your own evaluation set
Row splits put near-duplicates on both sides of an evaluation. Why to split by repository, document or entity, and how to decontaminate and report overlap.
PII masking for training data that fails closed: detect, mask, measure, hold back
Pattern, NER and context detection, masking versus pseudonymisation versus dropping, why uncleared records are held back, and how to measure recall honestly.
Fail-to-pass and pass-to-pass tests in clean containers: what makes a coding task valid
Fail-to-pass and pass-to-pass tests, clean containers, why a task whose tests pass without the patch is invalid, and what SWE-bench Verified screened for.
Exact, near and semantic: why near-duplicate detection for training data needs all three
Exact hashing, MinHash and embeddings each catch a different kind of duplicate. What each misses and costs, and how duplicates inflate evaluation scores.
12 Data Questions to Ask an AI Vendor
The data-handling questions that decide an AI procurement review, what a good answer sounds like, and which vague replies should stop the process.
From Raw to Model-Ready: A Practical Guide to AI Data Pipelines
A stage-by-stage guide to AI data pipelines: ingestion, cleaning, deduplication, LLM-powered extraction, PII masking and quality scoring.
06 · Infrastructure
Infrastructure
Layer 01 · storage that keeps GPUs fed. 7 posts, newest first. Infrastructure on the stack.
When the bucket is the truth: AI file system cache mode versus file-system mode
Cache mode and fs mode: what each guarantees about the bucket behind an AI file system, which workloads fit which, and what to ask a vendor about truth.
The small files problem in machine learning: millions of samples, one metadata bill
Why millions of small samples cost more than their bytes: per-object overhead, listing, metadata pressure, and packing formats versus a metadata service.
Object storage is not a file system, and what POSIX on object storage adds
Object storage has no rename, no partial writes and no directories. What a POSIX layer over a bucket has to add, and what it still cannot promise.
Keeping GPUs fed: where the GPU data loading bottleneck comes from
How input pipelines starve accelerators: dataloader workers, random reads, IOPS versus throughput, object storage latency, and honest utilisation metrics.
FUSE, CSI or SDK: three ways to mount object storage in Kubernetes for ML workloads
FUSE mount, CSI driver or SDK: three ways to get a bucket into ML pods on Kubernetes, what each changes in your code, how each fails, and when to pick which.
A multi-tier cache for AI training data: what goes where
Memory, SSD and HDD tiers for training data: what goes where, how eviction and warming work, page cache or distributed cache, and when caching is pointless.
Introducing Runix FS: an AI file system for object storage
Runix FS puts a POSIX file system and a multi-tier cache in front of object storage. Why the layer is missing, what it is built on, and how early access works.
07 · Company
Company
Announcements, and why the stack is drawn the way it is. 3 posts, newest first. Company on the stack.
Why five layers: how Runix drew its AI infrastructure stack
How Runix drew five layers over the customer's own compute, which Unix idea each layer rebuilds, why each must work alone, and what the layered view omits.
Rebuild AI Unix: why Runix Lab is building an operating system for AI
Unix lasted on a few ideas: files, pipes, a kernel, one call interface. Runix Lab is rebuilding them for enterprise AI, in five layers.
Introducing Runix
Why we started Runix and what we set out to build: the gateway now called Runix Router, the pipelines now Runix Pipeline and Runix Data, and what came next.