A multi-tier cache for AI training is a set of storage tiers, usually memory, SSD and HDD on the nodes near the GPUs, holding copies of data whose durable home is an object store. The point is to make the second read of a sample local, and the first read local for every node except the one that fetched it. This post covers what belongs on each tier, how eviction and warming should work for training, how this differs from the page cache every node already has, and when a cache does nothing useful.
Why the cache sits in front of the bucket
Object storage is the right durable home for training data: cheap, durable and shared. It is the wrong place to read from on every step, because each read is a network request with its own cost and latency, paid once per sample. A cache near the GPUs changes the arithmetic: the bucket is read once per block, and later reads are served from local or near-local media.
This is not the prompt cache on the inference side, which stores attention state and is governed by prefix matching; here the cached thing is bytes, and correctness is about staleness, whether the cache still matches the bucket.
What a multi-tier cache for AI training holds on each tier
Tiers differ in capacity, cost and random-read behaviour, and the assignment follows from that.
- Memory holds what is being read right now: the blocks of the current and next few batches, and the metadata needed to find them. It is the only tier where random access to small items is effectively free, so hot small files belong here.
- SSD holds the working set: the part of the dataset an epoch will touch. Random reads are cheap enough to serve a shuffled loader, and capacity is large enough that a dataset bigger than memory still fits.
- HDD holds the long tail and the cold copy: data cheaper to keep nearby than to fetch again, read in large blocks where its poor random-read behaviour matters least.
The write path should land on the fastest tier that has room and demote from there, so that a block written once and read soon after never touches the slower media.
Eviction for an access pattern with no locality
General-purpose caches assume temporal locality: what was read recently will be read again soon. A shuffled epoch has almost none, because every sample is read exactly once per epoch, in a random order, and again next epoch in a different order. Under that pattern, least-recently-used eviction with a cache smaller than the working set degenerates into evicting each block just before it is needed again.
Two rules follow. First, size the cache to the working set, not to a hit-rate target: a cache that holds the whole epoch hits on every read after the first, and one that holds most of it can still miss on most reads. Second, protect the working set from scans: a validation run or a checkpoint write should not push the training set out, which scan-resistant policies, pinned directories or a capacity budget per job all address.
Eviction between tiers is a separate decision from eviction out of the cache: demotion from memory to SSD keeps a block close, while eviction from HDD sends the next read back to the bucket.
Warming: the first epoch is the whole problem
A cold cache does not help the first epoch at all, and for a short fine-tuning run the first epoch may be most of the job. Warming pulls the working set into the cache before the loader asks for it, and is cheaper than it looks: one read of the dataset from the bucket in large sequential requests, instead of the loader's many small random ones.
It is worth doing when the dataset is known ahead of time, which for training it almost always is; trigger it from the job definition, not by hand, and make it idempotent so that a retried job does not fetch twice. With a shared cache, warming once serves every node in the job.
Page cache versus distributed cache
Every Linux node already has a cache: the kernel's page cache, which keeps recently read file pages in otherwise unused memory. It is free, automatic and effective for one node whose dataset fits in memory, and the wrong tool for a cluster, for four reasons.
- It is per node. Every node holds its own copy and fetches from the bucket separately.
- It is unmanaged. The training process's own memory pressure evicts it, and there is no way to reserve capacity for the dataset.
- It is memory only. There is no SSD or HDD tier behind it, so anything larger than free memory is not cached at all.
- It knows nothing about the bucket, so a miss goes to the network every time, with no shared record of what was fetched.
A distributed cache fixes all four: it is shared across nodes, has explicit capacity on each tier, survives process restarts, and knows which blocks it holds and which are still only in the bucket. It costs a network hop on a hit served by another node and a metadata lookup to find the block, which is why local memory remains the first tier even in a distributed design.
When a cache is pointless
A cache earns its keep on rereads. Where there are none, it adds a hop and a bill.
- Single-pass data. A streaming run that reads each sample once gets nothing from a cache except the warming cost; the fix there is packing and parallel reads.
- A working set that cannot fit. If the cache holds a small fraction of what an epoch touches and the order is random, the hit rate is at best roughly that fraction.
- A CPU-bound loader. If decode and augmentation are the limit, faster reads change nothing; the GPU is waiting on workers, not on storage.
- Data already on local NVMe that fits. A copy step before the job is a one-tier cache with no software, and sometimes the right answer.
- Write-once outputs. Checkpoints and logs benefit from write buffering and an asynchronous path to the bucket, a different mechanism from a read cache, to be evaluated separately.
The honest test is to measure the hit rate on the second epoch and the step time on the first. If the second epoch is not faster, the cache is not doing anything; if the first epoch is the whole job, warming is the feature to evaluate, not the tiers.
Runix FS is a cache of this kind: memory, SSD and HDD tiers on the workers, written to the fastest available tier first, in front of an S3-compatible bucket whose layout stays unchanged, built on the Curvine project. It is in early access, sized with each customer for their working set and deployed in their own cloud account; the quick start brings up a single-node cluster, and the introduction covers what we will and will not claim about it.
Questions this raises
What is the difference between the page cache and a distributed cache for training data?
The page cache is per node, memory only, unmanaged and unaware of the bucket. A distributed cache is shared across nodes, has explicit capacity on memory, SSD and HDD tiers, survives restarts and tracks which blocks it holds.
How big should a cache for AI training data be?
Large enough to hold the working set an epoch touches. Because shuffled training has almost no temporal locality, a cache that holds only part of the working set can miss on most reads.
When does a cache not help a training job?
When the data is read once and never again, when the loader is CPU-bound on decoding, when the working set cannot fit, or when the data already sits on local NVMe that is large enough.