Runix FS is a POSIX file system and a multi-tier distributed cache over object storage, developed in the public Runix FS repository, an Apache 2.0 fork of Curvine. Source build instructions and development notes are in the fork README. The engine commands below follow the Curvine documentation and retain the upstream binary names. In early access, we deploy and size the production cluster with you.
1. What you deploy
- Masters hold the metadata: the namespace, the mount table and scheduling. They replicate it with Raft, so a highly available cluster runs several; the Kubernetes examples use three.
- Workers hold the cache, in memory, SSD and HDD tiers, and serve data directly to clients. On a miss they read from the bucket.
- Access is whichever of these your workloads need: the FUSE daemon for a POSIX mount, the CSI driver for Kubernetes, the Hadoop-compatible client for Spark and Flink, or the Java, Python and Rust SDKs (which to pick for an ML workload).
File paths map one-to-one to object keys, so what is in the bucket keeps its original layout and stays readable without the cluster. How writes reach the bucket depends on the mount mode, in the next steps.
2. Try it on one machine
For Runix FS, build from source using the fork README. To evaluate the upstream engine instead, choose a package compatible with your Linux system from the Curvine releases page; those are upstream Curvine packages, not Runix FS releases. From the built or extracted distribution directory, start a local cluster:
export CURVINE_MASTER_HOSTNAME=localhost
./bin/restart-all.sh
This starts a master, a worker and the FUSE daemon, mounted at /curvine-fuse by default. Confirm the cluster is healthy and list the namespace:
bin/cv report
bin/cv fs mkdir /quick-start
bin/cv fs ls /
The default web UI ports are 9001 on the master and 9002 on the worker; check your cluster configuration if you override them. localhost is only right for a single machine: on more than one node, set the master, worker and client hostnames to addresses the other nodes can reach.
3. Attach a bucket
A mount attaches a bucket prefix to a path in the namespace, in one of two modes. cache_mode is for read-heavy data: the bucket stays the source of truth, writes pass straight through to it, and a read that misses the cache is served from the bucket while the cache fills in the background. fs_mode, the default, is for write-heavy work: every operation completes in Runix FS first and is synchronised to the bucket asynchronously. This example uses cache_mode:
bin/cv mount s3://training-data/imagenet /training-data/imagenet \
--write-type cache_mode \
-c s3.endpoint_url=https://s3.us-east-1.amazonaws.com \
-c s3.region_name=us-east-1 \
-c s3.credentials.access=$AWS_ACCESS_KEY_ID \
-c s3.credentials.secret=$AWS_SECRET_ACCESS_KEY
By default the path inside the namespace mirrors the bucket and prefix, as above. Mounts are stored in the metadata, so they survive a restart. Three cautions from the Curvine documentation: in cache_mode, deleting a file through the mount deletes it from the bucket too; in fs_mode, a rename on object storage becomes a delete and a full copy, so avoid renaming large directories; and changes made to the bucket by other tools are not picked up on their own: mount with --read-verify-ufs true in cache_mode, or run bin/cv mount resync in fs_mode. To warm the cache before a job starts, load a path ahead of time:
bin/cv load /training-data/imagenet/train/shard-00001.tar --watch
4. Mount it
The single-machine start in step 2 already launched the FUSE daemon, mounted at /curvine-fuse, so existing code can read files by path straight away:
ls /curvine-fuse/training-data/imagenet
To mount it somewhere else, stop the daemon first, then start it with a mount path; starting a second daemon while one is running fails:
bin/curvine-fuse.sh stop
bin/curvine-fuse.sh start --mnt-path /mnt/curvine
In fs_mode, and on paths that are not mounted from a bucket, standard file operations work, including rename, permissions, symbolic links and extended attributes. In cache_mode they depend on what the bucket itself supports; symbolic links and file locks, for example, may fail. FUSE 2 and FUSE 3 are both supported.
5. Kubernetes volumes
The CSI driver provisions a volume by creating a directory in the namespace, so claims bind without calling a cloud API. A StorageClass points the driver at the masters and names the directory new volumes are created under:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: curvine-sc
provisioner: curvine
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
master-addrs: "m0:8995,m1:8995,m2:8995"
fs-path: "/agents"
path-type: "DirectoryOrCreate"
Claims against curvine-sc can be ReadWriteMany, so several pods can share one volume. The example manifests ship in the Curvine source tree under curvine-csi/examples.
6. Managing the cache
List the mounts, check that the buckets behind them are reachable, and release cache space without touching the bucket:
bin/cv mount
bin/cv mount --check
bin/cv fs free -r /training-data/imagenet
free only releases data that also exists in the bucket; anything not yet synchronised to it is skipped.
7. Early access with Runix
The steps above are enough to evaluate the engine. A production cluster is a sizing question: how much of the working set the cache should hold, which tiers the workers get, and where the masters run. In early access we answer it with you, deploy the cluster into your own cloud account, and support it afterwards under a contract with Runix AI Inc.
Tell us about your data: where it lives, how big it is and what reads it. We reply within one business day.