Skip to main content
levers9 has one Go binary that holds the state and decides where things run, the controlplane, and many workers: pods that run your code in containers, on the platform’s own Kubernetes or on a cluster you attach. Your code, your requests and your files go through the controlplane; nothing in a worker is reached from outside any other way.

​
The controlplane

One process with three listeners: The scheduler and the pool controllers run inside it (pkg/cron). The gateway reaches a container by the address its worker reported, through the worker’s tunnel when it has one.

​
Where state lives

Each worker also keeps container logs on its own disk, and the dashboard spools the ones it streams.

​
Workers

A worker is one privileged pod. When it starts it:
  1. connects to the controlplane with the token it was given for this worker alone;
  2. starts the content cache (port 2050) that images and checkpoints load from, on the node’s cache directory;
  3. sets up its container runtimes, mounts JuiceFS at /data, and prepares CRIU if its pool has it on;
  4. creates the bridge network its containers live on and starts the container server (exec, logs, files);
  5. marks itself available and keeps a stream open on GetNextContainerRequest.
Each container gets its own overlay root on top of the image, a slot on the worker’s bridge network with a port forwarded to it, the volumes it asked for, and its secrets as environment variables.

​
How a call becomes a container

The scheduler picks pools by pool selector or GPU type (a preference list is tried in order), keeps a request that carries a checkpoint on pools with the same accelerator, and only places a workspace’s work on an attached cluster that cluster serves. An endpoint request that arrives over HTTP is buffered until a container is ready, then proxied to it.

​
Where workers come from

Every pool keeps a number of workers warm if it is configured to, and adds one when the backlog has nowhere to go.

​
Images

A blueprint build runs on the build pool: the worker builds the image with buildah, pushes it to the registry, and writes an archive index of it to S3. When a container needs the image, the worker mounts that archive lazily through FUSE and fetches only the files the container reads, through the content cache, so a large image starts without being downloaded first. A changed blueprint line rebuilds from that step on.

​
GPUs

A pool has one GPU type. Its worker pods go to nodes labelled with that type, and ask Kubernetes for nvidia.com/gpu where the pool enforces it. The worker reads the kubelet’s record of which cards it was given, writes a CDI spec for them with the NVIDIA toolkit, and hands the cards to the container with NVIDIA_VISIBLE_DEVICES. gpu_count=2/4/8 gives one container several cards of one node, and levers9.distributed runs one process per card (Train on 1, 2, 4 or 8 cards).

​
Checkpoint and restore

A model that takes minutes to load can start in seconds the second time. For an endpoint with checkpoint_enabled:
  • When. The first container starts cold. Once every process has finished on_start, the runner tells the worker it is ready, and the worker dumps the container without stopping it. A box is checkpointed once it answers on its port.
  • How. Under runc, CRIU with its CUDA plugin and cuda-checkpoint (NVIDIA driver 570 or newer); under gVisor, gVisor’s own checkpoint.
  • Where. A tar in the workspace’s bucket, a copy in the node’s cache, and a row in Postgres.
  • Restore. The next cold start carries the stub’s latest checkpoint; the scheduler sends it to a pool with the same accelerator, and the worker restores from its cache or from S3. If the restore fails, the container starts cold and the checkpoint is marked failed.
  • What it needs. At most one GPU, workspace storage, a pool with CRIU on and a worker with a cache. A dump that would not fit in the container’s memory is skipped. See Cold starts.

​
Attached clusters

The agent and the workers only ever dial out, so nothing needs to reach into your cluster. A cluster serves only the workspace that attached it unless a cluster admin shares it. Its workers get a configuration without the platform’s credentials. See Attach a cluster.

​
Volumes and secrets

A volume is a directory on JuiceFS (or in the workspace’s own bucket, when it has one) bind-mounted into the container; a CloudBucket is mounted per container with mountpoint-s3. Secrets are stored encrypted, decrypted by the controlplane into the container request, and arrive as environment variables.

​
The dashboard

Levers AI Lab is a Next.js app and a FastAPI backend. The backend calls the controlplane over REST and gRPC, streams changes to the browser as server-sent events (/events), proxies notebooks (/nb/...) to their boxes, and serves the workspace’s MCP server at /mcp. See The dashboard.