> ## Documentation Index > Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Architecture > How levers9 fits together: the controlplane, workers, pools, GPUs, checkpoints and attached clusters levers9 has one Go binary that holds the state and decides where things run, the **controlplane**, and many **workers**: pods that run your code in containers, on the platform's own Kubernetes or on a cluster you attach. Your code, your requests and your files go through the controlplane; nothing in a worker is reached from outside any other way. ```mermaid theme={null} flowchart LR subgraph clients["Clients"] sdk["levers9 CLI and SDK"] caller["HTTP callers"] web["Levers AI Lab web
Next.js"] end api["Dashboard API
FastAPI"] subgraph cp["Controlplane"] grpc["gRPC :1993"] gw["HTTP :1994
REST and gateway"] sched["Scheduler and
pool controllers"] end subgraph state["State"] pg[("Postgres")] rd[("Redis")] jfs[("JuiceFS /data")] s3[("S3 and a registry")] end subgraph pool["Pools on the platform's Kubernetes"] w1["Worker pod
runc or gVisor"] c1["containers"] end subgraph yours["A cluster you attach"] ag["levers9-agent"] w2["Worker pod
runc, GPU"] c2["containers"] end sdk -->|gRPC| grpc caller -->|"/endpoint /job /queue /box"| gw web --> api api -->|"REST and gRPC"| cp grpc --- sched sched -->|"Kubernetes Job"| w1 sched -->|"order over the agent's stream"| ag ag -->|starts| w2 ag -.->|"dials out, 443"| grpc w1 -->|"asks for work"| grpc w2 -.->|"tunnel, 443"| grpc gw -->|proxies| c1 gw -.->|"through the tunnel"| c2 w1 --> c1 w2 --> c2 cp --- state ``` ## The controlplane One process with three listeners: | port | what it serves | | - | - | | 1993, gRPC | everything the SDK and CLI call (stubs, jobs, endpoints, queues, boxes, blueprints, volumes, secrets, tokens), the services workers use to take work and report state, the tunnels workers open, and the stream each cluster agent holds open | | 1994, HTTP | REST under `/api/v1`, and the gateway: `/endpoint/NAME/vN`, `/job/NAME/vN`, `/queue/NAME/vN`, `/box/NAME/vN/PORT/...`, `/asgi`, and per-deployment subdomains | | 9090 | Prometheus metrics | The scheduler and the pool controllers run inside it (`pkg/cron`). The gateway reaches a container by the address its worker reported, through the worker's tunnel when it has one. ## Where state lives | store | what it holds | | - | - | | Postgres | apps, stubs, deployments, tasks, workspaces, tokens, secrets (encrypted), volumes, checkpoints, clusters | | Redis | the scheduler's backlog, each worker's queue of container requests, worker and container state, container addresses, job arguments and results | | JuiceFS at `/data` | uploaded code, volumes and outputs; its metadata is in its own Redis and its data in S3, and both the controlplane and the platform's workers mount it | | S3 | each workspace's bucket (with its checkpoints) and the image archives workers mount lazily | | a container registry | the images a blueprint build produces, and the runner base images | Each worker also keeps container logs on its own disk, and the dashboard spools the ones it streams. ## Workers A worker is one privileged pod. When it starts it: 1. connects to the controlplane with the token it was given for this worker alone; 2. starts the content cache (port 2050) that images and checkpoints load from, on the node's cache directory; 3. sets up its container runtimes, mounts JuiceFS at `/data`, and prepares CRIU if its pool has it on; 4. creates the bridge network its containers live on and starts the container server (exec, logs, files); 5. marks itself available and keeps a stream open on `GetNextContainerRequest`. | pool | runtime | | - | - | | the default CPU pool | gVisor (`runsc`), so untrusted code runs behind a user-space kernel | | build and GPU pools, attached clusters | runc, with the NVIDIA toolkit for GPU pools | Each container gets its own overlay root on top of the image, a slot on the worker's bridge network with a port forwarded to it, the volumes it asked for, and its secrets as environment variables. ## How a call becomes a container ```mermaid theme={null} sequenceDiagram autonumber participant C as CLI or SDK participant CP as Controlplane participant R as Redis participant K as Kubernetes or cluster agent participant W as Worker participant X as Container C->>CP: GetOrCreateStub: blueprint, resources, volumes, secrets C->>CP: JobInvoke, or an HTTP call to the gateway CP->>R: task and container request onto the scheduler backlog Note over CP: the scheduler tries a running worker,
then room on one that is starting,
then asks a pool controller for a new one CP->>K: a worker Job for the pool (GPU type, cache, privileged) K->>W: the pod starts: cache, runc or gVisor, JuiceFS, network, container server W->>CP: registers, then streams GetNextContainerRequest CP->>W: the request: image, mounts, secrets as env, a checkpoint if there is one W->>X: lazy image mount, overlay root, bridge network, runner starts X->>CP: JobSetResult CP-->>C: the result, on the JobInvoke stream ``` The scheduler picks pools by pool selector or GPU type (a preference list is tried in order), keeps a request that carries a checkpoint on pools with the same accelerator, and only places a workspace's work on an attached cluster that cluster serves. An endpoint request that arrives over HTTP is buffered until a container is ready, then proxied to it. ## Where workers come from | pool kind | how a worker starts | | - | - | | local | the controlplane creates a Kubernetes Job in its own cluster | | attached cluster | the controlplane sends the Job to that cluster's agent, which creates it there | | provider | the controlplane rents or reaches a machine (EC2, OCI, Lambda Labs and others) and starts the worker on it | | agent machines | a machine you joined with `levers9 machine create` runs workers as Docker containers | Every pool keeps a number of workers warm if it is configured to, and adds one when the backlog has nowhere to go. ## Images A blueprint build runs on the `build` pool: the worker builds the image with buildah, pushes it to the registry, and writes an archive index of it to S3. When a container needs the image, the worker mounts that archive lazily through FUSE and fetches only the files the container reads, through the content cache, so a large image starts without being downloaded first. A changed blueprint line rebuilds from that step on. ## GPUs A pool has one GPU type. Its worker pods go to nodes labelled with that type, and ask Kubernetes for `nvidia.com/gpu` where the pool enforces it. The worker reads the kubelet's record of which cards it was given, writes a CDI spec for them with the NVIDIA toolkit, and hands the cards to the container with `NVIDIA_VISIBLE_DEVICES`. `gpu_count=2/4/8` gives one container several cards of one node, and `levers9.distributed` runs one process per card ([Train on 1, 2, 4 or 8 cards](https://levers9.mintlify.site/guides/multi-gpu.md)). ## Checkpoint and restore A model that takes minutes to load can start in seconds the second time. For an endpoint with `checkpoint_enabled`: ```mermaid theme={null} flowchart LR a["first container
starts cold, runs on_start"] --> b["runner signals
ready for checkpoint"] b --> c["worker dumps it, still running:
CRIU + cuda-checkpoint"] c --> d["tar to the workspace bucket
and the node cache"] d --> e["next cold start: the scheduler
attaches the checkpoint"] e --> f["worker restores it"] f -->|"restore fails"| g["starts cold instead"] ``` * **When.** The first container starts cold. Once every process has finished `on_start`, the runner tells the worker it is ready, and the worker dumps the container without stopping it. A box is checkpointed once it answers on its port. * **How.** Under runc, CRIU with its CUDA plugin and `cuda-checkpoint` (NVIDIA driver 570 or newer); under gVisor, gVisor's own checkpoint. * **Where.** A tar in the workspace's bucket, a copy in the node's cache, and a row in Postgres. * **Restore.** The next cold start carries the stub's latest checkpoint; the scheduler sends it to a pool with the same accelerator, and the worker restores from its cache or from S3. If the restore fails, the container starts cold and the checkpoint is marked failed. * **What it needs.** At most one GPU, workspace storage, a pool with CRIU on and a worker with a cache. A dump that would not fit in the container's memory is skipped. See [Cold starts](https://levers9.mintlify.site/sdk/cold-starts.md). ## Attached clusters ```mermaid theme={null} sequenceDiagram participant U as You participant K as Your cluster participant A as levers9-agent participant CP as Controlplane participant W as Worker U->>CP: levers9 cluster join NAME --gpu TYPE CP-->>U: one manifest: namespace, RBAC, token, agent Deployment U->>K: kubectl apply, through your own kube context K->>A: the agent starts A->>CP: dials out (ClusterAgentService), reports GPU nodes CP->>A: create a worker Job when work needs it A->>K: creates the Job K->>W: the worker starts W->>CP: registers, opens a tunnel on 443 CP->>W: requests to its containers ride the tunnel ``` The agent and the workers only ever dial out, so nothing needs to reach into your cluster. A cluster serves only the workspace that attached it unless a cluster admin shares it. Its workers get a configuration without the platform's credentials. See [Attach a cluster](https://levers9.mintlify.site/guides/clusters.md). ## Volumes and secrets A volume is a directory on JuiceFS (or in the workspace's own bucket, when it has one) bind-mounted into the container; a `CloudBucket` is mounted per container with mountpoint-s3. Secrets are stored encrypted, decrypted by the controlplane into the container request, and arrive as environment variables. ## The dashboard Levers AI Lab is a Next.js app and a FastAPI backend. The backend calls the controlplane over REST and gRPC, streams changes to the browser as server-sent events (`/events`), proxies notebooks (`/nb/...`) to their boxes, and serves the workspace's MCP server at `/mcp`. See [The dashboard](https://levers9.mintlify.site/guides/dashboard.md). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.