Skip to main content
levers9 is a Go controlplane that schedules work onto workers over gRPC, workers that run each task in gVisor or runc and pull content-addressed images from a registry, Postgres and Redis for state, and JuiceFS over object storage for volumes. All of it ships as container images and a Helm chart (deploy/charts/levers9).

​
On AWS EKS (the Levers AI Lab setup)

Everything is in deploy/eks/ and reproducible from the CLI; nothing is clicked in the console.
Two settings in deploy/eks/values.yaml matter more than the rest:
  • worker.jobResourcesEnforced: true, so worker pods request nvidia.com/gpu and the NVIDIA device plugin hands them a GPU. Without it every GPU container fails with “not enough GPUs available”.
  • one pool per hardware class (default, build, l40s), each pinned to a node group with a nodeSelector.
Scale the GPU pool with eksctl scale nodegroup --cluster levers9 --region us-east-2 --name gpu-l40s --nodes N. Spot nodes can be reclaimed mid-run; a task on a reclaimed node is retried on another worker and the caller’s .remote() follows the retry.

​
Container hardening

worker.containerSecurity in the config hardens the containers runc starts (GPU and build pools, attached clusters; gVisor pools are sandboxed already). Every value ships as today’s behaviour or an observe mode, and a pool can override any field under pools.<name>.security. An attached cluster’s pool is built in code, so it takes the worker-wide value: switch the EKS pools first, through their overrides, and the global last. A stub can try the stricter settings before its pool does with env={"LEVERS9_SECURITY_PROFILE": "hardened"} or "LEVERS9_SECCOMP": "enforce"; it can never loosen them. worker.containerServerAuth (also per pool) and worker.volumeMountValidation work the same way, observe or log first. The order to switch them in, and what to check before each step, is in deploy/eks/README.md.

​
The dashboard and sign-in

deploy/eks/dashboard.yaml runs the dashboard web and API behind the same load balancer. Google sign-in, the allowed domains, the cookie domain and the session secret are environment variables on the API; they are listed on Sign-in, together with the development sign-in that accepts any email. The web app sends its security headers with every page (dashboard/web/src/lib/csp.ts): nosniff, a strict-origin-when-cross-origin referrer policy, no framing by other sites, and a content security policy. The policy names the API from NEXT_PUBLIC_API_BASE, the build argument the web image is built with, over https and wss (or http and ws for a local API), so a web image has to be built for the API it’ll talk to. For now only a small part of the policy is enforced (no plugins, no foreign <base>, no framing, forms post only to the app or the API). The full policy runs report-only: the browser blocks nothing under it and reports what it would have blocked to /csp-report, which writes one csp-report line per report to the web pod’s log. Once those lines stop showing anything the app needs, the full policy can be enforced. Storage isn’t in the policy, because the browser only reaches files through the API.

​
Secrets the controlplane runs on

The shipped config has defaults anyone can read: the database encryption key, the Postgres password password, and Redis with no password. The controlplane still starts on them, and logs a warning for each one it finds. The encryption key seals each workspace’s storage credentials. You can change it without downtime: make the new key database.postgres.encryptionKey and list the old one under database.postgres.previousEncryptionKeys. On start the controlplane opens stored values with either key and seals them again with the new one. Once its log says all_rows_on_current_key is true, drop the old key. After that, setting LEVERS9_REFUSE_DEFAULT_ENCRYPTION_KEY=true makes the controlplane refuse to start on the shipped key. The hosted setup’s step by step is in deploy/eks/README.md. A JuiceFS metadata URL with a password in it (redis://:PASSWORD@host:6379/0) is fine: the controlplane and workers hand the password to juicefs in its environment, so it never shows up in the process list.

​
The first token

On a fresh install, with no workspace in the database yet, the controlplane hands out one admin token: an Authorize call with no token creates the first workspace and returns it. It only does that for a caller that sends the bootstrap secret the controlplane was started with, so pick one before the controlplane first starts:
and give it to the controlplane as the LEVERS9_BOOTSTRAP_SECRET environment variable, from that secret (controlplane.auth.bootstrapSecret in the config works too, but the config file is mounted in more places than the controlplane). Without it bootstrap is off and the call answers “Bootstrap is off on this controlplane”. Then run any command on a machine that has no CLI config yet, with the same value in the environment, and leave the token empty when it asks; it saves the admin token it gets back:
The CLI sends it as x-levers9-bootstrap-secret gRPC metadata (over REST, the header is Grpc-Metadata-X-Levers9-Bootstrap-Secret on POST /api/v1/controlplane/auth/authorize). Once a workspace exists the secret does nothing and you can delete it. From then on tokens come from levers9 token create, and the dashboard reads its token from the levers9-dashboard-env secret. bin/local-up.sh does all of this for the local k3d cluster on its own.

​
Rolling out a change

The dashboard deploys itself: a push to main that touches dashboard/api or dashboard/web builds the changed image, pushes it to ECR as latest and restarts its deployment on the cluster, a few minutes end to end. The controlplane, worker and runner images are built on demand: run the build images workflow with the images you name (images=controlplane, …) or push an images-* tag for all of them; unless rollout=false, the matching deployments are restarted and worker pods recycled. The GitHub OIDC role has an EKS access entry with edit rights on the levers9 namespace and nothing else, and only the jobs that build or roll out can ask GitHub for a token to assume it. Every build pushes two tags: the one deployments follow (latest, or the tag you asked for) and a run tag, sha-<commit>-<run>-<attempt>, that no later build reuses, so you can always tell which commit an image came from and go back to it.

​
What the chart and the scripts assume

  • The controlplane isn’t on the internet by itself. Its Service is a ClusterIP; the only way in from outside is the TLS load balancer (levers9-grpc.<your domain>:443 for the CLI and SDK). Inside the cluster, workers and the dashboard use levers9-controlplane:1993 and :1994.
  • The chart has no default registry. Set images.controlplane.repository, config.worker.imageRegistry and config.imageService.runner.baseImageRegistry to your own; the chart won’t render without them.
  • Secrets stay out of the tree. deploy.sh reads the S3 keys from files under ~/.levers9, writes them to a private temporary file for helm, and removes it when it’s done. Each job (built images, JuiceFS, workspace storage) can have its own key: LEVERS9_IMAGES_KEYS, LEVERS9_JUICEFS_KEYS, LEVERS9_WORKSPACE_KEYS. Run deploy.sh rather than helm upgrade by hand, and pass extra helm flags to it.
  • Kubernetes access is narrow. The controlplane’s service account gets a ClusterRole that reads nodes and a Role that manages worker Jobs and reads their pods in the worker namespace. The chart still keeps the older every-verb ClusterRole while rbac.legacyClusterWide is true; turn it off once kubectl auth can-i agrees.
  • The controlplane runs privileged by default, because it mounts JuiceFS through FUSE. On a cluster that hands out /dev/fuse through a device plugin, controlplane.securityContext can drop that to SYS_ADMIN.
deploy/eks/README.md has the steps that need a hand on AWS: the CI role’s trust and permissions, giving the controlplane its own ECR role so the nodes can lose push rights, and locking down instance metadata.

​
Locally, for development

A GPU box anywhere joins as a worker with the agent: levers9 machine create --pool NAME writes the join command, with its token, to ~/.levers9/machines/ID.sh (readable by you only) and shows it with the secrets hidden; copy the file to the box and run it there. A controlplane that speaks TLS on a port other than 443 needs tls = true in the CLI context (levers9 config create NAME --tls) or LEVERS9_CONTROLPLANE_TLS=true; without either, only port 443 gets TLS. Plain gRPC to an in-cluster service name, as the dashboard API uses, is unchanged. The local controlplane can also build the agent from its own source for a machine that asks (pool join --dev, or a platform with no prebuilt binary). That’s on because the cluster-dev overlay sets LEVERS9_AGENT_DEV_BUILDS=true. Anywhere that setting is off, the controlplane only hands out prebuilt binaries, since the download link doesn’t ask for a token.

​
GPUs in a Kubernetes cluster you already run

The way to do this is Attach a cluster: levers9 cluster join puts an agent there that dials out, so the controlplane needs no key for that cluster and no way in. The older way, a pool with provider: kubernetes, uses the nodes of another cluster without an agent; the controlplane creates worker Jobs there over the cluster’s public API, with a service-account key and a firewall rule. Its GKE recipe is on its own page: GPUs in a GKE cluster.