> ## Documentation Index > Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Self-hosting > Running the controlplane, workers and dashboard on your own Kubernetes levers9 is a Go controlplane that schedules work onto workers over gRPC, workers that run each task in gVisor or runc and pull content-addressed images from a registry, Postgres and Redis for state, and JuiceFS over object storage for volumes. All of it ships as container images and a Helm chart (`deploy/charts/levers9`). ## On AWS EKS (the Levers AI Lab setup) Everything is in `deploy/eks/` and reproducible from the CLI; nothing is clicked in the console. | piece | what | | - | - | | cluster | `cluster.yaml` (eksctl): EKS 1.31, two m6i.xlarge system nodes, a Spot g6e.xlarge node group with one L40S each (0 to 4 nodes) | | images | ECR `levers9-controlplane`, `levers9-worker`, `levers9-runner`, `levers9-dashboard-api`, `levers9-dashboard-web`, built by the `build images` GitHub workflow and rolled out by its last job | | storage | S3 for JuiceFS volumes and for built blueprints; gp3 for Postgres and Redis | | public HTTPS | AWS Load Balancer Controller with the ControlPlane API: an ALB with an ACM certificate in front of the controlplane REST (`levers9.uselevers.com`), controlplane gRPC (`levers9-grpc.uselevers.com`), the dashboard (`levers9-app.uselevers.com`) and its API (`levers9-api.uselevers.com`) | ```bash theme={null} eksctl create cluster -f deploy/eks/cluster.yaml # about 20 minutes gh workflow run build-images.yml -R gnomefin/levers9 # images into ECR, then a rollout to the cluster deploy/eks/deploy.sh # storage class, runtime class, CRDs, Helm chart kubectl apply -f deploy/eks/controlplane.yaml # ALB + TLS; CNAME the hostnames to the ControlPlane address kubectl apply -f deploy/eks/dashboard.yaml # dashboard web + API behind the same ALB ``` Two settings in `deploy/eks/values.yaml` matter more than the rest: * `worker.jobResourcesEnforced: true`, so worker pods request `nvidia.com/gpu` and the NVIDIA device plugin hands them a GPU. Without it every GPU container fails with "not enough GPUs available". * one pool per hardware class (`default`, `build`, `l40s`), each pinned to a node group with a `nodeSelector`. Scale the GPU pool with `eksctl scale nodegroup --cluster levers9 --region us-east-2 --name gpu-l40s --nodes N`. Spot nodes can be reclaimed mid-run; a task on a reclaimed node is retried on another worker and the caller's `.remote()` follows the retry. ### Container hardening `worker.containerSecurity` in the config hardens the containers runc starts (GPU and build pools, attached clusters; gVisor pools are sandboxed already). Every value ships as today's behaviour or an observe mode, and a pool can override any field under `pools..security`. An attached cluster's pool is built in code, so it takes the worker-wide value: switch the EKS pools first, through their overrides, and the global last. | key | default | what it does | | - | - | - | | `profile` | `legacy` | `hardened` masks and makes read-only the `/proc` and `/sys` paths Docker protects, mounts sysfs read-only, sets `noNewPrivileges` (not for Docker sandboxes or a Box whose image user isn't root), and applies `apparmorProfile` where the node has it loaded | | `seccomp` | `log` | Docker's default profile, resolved for each container's capabilities, with GPU containers also allowed the NUMA and file-descriptor calls CUDA and NCCL make. `log` writes what it would refuse to the node's audit log and allows it; `enforce` refuses; `off` applies none | | `dropNetRaw` | `false` | removes `CAP_NET_RAW` and opens unprivileged ICMP instead, so `ping` keeps working | | `bridgePortIsolation` | `false` | stops containers on one worker from reaching each other directly | | `egress.mode` | `observe` | the chain that keeps containers off cloud metadata, the worker's own ports, other containers and private ranges. `observe` counts (metric `worker_container_egress_matches_total`, and a worker log line) without dropping; `enforce` drops | | `egress.allow` | `[]` | destinations containers may still reach, as `CIDR`, `CIDR:port` or `CIDR:port/proto`; DNS and the controlplane are always allowed | A stub can try the stricter settings before its pool does with `env={"LEVERS9_SECURITY_PROFILE": "hardened"}` or `"LEVERS9_SECCOMP": "enforce"`; it can never loosen them. `worker.containerServerAuth` (also per pool) and `worker.volumeMountValidation` work the same way, observe or log first. The order to switch them in, and what to check before each step, is in `deploy/eks/README.md`. ### The dashboard and sign-in `deploy/eks/dashboard.yaml` runs the dashboard web and API behind the same load balancer. Google sign-in, the allowed domains, the cookie domain and the session secret are environment variables on the API; they are listed on [Sign-in](https://levers9.mintlify.site/guides/sign-in.md), together with the development sign-in that accepts any email. The web app sends its security headers with every page (`dashboard/web/src/lib/csp.ts`): `nosniff`, a `strict-origin-when-cross-origin` referrer policy, no framing by other sites, and a content security policy. The policy names the API from `NEXT_PUBLIC_API_BASE`, the build argument the web image is built with, over `https` and `wss` (or `http` and `ws` for a local API), so a web image has to be built for the API it'll talk to. For now only a small part of the policy is enforced (no plugins, no foreign ``, no framing, forms post only to the app or the API). The full policy runs report-only: the browser blocks nothing under it and reports what it would have blocked to `/csp-report`, which writes one `csp-report` line per report to the web pod's log. Once those lines stop showing anything the app needs, the full policy can be enforced. Storage isn't in the policy, because the browser only reaches files through the API. ### Secrets the controlplane runs on The shipped config has defaults anyone can read: the database encryption key, the Postgres password `password`, and Redis with no password. The controlplane still starts on them, and logs a warning for each one it finds. The encryption key seals each workspace's storage credentials. You can change it without downtime: make the new key `database.postgres.encryptionKey` and list the old one under `database.postgres.previousEncryptionKeys`. On start the controlplane opens stored values with either key and seals them again with the new one. Once its log says `all_rows_on_current_key` is true, drop the old key. After that, setting `LEVERS9_REFUSE_DEFAULT_ENCRYPTION_KEY=true` makes the controlplane refuse to start on the shipped key. The hosted setup's step by step is in `deploy/eks/README.md`. A JuiceFS metadata URL with a password in it (`redis://:PASSWORD@host:6379/0`) is fine: the controlplane and workers hand the password to `juicefs` in its environment, so it never shows up in the process list. ### The first token On a fresh install, with no workspace in the database yet, the controlplane hands out one admin token: an `Authorize` call with no token creates the first workspace and returns it. It only does that for a caller that sends the bootstrap secret the controlplane was started with, so pick one before the controlplane first starts: ```bash theme={null} kubectl -n levers9 create secret generic levers9-bootstrap --from-literal=secret="$(openssl rand -hex 32)" ``` and give it to the controlplane as the `LEVERS9_BOOTSTRAP_SECRET` environment variable, from that secret (`controlplane.auth.bootstrapSecret` in the config works too, but the config file is mounted in more places than the controlplane). Without it bootstrap is off and the call answers "Bootstrap is off on this controlplane". Then run any command on a machine that has no CLI config yet, with the same value in the environment, and leave the token empty when it asks; it saves the admin token it gets back: ```bash theme={null} LEVERS9_BOOTSTRAP_SECRET= levers9 volume list ``` The CLI sends it as `x-levers9-bootstrap-secret` gRPC metadata (over REST, the header is `Grpc-Metadata-X-Levers9-Bootstrap-Secret` on `POST /api/v1/controlplane/auth/authorize`). Once a workspace exists the secret does nothing and you can delete it. From then on tokens come from `levers9 token create`, and the dashboard reads its token from the `levers9-dashboard-env` secret. `bin/local-up.sh` does all of this for the local k3d cluster on its own. ### Rolling out a change The dashboard deploys itself: a push to `main` that touches `dashboard/api` or `dashboard/web` builds the changed image, pushes it to ECR as `latest` and restarts its deployment on the cluster, a few minutes end to end. The controlplane, worker and runner images are built on demand: run the `build images` workflow with the images you name (`images=controlplane`, ...) or push an `images-*` tag for all of them; unless `rollout=false`, the matching deployments are restarted and worker pods recycled. The GitHub OIDC role has an EKS access entry with edit rights on the `levers9` namespace and nothing else, and only the jobs that build or roll out can ask GitHub for a token to assume it. Every build pushes two tags: the one deployments follow (`latest`, or the tag you asked for) and a run tag, `sha---`, that no later build reuses, so you can always tell which commit an image came from and go back to it. ### What the chart and the scripts assume * **The controlplane isn't on the internet by itself.** Its Service is a ClusterIP; the only way in from outside is the TLS load balancer (`levers9-grpc.:443` for the CLI and SDK). Inside the cluster, workers and the dashboard use `levers9-controlplane:1993` and `:1994`. * **The chart has no default registry.** Set `images.controlplane.repository`, `config.worker.imageRegistry` and `config.imageService.runner.baseImageRegistry` to your own; the chart won't render without them. * **Secrets stay out of the tree.** `deploy.sh` reads the S3 keys from files under `~/.levers9`, writes them to a private temporary file for helm, and removes it when it's done. Each job (built images, JuiceFS, workspace storage) can have its own key: `LEVERS9_IMAGES_KEYS`, `LEVERS9_JUICEFS_KEYS`, `LEVERS9_WORKSPACE_KEYS`. Run `deploy.sh` rather than `helm upgrade` by hand, and pass extra helm flags to it. * **Kubernetes access is narrow.** The controlplane's service account gets a ClusterRole that reads nodes and a Role that manages worker Jobs and reads their pods in the worker namespace. The chart still keeps the older every-verb ClusterRole while `rbac.legacyClusterWide` is true; turn it off once `kubectl auth can-i` agrees. * **The controlplane runs privileged by default**, because it mounts JuiceFS through FUSE. On a cluster that hands out `/dev/fuse` through a device plugin, `controlplane.securityContext` can drop that to `SYS_ADMIN`. `deploy/eks/README.md` has the steps that need a hand on AWS: the CI role's trust and permissions, giving the controlplane its own ECR role so the nodes can lose push rights, and locking down instance metadata. ## Locally, for development ```bash theme={null} make k3d-up # a k3d cluster named levers9 make build && make build-runner kustomize build --enable-helm manifests/kustomize/overlays/cluster-dev | kubectl apply -f- ``` A GPU box anywhere joins as a worker with the agent: `levers9 machine create --pool NAME` writes the join command, with its token, to `~/.levers9/machines/ID.sh` (readable by you only) and shows it with the secrets hidden; copy the file to the box and run it there. A controlplane that speaks TLS on a port other than 443 needs `tls = true` in the CLI context (`levers9 config create NAME --tls`) or `LEVERS9_CONTROLPLANE_TLS=true`; without either, only port 443 gets TLS. Plain gRPC to an in-cluster service name, as the dashboard API uses, is unchanged. The local controlplane can also build the agent from its own source for a machine that asks (`pool join --dev`, or a platform with no prebuilt binary). That's on because the `cluster-dev` overlay sets `LEVERS9_AGENT_DEV_BUILDS=true`. Anywhere that setting is off, the controlplane only hands out prebuilt binaries, since the download link doesn't ask for a token. ## GPUs in a Kubernetes cluster you already run The way to do this is [Attach a cluster](https://levers9.mintlify.site/guides/clusters.md): `levers9 cluster join` puts an agent there that dials out, so the controlplane needs no key for that cluster and no way in. The older way, a pool with `provider: kubernetes`, uses the nodes of another cluster without an agent; the controlplane creates worker Jobs there over the cluster's public API, with a service-account key and a firewall rule. Its GKE recipe is on its own page: [GPUs in a GKE cluster](https://levers9.mintlify.site/guides/gke-pool.md). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.