provider: kubernetes runs workers on the nodes of a cluster you already have, without an agent on
them. The controlplane calls that cluster’s API over the internet, registers every Ready node that matches the
pool’s selector as a machine, and starts workers there as privileged Jobs pinned to their node. Workers talk back
only to the controlplane’s public endpoints, keep objects on the node, and mount workspace volumes through the
workspace’s own bucket credentials. The cluster sizes itself: the pool never creates or deletes nodes.
This page is the GKE recipe. The same provider takes any Kubernetes cluster; only the authentication and the
node defaults differ, which is what flavor selects.
What has to be true
The controlplane’s egress IPs are the EKS nodes’ public IPs (they sit in public subnets) plus the NAT gateway.
They change when nodes are replaced; rerun the script then, it only appends.
One script for the cloud side
gcloud, kubectl with the EKS context and aws logged in, and it is safe to rerun: the service
account, key, registry, image copy and firewall rule are created once and updated after. It ends by printing the
values block for the pool with the cluster’s endpoint and CA filled in.
One values block, one deploy
Paste the printed block underconfig.worker.pools in your values. The key itself doesn’t go in that file
or anywhere in the repository: put it in a values file of its own, readable only by you, and hand that to
the deploy, which passes extra arguments on to helm:
deploy.sh used to read the key from LEVERS9_GKE_SA_JSON and write it into a values file in the
repository. It doesn’t any more: the hosted deployment attaches its GKE cluster with levers9 cluster join,
which needs no key at all.) The pool’s settings, and why each is there:
credentials (the key, inline) or credentialsFile complete the block; the private values file above supplies the first.
The selector picks nodes by the card they carry, so the 1-, 2- and 4-GPU node pools all join one levers9
pool. To pin the pool to a single node pool instead, select on cloud.google.com/gke-nodepool.
Tunnel: no inbound access at all (opt-in)
By default the controlplane dials each worker for the things it initiates: the container gRPC service, log streams, exec, and endpoint traffic. That is why the recipe above needs public node addresses, the firewall rule, and an allow-list of controlplane addresses that goes stale whenever a controlplane node is replaced. Withtunnel: true under the pool’s kubernetes: block, the worker instead opens one outbound
stream to the controlplane and every one of those connections is multiplexed back over it. Nothing on the
cluster side is dialed: the node pool can be private, the firewall rule and the node-address allow-lists
go away, and only the controlplane’s call to the cluster API (to create worker Jobs) remains.
hostNetwork: false; set that only with tunnel: true, since nothing
outside the cluster can dial a pod address. Try it on one pool first and check a GPU endpoint, a box port,
a checkpoint restore and a multi-card job before you rely on it.
Sharing nodes with other workloads
The nodes do not have to be empty. Every reconcile (about every 15 s) reads the pods on each node and offers only what they leave: GPUs from their limits, CPU and memory from their requests. A model server holding two of a node’s four cards leaves a two-GPU machine; when it scales down, the machine grows; when the node is full, it is not a machine until a GPU frees up. A worker that already holds GPUs keeps them, so a neighbour scaling back up waits on Kubernetes like anyone else. A worker takes every card the node has free when it is created and serves several containers across them. The controlplane always counts its own workers on a node, pending or running, before adding another, so a node whose worker took two cards while a neighbour held the rest keeps a two-card worker until it winds down (a few idle minutes); the next worker then takes everything free. A multi-card job needs one worker with that many cards, so it waits for a node with that many free at once. To borrow capacity from a running service, scale it down (kubectl scale deploy <name> --replicas=1) and the freed cards show up in
levers9 machine list within a reconcile.
Check it works
The CLI’s own wait is shorter than a cold start, so
levers9 run may print DEADLINE_EXCEEDED on the first
job while the task completes; the check script reads the task record instead. The dashboard shows the task
under Apps either way.
Day two
- A new worker image. GKE pulls from the project’s registry, not from ECR, so after a worker build copy it
again and restart the worker: rerun
attach-gke-pool.sh(it copieslatestand is idempotent), thenkubectl -n levers9 delete jobs --allon the GKE side; the controlplane creates a fresh worker within a minute of the old record expiring (about ten minutes). - EKS nodes replaced. The controlplane’s egress IPs change with its nodes. Rerun
attach-gke-pool.sh: it appends the new IPs to the authorized networks and the firewall rule. - More or fewer GPUs. Resize the node pool (or enable its autoscaler); every Ready node becomes a machine, a drained node stops being one. The pool itself never touches nodes.
- Removing the pool. Delete its block from values and deploy; delete the
levers9namespace on the cluster; the firewall rule, registry and service account can stay for the next pool or be deleted by name (levers9-controlplane-to-<cluster>,levers9,levers9-controlplane@<project>).
When it does not
Workload logs are not in Cloud Logging unless the cluster enables them; to read a worker pod’s log, add your
own IP to the authorized networks and use
kubectl.
What is deliberately not automated
- The pool block goes into values by hand, so the change is reviewed like any other config change.
- Nodes: the pool uses what the node pool has. Turn GKE autoscaling on for the node pool if you want capacity to follow demand; the pool registers new nodes as they appear.
- A second cluster is a second pool with its own block; the script is per cluster.