> ## Documentation Index > Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # GPUs in a GKE cluster > Attach a GKE GPU node pool to the controlplane as a worker pool: one script for the cloud side, one values block, one deploy > [!WARNING] > > This is the older way in, and it is the one that needs cloud-side network setup: a service account key, the > controlplane's egress IPs on the cluster's authorized networks, and a firewall rule to the nodes. For a new > cluster use [Attach a cluster](https://levers9.mintlify.site/guides/clusters.md) instead: one command, an agent that dials out, nothing to > allow-list. Keep reading only for a pool that already runs this way. A pool with `provider: kubernetes` runs workers on the nodes of a cluster you already have, without an agent on them. The controlplane calls that cluster's API over the internet, registers every Ready node that matches the pool's selector as a machine, and starts workers there as privileged Jobs pinned to their node. Workers talk back only to the controlplane's public endpoints, keep objects on the node, and mount workspace volumes through the workspace's own bucket credentials. The cluster sizes itself: the pool never creates or deletes nodes. This page is the GKE recipe. The same provider takes any Kubernetes cluster; only the authentication and the node defaults differ, which is what `flavor` selects. ## What has to be true | on the GKE side | why | | - | - | | the controlplane's egress IPs are on the cluster's **authorized networks** | the controlplane calls the Kubernetes API to create worker Jobs | | a service account with `roles/container.developer` | that is the identity it calls with; its key goes into the pool config | | the worker image is **pullable from that project** | GKE nodes cannot pull from the controlplane's ECR | | a **firewall rule** from the same egress IPs to the node pool, all TCP | the controlplane dials each worker to stream logs and proxy to containers | The controlplane's egress IPs are the EKS nodes' public IPs (they sit in public subnets) plus the NAT gateway. They change when nodes are replaced; rerun the script then, it only appends. ## One script for the cloud side ```bash theme={null} deploy/gke/attach-gke-pool.sh \ --project --zone --cluster \ --node-pool --gpu RTXPro6000 ``` It needs `gcloud`, `kubectl` with the EKS context and `aws` logged in, and it is safe to rerun: the service account, key, registry, image copy and firewall rule are created once and updated after. It ends by printing the values block for the pool with the cluster's endpoint and CA filled in. ## One values block, one deploy Paste the printed block under `config.worker.pools` in your values. The key itself doesn't go in that file or anywhere in the repository: put it in a values file of its own, readable only by you, and hand that to the deploy, which passes extra arguments on to helm: ```bash theme={null} umask 077 python3 -c 'import json,sys; print(json.dumps({"config": {"worker": {"pools": {"gke-rtx6000": {"kubernetes": {"credentials": open(sys.argv[1]).read()}}}}}}))' \ ~/.levers9/gke-sa.json > ~/.levers9/gke-pool-values.json deploy/eks/deploy.sh -f ~/.levers9/gke-pool-values.json ``` (`deploy.sh` used to read the key from `LEVERS9_GKE_SA_JSON` and write it into a values file in the repository. It doesn't any more: the hosted deployment attaches its GKE cluster with `levers9 cluster join`, which needs no key at all.) The pool's settings, and why each is there: ```yaml theme={null} gke-rtx6000: mode: external provider: kubernetes gpuType: RTXPro6000 imagesPath: /var/lib/levers9/images # COS keeps / read-only; /var/lib is writable durableDisksPath: /var/lib/levers9/disks cache: { enabled: false } # no cache locality off the mesh kubernetes: flavor: gke # tolerates the nvidia.com/gpu taint GKE puts on GPU nodes endpoint: https:// caCertificate: namespace: levers9 # created if missing nodeSelector: { cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000 } workerImage: -docker.pkg.dev//levers9/worker:latest ``` `credentials` (the key, inline) or `credentialsFile` complete the block; the private values file above supplies the first. The selector picks nodes by the card they carry, so the 1-, 2- and 4-GPU node pools all join one levers9 pool. To pin the pool to a single node pool instead, select on `cloud.google.com/gke-nodepool`. ## Tunnel: no inbound access at all (opt-in) By default the controlplane dials each worker for the things it initiates: the container gRPC service, log streams, exec, and endpoint traffic. That is why the recipe above needs public node addresses, the firewall rule, and an allow-list of controlplane addresses that goes stale whenever a controlplane node is replaced. With `tunnel: true` under the pool's `kubernetes:` block, the worker instead opens one outbound stream to the controlplane and every one of those connections is multiplexed back over it. Nothing on the cluster side is dialed: the node pool can be private, the firewall rule and the node-address allow-lists go away, and only the controlplane's call to the cluster API (to create worker Jobs) remains. ```yaml theme={null} kubernetes: flavor: gke tunnel: true ``` The tunnel only ever reaches the worker's own containers: a worker refuses a target that is not itself, and only a worker token can open one. It is off by default so a pool behaves exactly as before until you turn it on; the trade is one more long-lived stream per worker and the controlplane pod holding the sessions, against no inbound networking to maintain. With the tunnel on, the worker doesn't need the node's network either. Workers of a provider pool use the host network unless the pool says `hostNetwork: false`; set that only with `tunnel: true`, since nothing outside the cluster can dial a pod address. Try it on one pool first and check a GPU endpoint, a box port, a checkpoint restore and a multi-card job before you rely on it. ```yaml theme={null} gke-rtx6000: hostNetwork: false kubernetes: flavor: gke tunnel: true ``` ## Sharing nodes with other workloads The nodes do not have to be empty. Every reconcile (about every 15 s) reads the pods on each node and offers only what they leave: GPUs from their limits, CPU and memory from their requests. A model server holding two of a node's four cards leaves a two-GPU machine; when it scales down, the machine grows; when the node is full, it is not a machine until a GPU frees up. A worker that already holds GPUs keeps them, so a neighbour scaling back up waits on Kubernetes like anyone else. A worker takes **every card the node has free** when it is created and serves several containers across them. The controlplane always counts its own workers on a node, pending or running, before adding another, so a node whose worker took two cards while a neighbour held the rest keeps a two-card worker until it winds down (a few idle minutes); the next worker then takes everything free. A multi-card job needs one worker with that many cards, so it waits for a node with that many free at once. To borrow capacity from a running service, scale it down (`kubectl scale deploy --replicas=1`) and the freed cards show up in `levers9 machine list` within a reconcile. ## Check it works ```bash theme={null} deploy/gke/check-gke-pool.sh --gpu RTXPro6000 ``` It runs a job that asks for the GPU and prints what the container saw, reading the result from the task record so a slow first start does not hide it. A good run, from the first attach on 2026-09-18: ``` COMPLETE { "host": "runc", "nvidia_smi": "NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB", "cuda": true, "device": "NVIDIA RTX PRO 6000 Blackwell Server Edition", "matmul_ok": true } ``` What you see along the way, in order, with the time each took on that first run: | where | line | when | | - | - | - | | controlplane log | `loaded controller gpu_type=RTXPro6000 mode=external pool_name=gke-rtx6000` | at start | | controlplane log | `using existing machine hostname= machine_id=…` | within 15 s | | controlplane log | `added new worker to occupy existing machine … pool_name=gke-rtx6000` | right after | | GKE | `kubectl -n levers9 get pods` shows `worker-gke-rtx6000-…` Running on the GPU node | 30 to 60 s (image pull) | | worker log | `host driver libraries registered with the loader dirs=["/usr/local/nvidia/lib64"]` | at start | | worker log | `resolved NVIDIA_VISIBLE_DEVICES for GPU filtering resolved_visible_devices=0` | at start | | `levers9 worker list` | the worker goes pending → available | about a minute | | first job | container start, `worker.gpu_assignment` phase without error, result | about 2 min cold, 2 s warm | The CLI's own wait is shorter than a cold start, so `levers9 run` may print `DEADLINE_EXCEEDED` on the first job while the task completes; the check script reads the task record instead. The dashboard shows the task under Apps either way. ## Day two * **A new worker image.** GKE pulls from the project's registry, not from ECR, so after a worker build copy it again and restart the worker: rerun `attach-gke-pool.sh` (it copies `latest` and is idempotent), then `kubectl -n levers9 delete jobs --all` on the GKE side; the controlplane creates a fresh worker within a minute of the old record expiring (about ten minutes). * **EKS nodes replaced.** The controlplane's egress IPs change with its nodes. Rerun `attach-gke-pool.sh`: it appends the new IPs to the authorized networks and the firewall rule. * **More or fewer GPUs.** Resize the node pool (or enable its autoscaler); every Ready node becomes a machine, a drained node stops being one. The pool itself never touches nodes. * **Removing the pool.** Delete its block from values and deploy; delete the `levers9` namespace on the cluster; the firewall rule, registry and service account can stay for the next pool or be deleted by name (`levers9-controlplane-to-`, `levers9`, `levers9-controlplane@`). ## When it does not | symptom | cause | fix | | - | - | - | | `unable to load controller error="invalid provider name"` | the controlplane image predates the provider | rebuild the controlplane | | `unable to reconcile cluster nodes ... 403` or a timeout | egress IP not on the authorized networks, or the key lacks the role | rerun the script | | worker pod `ImagePullBackOff` | image not in the project's registry | rerun the script (it copies `latest`) | | worker exits with `failed to generate CDI spec` | driver mounted where the loader does not look | worker images from 2026-09-18 register `/usr/local/nvidia` themselves | | job fails with `not enough GPUs available ... visible=0` | device ids in the kubelet checkpoint are `nvidia0` style | worker images from 2026-09-18 read them as indices; re-copy the image | | job fails with `No such file or directory: 'nvidia-smi'` | the driver's binaries live in `/usr/local/nvidia/bin` | on the container PATH from the next worker build; call it by full path meanwhile | | CLI shows `dial tcp: lookup ` | the worker advertised the node name instead of its public IP | controlplane from 2026-09-18 advertises the node's external IP; the firewall rule must exist | | a worker Job stays `Pending` on a node that has the GPU | the machine was sized from the node's allocatable, not what other pods leave free | controlplane from 2026-09-18 subtracts the neighbours' requests; delete the pending Job | | endpoint container log: `S3 GetObject ... context canceled` while pulling the image | not permissions: the caller gave up (a dashboard warm-up probe times out at 3 min) and the autoscaler stopped the container at zero requests in flight, cancelling the pull | keep a request open through the cold start (`curl -m 1500`); the pull is not retried on its own | | vLLM dies capturing CUDA graphs: `FlashInfer backend is not available` | on Blackwell (RTX PRO 6000, B200) vLLM 0.29 prefers FlashInfer, whose Blackwell decode path is newer than the image's | SDK from 2026-09-18 defaults `VLLM_ATTENTION_BACKEND=TRITON_ATTN` on compute capability 10+; set it in the endpoint's `env` on an older worker image | Workload logs are not in Cloud Logging unless the cluster enables them; to read a worker pod's log, add your own IP to the authorized networks and use `kubectl`. ## What is deliberately not automated * The pool block goes into values by hand, so the change is reviewed like any other config change. * Nodes: the pool uses what the node pool has. Turn GKE autoscaling on for the node pool if you want capacity to follow demand; the pool registers new nodes as they appear. * A second cluster is a second pool with its own block; the script is per cluster. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.