> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Attach a cluster

> Turn any Kubernetes cluster with GPUs into a pool of workers with one command; the cluster dials in, nothing dials into it

`levers9 cluster join` attaches a cluster you already run, GKE, EKS, AKS or your own, as a pool of GPU
workers. It works the way `levers9 login` works for a person: the cluster side initiates everything.
One agent runs inside the cluster, holds an outbound stream to the controlplane, reports the nodes and
what they have free, and starts worker Jobs when asked. The workers carry their own inbound connections
back over the same kind of stream. So there is no public node address, no firewall rule, no allow-list
of controlplane addresses, and the controlplane holds no credentials for your cluster.

## The shape

```mermaid theme={null}
flowchart LR
  subgraph yours["your cluster · GKE, EKS, AKS, yours"]
    direction TB
    agent["levers9-agent<br/>one pod · Kubernetes service account"]
    workers["workers<br/>one per node · every free card"]
    nodes["GPU nodes<br/>private is fine · shared is fine"]
    agent -. "creates worker Jobs" .-> workers
    workers --- nodes
  end
  subgraph cp["levers9 controlplane"]
    direction TB
    sched["scheduler<br/>one queue, many pools"]
    reg["cluster registry<br/>no keys to your cloud"]
    tun["tunnel registry<br/>logs · exec · endpoints"]
  end
  agent -- "agent stream: nodes and free cards up, Job orders down" --> sched
  workers -- "worker tunnel: container gRPC, logs, exec, endpoint traffic ride back" --> tun
  sched --- reg
```

Both arrows are opened from inside your cluster and stay open; nothing on the controlplane side ever
dials the cluster or its nodes. That is the whole difference from a configured pool:

| | configured pool (`provider: kubernetes`) | joined cluster |
| - | - | - |
| who calls the cluster API | the controlplane, with a credential you give it | an agent inside, with a Kubernetes service account |
| who reaches the workers | the controlplane dials each worker's public address | the worker dials out; the controlplane rides its tunnel |
| cloud-side setup | service account and key, authorized networks, firewall rule, image mirror, values block, redeploy | one command; mirror the image if the cluster cannot pull ours |
| nodes | public, sized by allocatable | private allowed, sized by what other tenants leave free |
| when controlplane nodes change | rerun the attach script | nothing |

## What the cluster needs

Three things, and only the first involves your cloud's network settings:

| | what | why |
| - | - | - |
| your `kubectl` reaches the cluster API, once | the join applies one manifest through your current context | a cluster with **authorized networks** (GKE) or an equivalent allow-list must include the address you run the command from, exactly as it must for any `kubectl apply`. Nothing else is ever added to that list: the controlplane does not call your API server |
| the cluster can open outbound HTTPS to the controlplane | the agent and every worker dial out and keep the stream open | private nodes are fine as long as they egress through NAT; no inbound rule, no public node address, no load balancer |
| the nodes can pull the worker image | Kubernetes starts the worker pod | mirror it and pass `--worker-image` (with `--pull-secret` if that registry needs one) when the cluster cannot pull ours |

If you cannot reach the API server from your own machine, run `levers9 cluster join … --manifest-only`
and apply the manifest from wherever you do have access, such as a bastion or Cloud Shell; the join
completes when the agent dials in.

> [!NOTE]
>
> Upgrading from a configured pool? The controlplane egress addresses that pool required on your
> authorized networks are no longer used by anything and can come off the list once the agent is up.

## Join

Point your kube context at the cluster, then:

```bash theme={null}
levers9 cluster join my-gke --gpu RTXPro6000
```

What it does, and prints as it goes:

1. Registers the cluster with the controlplane, which mints a token for the agent.
2. Applies one manifest through your kube context: the namespace, a service account with the least
   access the agent needs, the token as a secret, and the agent Deployment.
3. Waits until the agent has dialed in and reported its nodes, and prints them.

From then on a job from your workspace that asks for that GPU type can land on the cluster:

```python theme={null}
@job(gpu="RTXPro6000", image=Blueprint(python_packages=["torch"]))
def train(): ...
```

Options that matter:

| flag | when |
| - | - |
| `--selector key=value` | only some nodes are lent. `key=value` for one label, `key=in (a,b)` for several node pools at once, a bare `key` for "the label exists"; repeat the flag to require several. Default: every node, GPU or not, because CPU-only jobs land here too |
| `--worker-image` | the cluster cannot pull the controlplane's worker image; give it one it can |
| `--pull-secret` | that image needs a pull secret, already created in the namespace |
| `--flavor gke\|eks\|aks` | detected from node labels; set it for a cluster that does not label its nodes the usual way |
| `--manifest-only` | print the manifest instead of applying it, for a cluster you reach another way |

The same flow exists on the dashboard's **Clusters** page: register there, then apply the manifest it
shows, or run the printed command.

## Who a cluster serves

A cluster you attach works for your workspace and nobody else's. Your jobs land there by GPU type, with
no pool selector, and no other workspace's job is ever sent to it. Its workers carry a token that's only
good for your workspace, and the settings they get hold none of the platform's own credentials: images
come through the controlplane, which hands the worker a short-lived link for each image your job uses.
Secrets and the `LEVERS9_TOKEN` your code sees are fetched for each container as it starts, the way they
are on a private pool, and that token is a restricted one.

A **shared** cluster works for every workspace, the way the platform's own GPU cluster does: any job asking
for its card can land there. Only a platform admin can share a cluster, and only a platform admin can
detach a shared one:

```bash theme={null}
levers9 cluster share gke-dammam          # every workspace's jobs can land there
levers9 cluster share gke-dammam --off    # back to the workspace that attached it
```

`levers9 cluster list` shows your own clusters and the shared ones, with a **Serves** column. A shared
cluster someone else attached shows its card, its state and what each node has free, but not its node
names, namespace, image or manifest. Taking a shared cluster back takes effect with the next job. Sharing
one takes effect as new workers start there. Either way, workers already running keep the token and
settings they started with until they wind down, so delete their Jobs in the cluster if that can't wait.

A workspace can attach up to five clusters (`worker.attachedClusters.quotaPerWorkspace` in the
controlplane config; 0 means no limit). Attaching and detaching need your own token: a container's token
can't do either.

## Keeping the agent current

The worker image is pushed to your cluster's registry by the same build that pushes ours, and each worker
Job pulls the tag fresh, so workers are never behind. The **agent** is a Deployment and stays on the image
it started with until something restarts it:

```bash theme={null}
kubectl -n levers9 rollout restart deploy/levers9-agent
```

Nothing in flight is lost — the agent only launches and stops worker Jobs, and the ones already launched keep
running while it restarts. Do it after an upgrade, or whenever `levers9 cluster list` shows an agent that has
been up far longer than the platform's last release.

## What runs in the cluster

Exactly what the manifest contains, nothing more:

| object | scope | why |
| - | - | - |
| ServiceAccount `levers9-agent` | the namespace | the identity the agent runs as; Kubernetes issues and rotates its credential |
| ClusterRole `levers9-agent-observe` | nodes and pods, read-only, cluster-wide | to report the nodes it lends and what other tenants' pods leave free on them |
| Role `levers9-agent-launch` | Jobs and pods in the namespace | to start and stop worker Jobs |
| Secret `levers9-agent` | the namespace | the cluster token, minted by the controlplane at join |
| Deployment `levers9-agent` | the namespace | one small pod; it never listens |

The agent only ever creates Jobs in its own namespace and refuses orders for any other. The worker
tunnel only ever reaches the worker's own containers and refuses any other target. Neither can be used
as a way into the cluster.

Check it yourself against the service account, with `kubectl auth can-i --as`:

| question | answer |
| - | - |
| create Jobs in the levers9 namespace | yes |
| create Jobs in any other namespace | no |
| list nodes and pods anywhere | yes, read-only |
| read Secrets, including its own | no |
| exec into a pod | no |
| delete anything outside the namespace, or grant itself more | no |

Reading pods cluster-wide is what makes shared nodes work: the agent reports the cards other tenants
leave free, so the scheduler never sends a job to a node whose GPUs are already taken.

What the controlplane learns about your cluster is exactly the nodes the selector names: their names,
their cards, and their free CPU and memory. With no selector that is every node in the cluster, node
pools you never meant to lend included, so name them with `--selector` if that matters to you.

## Who talks to whom

The agent and the workers never talk to each other. Each has its own line to the controlplane and
nothing else; the picture is a hub with spokes, and the hub is the controlplane.

| connection | opened by | carries |
| - | - | - |
| agent → controlplane | the agent | node reports up; Job create and delete orders down |
| agent → Kubernetes API | the agent | list nodes and pods; create and delete worker Jobs in its namespace |
| worker → controlplane, gRPC | the worker | registration, heartbeats, container requests, results, image publishing |
| worker → controlplane, tunnel | the worker | container gRPC, log streams, exec and endpoint traffic, carried back down |

```mermaid theme={null}
sequenceDiagram
  participant A as agent (in the cluster)
  participant K as Kubernetes API
  participant C as controlplane
  participant W as worker (in the cluster)
  A->>C: Connect (cluster token)
  loop every 10 s
    A->>K: list nodes, pods
    A->>C: node report: allocatable, free
  end
  Note over C: a job asks for a card the cluster has free
  C->>A: CreateJob (worker Job, serialized)
  A->>K: create Job pinned to the node
  A->>C: JobStatus created
  W->>C: register, heartbeat (worker token)
  W->>C: Tunnel.Connect
  C-->>W: container start, logs, exec over the tunnel
  W->>C: results
```

The agent reads the worker Jobs' outcome only to answer the controlplane; once a Job exists the agent is
out of the picture for it. If the agent pod dies, running workers keep running and keep serving jobs;
what stops is new placement, because the nodes' last report goes stale after ninety seconds. Kubernetes
restarts the agent Deployment on its own, so that gap is short. If a worker dies, its Job ends and the
controlplane asks the agent for a new one when the next job needs the node.

## Failure modes, and what covers each

| what fails | what happens | what covers it |
| - | - | - |
| the balancer's path to the controlplane (a node reclaimed under it) | every stream through it drops at once | the balancer targets the controlplane pod itself, not a node, so node churn never touches a stream |
| the controlplane restarts | every agent and worker stream drops | the controlplane pings each agent every 15 s; an agent that hears nothing for 45 s reconnects, with backoff; worker tunnels reconnect on their own; clusters are reloaded from the database at start |
| a caller's stream drops mid-job (a laptop changing networks, a proxy recycling) | the controlplane can no longer tell a drop from a deliberate exit | it waits two minutes before treating the drop as a cancel; the SDK reattaches by task id and resumes the stream, or receives the result if the task finished meanwhile |
| the agent pod dies | node reports stop | running workers keep serving; placement pauses once the last report is 90 s old; Kubernetes restarts the agent, and its `Recreate` strategy means there are never two |
| the controlplane loses a worker's record (a long outage) | it would place a second worker on the same node | the agent counts worker pods as node usage, so a node with a worker on it offers nothing until that pod is gone |
| a node is removed while a worker runs on it | the worker's containers are lost | the dispatcher retries the task on a fresh container and the caller's stream follows it |
| the worker image cannot be pulled | the worker Job pends | the pending worker times out and is retried; check the Job's pod on the cluster, and set `--worker-image` or `--pull-secret` |
| the cluster token is revoked or the cluster deleted | the agent's connect is refused | it backs off to one attempt every 30 s and logs why; `cluster leave` removes it |
| a node's disk fills up | the kubelet reports `DiskPressure` and evicts the worker, and the job on it is lost | every worker keeps the image cache it leaves on the node bounded and gives it up first when the disk runs low (see [Disk on the nodes](#disk-on-the-nodes)); what it can't free is the jobs' own disk |

## How the nodes are sized

Every ten seconds the agent reports each node's allocatable CPU, memory and GPUs, and what is free net
of the other pods on it. A node shared with another workload offers only what that workload leaves; a
node with no free GPU is not offered at all until one frees up. A worker takes every card the node has
free when it is created and serves several jobs across them. The controlplane always counts its own
workers on a node, pending or running, before adding another, so a node whose worker was created when
fewer cards were free keeps that size until the worker winds down (a few idle minutes after its last
job); the next worker then takes everything free. A job asking for several cards needs one worker with
that many, so it waits for a node with that many free at once.

Two ceilings hold regardless of what the reports say: a node never gets a second worker while one of its
workers is still pending (its pod has not started, so the node report cannot see it), and a cluster pool
never carries more pending workers than it has nodes. The agent enforces the same ceiling on its side
(no more pending worker pods than nodes) and removes a worker Job whose pod has not started within twenty
minutes. Above all pools, attached or not, the scheduler holds a request when a pool already has twenty
workers pending or received more than thirty in a minute (`worker.provisioningGuard` in the controlplane
config; a negative value disables a check). A bug in the accounting above them can cost at most one idle
worker per node, never a flood. How a job uses several cards, one
training across them or many trainings side by side, is on [Train on 1, 2, 4 or 8 cards](https://levers9.mintlify.site/guides/multi-gpu.md).

## Disk on the nodes

Workers keep the images they pull in a cache on the node, at `/var/lib/levers9/images/cache`, so the next
worker there starts without pulling them again. It outlives every worker, so every worker keeps it in check:

* it's pruned to 100 GB, the images used longest ago going first;
* anything nobody has used for seven days is removed, once an hour, even when there's room;
* when less than 20% of the node's disk is free, it gives up what it can until 25% is, whatever its size.

An image that a running worker on the node has mounted is never removed, and neither is anything used in the
last ten minutes. The 20% is there because the kubelet's defaults evict pods at 10% free on the node's disk
and at 15% free on the disk that holds the container images, which on GKE is the same disk. The worker is one
of those pods, and so is the job running in it.

What the cache can't free is the jobs' own disk: a container's scratch space and anything it writes. If a
node still reports `DiskPressure`, the worker's log says `what is left is mounted or in use`, and the fix is
a bigger disk or a smaller job. The limits are `worker.nodeCache` in the controlplane's config
(`imageCacheMaxGB`, `minFreePct`); the controlplane hands them to the workers it starts on your cluster.

## Day two

* **See it.** `levers9 cluster list` for your clusters and the shared ones, `levers9 cluster nodes <name>`
  for a cluster's nodes and free cards; the dashboard's Clusters page shows the same, live.
* **Share it.** `levers9 cluster share <name>` (platform admins) lets every workspace use it; `--off` gives
  it back to the workspace that attached it.
* **Host network.** Workers on attached clusters use the node's network, as they always have. A platform
  admin can turn that off with `worker.attachedClusters.hostNetwork: false`: the workers then run on the
  pod network and reach the controlplane through their tunnel. Try it on one cluster first.
* **A new worker image.** Point the cluster at it with `--worker-image` at join, or edit the Deployment's
  image; the agent creates workers from the image the controlplane names in each Job.
* **Re-apply the agent.** `levers9 cluster manifest <name> | kubectl apply -f -` reinstalls everything but
  the token, which stays in the secret already there.
* **Leave.** `levers9 cluster leave <name>` revokes the token, forgets the pool, and deletes the agent's
  objects from the cluster. A shared cluster can only be detached by a platform admin, since other
  workspaces' jobs run there. Joining again under the same name gives you a cluster that isn't shared.

## Compared with a configured pool

A pool with `provider: kubernetes` in the controlplane's values (see [GPUs in a GKE cluster](https://levers9.mintlify.site/guides/gke-pool.md))
has the controlplane call the cluster's API itself, with a credential you give it, and dial each worker.
It still works and needs nothing inside the cluster, at the price of public nodes, a firewall rule and
allow-lists that follow the controlplane's addresses. Joining with the agent is the default for anything
outside the controlplane's own network.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.