levers9 cluster join attaches a cluster you already run, GKE, EKS, AKS or your own, as a pool of GPU
workers. It works the way levers9 login works for a person: the cluster side initiates everything.
One agent runs inside the cluster, holds an outbound stream to the controlplane, reports the nodes and
what they have free, and starts worker Jobs when asked. The workers carry their own inbound connections
back over the same kind of stream. So there is no public node address, no firewall rule, no allow-list
of controlplane addresses, and the controlplane holds no credentials for your cluster.
The shape
Both arrows are opened from inside your cluster and stay open; nothing on the controlplane side ever dials the cluster or its nodes. That is the whole difference from a configured pool:What the cluster needs
Three things, and only the first involves your cloud’s network settings:
If you cannot reach the API server from your own machine, run
levers9 cluster join … --manifest-only
and apply the manifest from wherever you do have access, such as a bastion or Cloud Shell; the join
completes when the agent dials in.
Upgrading from a configured pool? The controlplane egress addresses that pool required on your
authorized networks are no longer used by anything and can come off the list once the agent is up.
Join
Point your kube context at the cluster, then:- Registers the cluster with the controlplane, which mints a token for the agent.
- Applies one manifest through your kube context: the namespace, a service account with the least access the agent needs, the token as a secret, and the agent Deployment.
- Waits until the agent has dialed in and reported its nodes, and prints them.
The same flow exists on the dashboard’s Clusters page: register there, then apply the manifest it
shows, or run the printed command.
Who a cluster serves
A cluster you attach works for your workspace and nobody else’s. Your jobs land there by GPU type, with no pool selector, and no other workspace’s job is ever sent to it. Its workers carry a token that’s only good for your workspace, and the settings they get hold none of the platform’s own credentials: images come through the controlplane, which hands the worker a short-lived link for each image your job uses. Secrets and theLEVERS9_TOKEN your code sees are fetched for each container as it starts, the way they
are on a private pool, and that token is a restricted one.
A shared cluster works for every workspace, the way the platform’s own GPU cluster does: any job asking
for its card can land there. Only a platform admin can share a cluster, and only a platform admin can
detach a shared one:
levers9 cluster list shows your own clusters and the shared ones, with a Serves column. A shared
cluster someone else attached shows its card, its state and what each node has free, but not its node
names, namespace, image or manifest. Taking a shared cluster back takes effect with the next job. Sharing
one takes effect as new workers start there. Either way, workers already running keep the token and
settings they started with until they wind down, so delete their Jobs in the cluster if that can’t wait.
A workspace can attach up to five clusters (worker.attachedClusters.quotaPerWorkspace in the
controlplane config; 0 means no limit). Attaching and detaching need your own token: a container’s token
can’t do either.
Keeping the agent current
The worker image is pushed to your cluster’s registry by the same build that pushes ours, and each worker Job pulls the tag fresh, so workers are never behind. The agent is a Deployment and stays on the image it started with until something restarts it:levers9 cluster list shows an agent that has
been up far longer than the platform’s last release.
What runs in the cluster
Exactly what the manifest contains, nothing more:
The agent only ever creates Jobs in its own namespace and refuses orders for any other. The worker
tunnel only ever reaches the worker’s own containers and refuses any other target. Neither can be used
as a way into the cluster.
Check it yourself against the service account, with
kubectl auth can-i --as:
Reading pods cluster-wide is what makes shared nodes work: the agent reports the cards other tenants
leave free, so the scheduler never sends a job to a node whose GPUs are already taken.
What the controlplane learns about your cluster is exactly the nodes the selector names: their names,
their cards, and their free CPU and memory. With no selector that is every node in the cluster, node
pools you never meant to lend included, so name them with
--selector if that matters to you.
Who talks to whom
The agent and the workers never talk to each other. Each has its own line to the controlplane and nothing else; the picture is a hub with spokes, and the hub is the controlplane.
The agent reads the worker Jobs’ outcome only to answer the controlplane; once a Job exists the agent is
out of the picture for it. If the agent pod dies, running workers keep running and keep serving jobs;
what stops is new placement, because the nodes’ last report goes stale after ninety seconds. Kubernetes
restarts the agent Deployment on its own, so that gap is short. If a worker dies, its Job ends and the
controlplane asks the agent for a new one when the next job needs the node.
Failure modes, and what covers each
How the nodes are sized
Every ten seconds the agent reports each node’s allocatable CPU, memory and GPUs, and what is free net of the other pods on it. A node shared with another workload offers only what that workload leaves; a node with no free GPU is not offered at all until one frees up. A worker takes every card the node has free when it is created and serves several jobs across them. The controlplane always counts its own workers on a node, pending or running, before adding another, so a node whose worker was created when fewer cards were free keeps that size until the worker winds down (a few idle minutes after its last job); the next worker then takes everything free. A job asking for several cards needs one worker with that many, so it waits for a node with that many free at once. Two ceilings hold regardless of what the reports say: a node never gets a second worker while one of its workers is still pending (its pod has not started, so the node report cannot see it), and a cluster pool never carries more pending workers than it has nodes. The agent enforces the same ceiling on its side (no more pending worker pods than nodes) and removes a worker Job whose pod has not started within twenty minutes. Above all pools, attached or not, the scheduler holds a request when a pool already has twenty workers pending or received more than thirty in a minute (worker.provisioningGuard in the controlplane
config; a negative value disables a check). A bug in the accounting above them can cost at most one idle
worker per node, never a flood. How a job uses several cards, one
training across them or many trainings side by side, is on Train on 1, 2, 4 or 8 cards.
Disk on the nodes
Workers keep the images they pull in a cache on the node, at/var/lib/levers9/images/cache, so the next
worker there starts without pulling them again. It outlives every worker, so every worker keeps it in check:
- it’s pruned to 100 GB, the images used longest ago going first;
- anything nobody has used for seven days is removed, once an hour, even when there’s room;
- when less than 20% of the node’s disk is free, it gives up what it can until 25% is, whatever its size.
DiskPressure, the worker’s log says what is left is mounted or in use, and the fix is
a bigger disk or a smaller job. The limits are worker.nodeCache in the controlplane’s config
(imageCacheMaxGB, minFreePct); the controlplane hands them to the workers it starts on your cluster.
Day two
- See it.
levers9 cluster listfor your clusters and the shared ones,levers9 cluster nodes <name>for a cluster’s nodes and free cards; the dashboard’s Clusters page shows the same, live. - Share it.
levers9 cluster share <name>(platform admins) lets every workspace use it;--offgives it back to the workspace that attached it. - Host network. Workers on attached clusters use the node’s network, as they always have. A platform
admin can turn that off with
worker.attachedClusters.hostNetwork: false: the workers then run on the pod network and reach the controlplane through their tunnel. Try it on one cluster first. - A new worker image. Point the cluster at it with
--worker-imageat join, or edit the Deployment’s image; the agent creates workers from the image the controlplane names in each Job. - Re-apply the agent.
levers9 cluster manifest <name> | kubectl apply -f -reinstalls everything but the token, which stays in the secret already there. - Leave.
levers9 cluster leave <name>revokes the token, forgets the pool, and deletes the agent’s objects from the cluster. A shared cluster can only be detached by a platform admin, since other workspaces’ jobs run there. Joining again under the same name gives you a cluster that isn’t shared.
Compared with a configured pool
A pool withprovider: kubernetes in the controlplane’s values (see GPUs in a GKE cluster)
has the controlplane call the cluster’s API itself, with a credential you give it, and dial each worker.
It still works and needs nothing inside the cluster, at the price of public nodes, a firewall rule and
allow-lists that follow the controlplane’s addresses. Joining with the agent is the default for anything
outside the controlplane’s own network.