> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Train on 1, 2, 4 or 8 cards

> gpu_count is the only knob: the same training function runs on one card, on every card of a node, or as many single-card runs side by side

A job asks for cards with `gpu_count`. What you do with them is one of two shapes, and both are one line.

| shape | when | how |
| - | - | - |
| **many trainings, one card each** | comparing settings, seeds, datasets | `gpu_count=1` and `fn.map([...])`: each cell is its own job on its own card, all at once |
| **one training, many cards** | one run has to finish sooner | `gpu_count=2 / 4 / 8` and `distributed.run(fn)`: one process per card, gradients averaged |

## Many trainings, one card each

```python theme={null}
@job(gpu="RTXPro6000", gpu_count=1, ...)
def cell(lora_r: int, lr: float) -> dict:
    return finetune(lora_r=lora_r, lr=lr)

results = list(cell.map([(8, 1e-4), (16, 1e-4), (32, 1e-4), (64, 2e-4)]))
```

Four cells, four cards, one call; the wall clock is one cell's. Nothing here talks to NCCL, which is why
this is the shape to reach for first: on a cluster of RTX PRO 6000s a four-cell sweep finished in 92 s
against 73 to 78 s for any single cell.

## One training, many cards

```python theme={null}
from levers9 import distributed, job

@job(gpu="RTXPro6000", gpu_count=4, ...)
def train(steps: int = 200) -> dict:
    return distributed.run(train_one_rank, {"steps": steps})
```

`distributed.run` starts one process per card the container was given, gives each its rank, joins them
into a process group, calls your function on every rank and returns what rank 0 returned. Change
`gpu_count` to 2 or 8 and nothing else changes. With `gpu_count=1` there is nothing to spawn: the function
runs in place and every helper answers as a world of one, so one function covers 1x, 2x, 4x and 8x.

Inside the function, the helpers are the handful a loop needs:

```python theme={null}
from levers9 import distributed as dist

def train_one_rank(cfg: dict) -> dict:
    dev = dist.device()                      # this rank's card
    model = load_model().to(dev)
    model = dist.wrap(model)                 # DistributedDataParallel, or the model itself on one card
    for step in range(cfg["steps"]):
        audio, text = rows[(step * dist.world_size() + dist.rank()) % len(rows)]   # each rank its own clip
        loss = model(**batch(audio, text)).loss
        loss.backward(); opt.step(); opt.zero_grad()
        dist.print0(f"step {step} loss={dist.all_reduce_mean(loss).item():.4f}")  # rank 0 speaks, mean across cards
    dist.barrier()
    if not dist.is_main():
        return {}
    return evaluate_and_save(model)          # rank 0's return is what the job returns
```

`dist.shard(rows)` gives a rank its slice of a list when a static split is simpler than indexing.

The function must be a module-level function (ranks are separate processes and import it by name), and
everything it needs goes in through its arguments.

> [!NOTE]
>
> A step trains `world_size` examples instead of one, so an effective batch of 32 on four cards is 8 per
> card; adjust gradient accumulation accordingly if a run is meant to be comparable to a single-card one.

### torchrun and accelerate

Under `torchrun` or `accelerate launch` the process is already a rank. `distributed.run` notices, joins the
group and calls the function; the helpers read the same `RANK` / `WORLD_SIZE` / `LOCAL_RANK` those launchers
set. Code written for them keeps working.

## What the cards need

Cards have quirks, and they live in `levers9.distributed` rather than in every training script.

| card | quirk | what the SDK does |
| - | - | - |
| RTX PRO 6000 (Blackwell) | peer-to-peer transport hangs | sets `NCCL_P2P_DISABLE=1`, a few percent slower through shared memory |
| RTX PRO 6000 (Blackwell) | NCCL before 2.26 has no kernels for it; the first collective dies with `Cuda failure 1 'invalid argument'` after the rings connect | checks the NCCL version before the first collective and says which package to add, instead of failing inside DDP |

Anything you set yourself wins over these defaults.

## Sizing, measured

On four RTX PRO 6000s in one node, the same 10,170-step fine-tune:

| ranks | wall clock | speedup |
| - | - | - |
| 1 | 4 h 10 min | 1.0x |
| 2 | 2 h 16 min | 1.85x |
| 4 | 1 h 19 min | 3.15x |

3.15x on four cards is what NCCL through shared memory costs. Four single-card cells, when the work is
independent, get the full 4x, which is why the sweep shape comes first above.

## What a multi-card job waits for

A multi-card job needs one worker holding that many cards, so it waits for a node with that many free at
the same moment. On an attached cluster the agent reports what each node has free; a node whose other
cards belong to another deployment offers only what is left. To borrow a whole node, scale the neighbour
down and the job starts as soon as the cards show as free.

Two limits gate more than one card per job: the controlplane's per-stub cap (`stubLimits.maxGpuCount`,
raised to 8 on the hosted deployment) and the workspace's own switch:

```bash theme={null}
levers9 workspace multi-gpu          # on or off?
levers9 workspace multi-gpu on       # allow gpu_count > 1 here
```

It is a workspace setting because several cards in one container is a whole node held by one call. It takes
effect on the next call: the change clears the cached copy of the workspace that tokens carry.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.