> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU pipeline

> Demo: prep on CPU, train ×4 on L40S in parallel, evaluate on L40S, package on CPU

Source: `demo/gpu_pipeline/app.py` in the repository.

The same shape as the [pipeline demo](https://levers9.mintlify.site/demos/pipeline.md), with the training and evaluation stages asking for an
L40S (`gpu="L40S"`). Those containers land on the GPU pool while `prep` and `package` stay on cheap CPU workers.
Artifacts move between stages on the `gpu-pipeline-artifacts` volume.

```bash theme={null}
cd demo/gpu_pipeline
python app.py
```

What you see while it runs: every container's prints stream back to the terminal, so each training container
reports its epochs as it goes, and the orchestrator prints a summary and a leaderboard at the end.

```
[wide] epoch  24/40  train=0.0108  val=0.0117 *  lr=3.45e-04  776k samples/s  103ms mem=62MiB
[deep] done: best val=0.0109 at epoch 38, 40 epochs in 11.8s, gpu util=11% temp=31C
      all 4 trainings finished in 41.9s wall-clock
      model  gpu          epochs  best_epoch  val_loss  samples/s  peak_mem  time
      small  NVIDIA L40S  40      38          0.01069   429,013    26 MiB    8.1s
      wide   NVIDIA L40S  40      40          0.01116   736,210    62 MiB    5.5s
      leaderboard on 4 models (baseline mse 7.32203):
      rank  name   mse      mae      r2      infer_ms  params
      1     small  0.01077  0.08209  0.9985  0.41      5313
```

Afterwards: `levers9 ls gpu-pipeline-artifacts` lists the models, per-epoch histories, metrics and the release
tarball.

Notes

* The first run builds the torch blueprint (about ten minutes, once). Later runs start in seconds.
* With four configs and three GPU nodes, three trainings run at once and the fourth queues; `levers9 worker list`
  shows the L40S workers.
* If a Spot node is reclaimed mid-run, the task is retried on another worker and `.map()` still returns all results.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.