GPU pipeline

Source: demo/gpu_pipeline/app.py in the repository. The same shape as the pipeline demo, with the training and evaluation stages asking for an L40S (gpu="L40S"). Those containers land on the GPU pool while prep and package stay on cheap CPU workers. Artifacts move between stages on the gpu-pipeline-artifacts volume.

cd demo/gpu_pipeline
python app.py

What you see while it runs: every container’s prints stream back to the terminal, so each training container reports its epochs as it goes, and the orchestrator prints a summary and a leaderboard at the end.

[wide] epoch  24/40  train=0.0108  val=0.0117 *  lr=3.45e-04  776k samples/s  103ms mem=62MiB
[deep] done: best val=0.0109 at epoch 38, 40 epochs in 11.8s, gpu util=11% temp=31C
      all 4 trainings finished in 41.9s wall-clock
      model  gpu          epochs  best_epoch  val_loss  samples/s  peak_mem  time
      small  NVIDIA L40S  40      38          0.01069   429,013    26 MiB    8.1s
      wide   NVIDIA L40S  40      40          0.01116   736,210    62 MiB    5.5s
      leaderboard on 4 models (baseline mse 7.32203):
      rank  name   mse      mae      r2      infer_ms  params
      1     small  0.01077  0.08209  0.9985  0.41      5313

Afterwards: levers9 ls gpu-pipeline-artifacts lists the models, per-epoch histories, metrics and the release tarball. Notes

The first run builds the torch blueprint (about ten minutes, once). Later runs start in seconds.
With four configs and three GPU nodes, three trainings run at once and the fourth queues; levers9 worker list shows the L40S workers.
If a Spot node is reclaimed mid-run, the task is retried on another worker and .map() still returns all results.

Get started

Build with the SDK

Dashboard

Data Lab

RL Arena

Operate