Skip to main content
In the game these are missions 3 to 6: Duel (compare), Level up (train), Boss fight (evaluate), Crown the champion (make live).

​
Compare

Compare & approve shows one moment (a real turn from a call, or a synthetic one) and two replies, A and B. Pick the better one and tick the reasons (dialect, respect, accuracy, progress, family, timing, compliance). The review queue orders moments weakest first; a scenario or a map selection can preload it. Each pick is a preference. Approve also lets a reviewer overrule the judge on a real reply, which feeds the next rescoring.

​
Freeze

Preferences accumulate until you freeze them into a numbered preference version: a TSV on the datasets volume with every pick, the reasons and who made it. Frozen versions are what training reads, so a version is reproducible.

​
What runs on a model

Three roles, set in three places. The judge is the model named under Environment → Judge. The agent is per version: every version starts on the simulator (scripted replies), and Runs on under a version in the Policies tab moves it to the judge’s model with one click, to any vLLM endpoint with its own fields, to an OpenAI or Anthropic model by name (offered when the dashboard pod has OPENAI_API_KEY or ANTHROPIC_API_KEY; otherwise the choice shows the env var it is missing; the key stays on the pod and the version only records the model name), or back. A trained version arrives on its own endpoint, and that endpoint can be released (the cards go back to the pool; the merged weights stay on the models volume) and served again on four cards from the same line. A released version says so and answers nothing: the Duel, Train and Evaluate pick the newest version that can answer, so freeing the cards after a checkpoint never leaves a page asking a dead deployment for replies. The same over the API: POST /api/rl/policies/{id}/release, POST .../serve {"gpu_count": 4} (a job). A version whose endpoint was deleted from Endpoints directly is marked released the next time the list is read. The customer is never a model: once the world has real calls, an auto-pair starts from a moment in a real call, both versions reply, the real agent’s reply is a third candidate, and the judge’s pair is the best against the worst. The world’s header badge names all three. Checkpoints and Evaluate replay real calls too once the world has them (below).

​
Train

Train makes the next numbered version of the agent, and its three cards are named by what they change:
  • Move strategy: which move the agent goes for at each point (greet, give details, confirm, handle the objection, reassure, propose a viewing, offer a slot, close). Q-learning on the simulator, two minutes, no GPU, nothing else needed. It is the same q-learning-online the Engines step offers with knobs and an off-line comparison; this card runs it with default knobs and always makes a version. The wording stays the parent’s.
  • Wording: RLHF, the default and the route to ship: a reward model from the pairs, then PPO against it (lab/rl/train_rlhf.py), on the base model you choose (Qwen3 14B by default; Qwen3 8B, Gemma 3 12B, Llama 3.1 8B and the small Qwen3 sizes are on the list, the gated ones through the HF_TOKEN secret), on the card you choose (an attached RTX PRO 6000 by default, then the cluster’s own pools) and as many of it as you pick (four by default: the whole four-card node, one job or served version at a time). Training is data parallel: every card holds the whole base and trains its share of the rows, gradients averaged each step, so four cards are close to four times the throughput; the trained weights are served on as many. Two fits: the reward model is the base with a LoRA and a scalar head on the last token, fitted with the Bradley-Terry loss, a tenth of the pairs held out and its accuracy reported; then the policy (a second LoRA on the same base) samples replies to the pairs’ real-call openings, the reward model scores them, and PPO’s clipped update per reply token moves the policy toward higher scores with a KL penalty to the base. Advantages are normalised within each batch in place of a learned value head. Merged and served as an endpoint, with the reward adapter saved beside the weights; it asks for 500 pairs. The reward model can learn from the Duel’s picks (what the rubric prefers) or from the outcome labels: every agent turn in the opening of every labelled real call as (call so far, reply, won), fitted with the logistic loss, held out by call, AUC reported. That makes lead-to-order the training signal, as far as recorded calls allow. Over moves the same route exists on the Engines step (reward-model, ppo).
  • Benchmark: DPO on the weights: not a route to ship, the baseline an RLHF version has to beat. The same picks are moved straight into the same base with the preference loss on a LoRA (lab/rl/train_dpo.py), merged and served, checkpointed on the same held-out calls. What the RLHF version gains over this number is what the reward model taught; a DPO version that wins says the reward model is the thing to fix, not the policy. Same base, card and count controls as RLHF; 200 pairs.

​
How each method learns

The picks are the same input to both wording methods; the routes differ. DPO treats the picks as the target and moves the weights toward them directly: one job, no sampling, no second model. RLHF first fits a reward model on the picks so it can score any reply, then lets the policy write replies and pushes it toward higher scores with PPO, held near the base by a KL penalty. It can improve beyond the replies people saw, and its reward model can be fitted on the outcome labels, which no direct route can use; it can also drift toward what the reward model rewards rather than what people picked, it keeps two models in memory, and it needs roughly twice the pairs to fit a reward model that generalises. RLHF with a model-based reward is the route the Arena ships; DPO is kept only as its benchmark, run on the same picks and the same base so the two land on the same checkpoint. Over moves, reward-model and ppo exist on the Engines step too. The Train step shows this flow with the world’s own numbers under How each method learns. Versions are numbered, keep their parent, and record what they learned from. The difference from Engines is the output: Train always ends in a version; Engines ends in a measurement, and a run becomes a version only when you promote it.

​
Weights

The two weight scripts run from a terminal too, on the spec the Train step writes (pack, version, base, gpu, epochs, system, the pairs, and for RLHF reward_source and the outcome examples):
Serve the adapter with vLLM (enable_lora, see LLM serving) and register the endpoint as a policy with provider vllm.

​
A training survives a dashboard restart

The GPU job runs headless on the platform: if the dashboard restarts while it trains (a deploy, a crash), the job keeps running, and the dashboard’s record of it is marked interrupted with the platform task it carries on as. On startup the dashboard adopts it: it waits for the task to finish, takes the trainer’s result, and serves, registers and checkpoints the version exactly as it would have. The Resume button on the interrupted job does the same by hand. A job interrupted before it reached the platform (still packing the pairs) says so and is started again from Train. Under the hood: POST /api/rl/jobs/{id}/resume; the job record carries task_id, the workspace it was started in, and what the finish needs. The task is read as the workspace the job ran in, never as whoever pressed Resume, and its result is read without running anything: the trainers return plain values (numbers, strings, lists), and a result that names anything else is refused and the job says so. If the launcher never found the task, Resume takes one in its body ({"task_id": "..."}), but only a task of that trainer, in the job’s own workspace, started after the job was. The base model can be one from the list or any Hugging Face Hub repo id, written owner/name (no URL, no path). An operator who wants the list enforced sets LEVERS9_TRAIN_BASE_ALLOWLIST to listed, or to the repo ids allowed.

​
Evaluate

Evaluate runs two versions against the same calls, so the only difference is the policy. With real calls in the world those are forty fixed real calls, replayed: each version answers at the call’s first four agent turns given the call so far, the judge scores every answer, and the yardstick at each turn is what the real agent said there and how it scored (the checkpoint says how often the version beat the real agent; the outcome of a replayed call is the call’s own and does not move). Without real calls they are scripted customers on fixed seeds. You get win rate, mean reward, a per-dimension radar, and the regressions: the calls where the new version did worse, each openable as an episode. Evaluate before every promotion. A replayed version has no lead-to-order of its own: a replay cannot change what happened on a recorded call, so the won rate a version causes exists only once it takes live calls (the canary in Release). What Evaluate shows meanwhile is the lead-to-order lens, observational and labelled so: on the world’s real calls, the won rate by band of judge score, with the version’s and its parent’s returns placed on it. A version whose return moves it up a band is worth taking to the canary; it is not yet a won rate. GET /api/rl/packs/<pack>/score-vs-outcome is the same numbers.

​
Policies

Versions lists every policy version with its lineage, provider (mock, vLLM endpoint, OpenAI, Anthropic), evaluation results, and a Make live button. v1 · baseline is the hand-written start every tenant gets; v2 · trained and later come from Train. Only a version that won its evaluation should go live; the button does not stop you, the numbers should.