> ## Documentation Index > Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Compare, train, evaluate > Turning picks into a version, proving it on the same customers, and making it live > [!NOTE] > > In the game these are missions 3 to 6: **Duel** (compare), **Level up** (train), **Boss fight** (evaluate), **Crown the champion** (make live). ## Compare **Compare & approve** shows one moment (a real turn from a call, or a synthetic one) and two replies, A and B. Pick the better one and tick the reasons (dialect, respect, accuracy, progress, family, timing, compliance). The review queue orders moments weakest first; a scenario or a map selection can preload it. Each pick is a *preference*. Approve also lets a reviewer overrule the judge on a real reply, which feeds the next rescoring. ## Freeze Preferences accumulate until you **freeze** them into a numbered preference version: a TSV on the `datasets` volume with every pick, the reasons and who made it. Frozen versions are what training reads, so a version is reproducible. ## What runs on a model Three roles, set in three places. The **judge** is the model named under Environment → Judge. The **agent** is per version: every version starts on the simulator (scripted replies), and **Runs on** under a version in the Policies tab moves it to the judge's model with one click, to any vLLM endpoint with its own fields, to an OpenAI or Anthropic model by name (offered when the dashboard pod has `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`; otherwise the choice shows the env var it is missing; the key stays on the pod and the version only records the model name), or back. A trained version arrives on its own endpoint, and that endpoint can be **released** (the cards go back to the pool; the merged weights stay on the models volume) and **served again** on four cards from the same line. A released version says so and answers nothing: the Duel, Train and Evaluate pick the newest version that can answer, so freeing the cards after a checkpoint never leaves a page asking a dead deployment for replies. The same over the API: `POST /api/rl/policies/{id}/release`, `POST .../serve {"gpu_count": 4}` (a job). A version whose endpoint was deleted from Endpoints directly is marked released the next time the list is read. The **customer** is never a model: once the world has real calls, an auto-pair starts from a moment in a real call, both versions reply, the real agent's reply is a third candidate, and the judge's pair is the best against the worst. The world's header badge names all three. Checkpoints and Evaluate replay real calls too once the world has them (below). ## Train **Train** makes the next numbered version of the agent, and its three cards are named by what they change: * **Move strategy**: *which move* the agent goes for at each point (greet, give details, confirm, handle the objection, reassure, propose a viewing, offer a slot, close). Q-learning on the simulator, two minutes, no GPU, nothing else needed. It is the same `q-learning-online` the [Engines](https://levers9.mintlify.site/lab/your-own-engine.md) step offers with knobs and an off-line comparison; this card runs it with default knobs and always makes a version. The wording stays the parent's. * **Wording: RLHF**, the default and the route to ship: a reward model from the pairs, then PPO against it (`lab/rl/train_rlhf.py`), on the base model you choose (Qwen3 14B by default; Qwen3 8B, Gemma 3 12B, Llama 3.1 8B and the small Qwen3 sizes are on the list, the gated ones through the `HF_TOKEN` secret), on the card you choose (an attached RTX PRO 6000 by default, then the cluster's own pools) and as many of it as you pick (four by default: the whole four-card node, one job or served version at a time). Training is data parallel: every card holds the whole base and trains its share of the rows, gradients averaged each step, so four cards are close to four times the throughput; the trained weights are served on as many. Two fits: the reward model is the base with a LoRA and a scalar head on the last token, fitted with the Bradley-Terry loss, a tenth of the pairs held out and its accuracy reported; then the policy (a second LoRA on the same base) samples replies to the pairs' real-call openings, the reward model scores them, and PPO's clipped update per reply token moves the policy toward higher scores with a KL penalty to the base. Advantages are normalised within each batch in place of a learned value head. Merged and served as an endpoint, with the reward adapter saved beside the weights; it asks for 500 pairs. The reward model can learn from the Duel's picks (what the rubric prefers) or from **the outcome labels**: every agent turn in the opening of every labelled real call as (call so far, reply, won), fitted with the logistic loss, held out by call, AUC reported. That makes lead-to-order the training signal, as far as recorded calls allow. Over moves the same route exists on the Engines step (`reward-model`, `ppo`). * **Benchmark: DPO on the weights**: not a route to ship, the baseline an RLHF version has to beat. The same picks are moved straight into the same base with the preference loss on a LoRA (`lab/rl/train_dpo.py`), merged and served, checkpointed on the same held-out calls. What the RLHF version gains over this number is what the reward model taught; a DPO version that wins says the reward model is the thing to fix, not the policy. Same base, card and count controls as RLHF; 200 pairs. ### How each method learns ```mermaid theme={null} flowchart LR subgraph inputs [this world] P[parent version: prompt] K[culture pack: personas, rubric, guardrails] J[judge: scores a reply 0-100] D[Duel picks: call so far + chosen reply + rejected reply] B[base model + one GPU] end subgraph strategy [Move strategy · no GPU] S1[600 simulated calls, 8 moves per turn] --> S2[judge scores each turn] --> S3["Q(state, move)"] end P --> S1 K --> S1 J --> S2 S3 --> V1[version: parent prompt + move table] subgraph dpo [Benchmark: DPO] D1["LoRA on the base: make chosen likelier than rejected, −log σ(β·margin)"] --> D2[merge, serve as endpoint] end D --> D1 B --> D1 D2 --> V2[version: provider = the endpoint] subgraph rlhf [Wording: RLHF] R1[fit a reward model on the pairs] --> R2[policy writes replies, reward model scores them] --> R3[PPO raises the score, KL keeps it near the base] end D --> R1 B --> R2 R3 --> V3[version: provider = the endpoint] ``` The picks are the same input to both wording methods; the routes differ. **DPO** treats the picks as the target and moves the weights toward them directly: one job, no sampling, no second model. **RLHF** first fits a reward model on the picks so it can score *any* reply, then lets the policy write replies and pushes it toward higher scores with PPO, held near the base by a KL penalty. It can improve beyond the replies people saw, and its reward model can be fitted on the outcome labels, which no direct route can use; it can also drift toward what the reward model rewards rather than what people picked, it keeps two models in memory, and it needs roughly twice the pairs to fit a reward model that generalises. RLHF with a model-based reward is the route the Arena ships; DPO is kept only as its benchmark, run on the same picks and the same base so the two land on the same checkpoint. Over moves, `reward-model` and `ppo` exist on the Engines step too. The Train step shows this flow with the world's own numbers under **How each method learns**. Versions are numbered, keep their parent, and record what they learned from. The difference from Engines is the output: Train always ends in a version; Engines ends in a measurement, and a run becomes a version only when you promote it. ### Weights The two weight scripts run from a terminal too, on the spec the Train step writes (pack, version, base, gpu, epochs, system, the pairs, and for RLHF `reward_source` and the outcome examples): ```bash theme={null} cd lab/rl && python train_rlhf.py --spec spec.json # the route to ship cd lab/rl && python train_dpo.py --spec spec.json # the benchmark, same pairs and base ``` Serve the adapter with vLLM (`enable_lora`, see [LLM serving](https://levers9.mintlify.site/sdk/llm-serving.md)) and register the endpoint as a policy with provider `vllm`. ### A training survives a dashboard restart The GPU job runs headless on the platform: if the dashboard restarts while it trains (a deploy, a crash), the job keeps running, and the dashboard's record of it is marked interrupted with the platform task it carries on as. On startup the dashboard adopts it: it waits for the task to finish, takes the trainer's result, and serves, registers and checkpoints the version exactly as it would have. The **Resume** button on the interrupted job does the same by hand. A job interrupted before it reached the platform (still packing the pairs) says so and is started again from Train. Under the hood: `POST /api/rl/jobs/{id}/resume`; the job record carries `task_id`, the workspace it was started in, and what the finish needs. The task is read as the workspace the job ran in, never as whoever pressed Resume, and its result is read without running anything: the trainers return plain values (numbers, strings, lists), and a result that names anything else is refused and the job says so. If the launcher never found the task, Resume takes one in its body (`{"task_id": "..."}`), but only a task of that trainer, in the job's own workspace, started after the job was. The base model can be one from the list or any Hugging Face Hub repo id, written `owner/name` (no URL, no path). An operator who wants the list enforced sets `LEVERS9_TRAIN_BASE_ALLOWLIST` to `listed`, or to the repo ids allowed. ## Evaluate **Evaluate** runs two versions against the same calls, so the only difference is the policy. With real calls in the world those are forty fixed real calls, replayed: each version answers at the call's first four agent turns given the call so far, the judge scores every answer, and the yardstick at each turn is what the real agent said there and how it scored (the checkpoint says how often the version beat the real agent; the outcome of a replayed call is the call's own and does not move). Without real calls they are scripted customers on fixed seeds. You get win rate, mean reward, a per-dimension radar, and the regressions: the calls where the new version did worse, each openable as an episode. Evaluate before every promotion. A replayed version has no lead-to-order of its own: a replay cannot change what happened on a recorded call, so the won rate a version causes exists only once it takes live calls (the canary in Release). What Evaluate shows meanwhile is the **lead-to-order lens**, observational and labelled so: on the world's real calls, the won rate by band of judge score, with the version's and its parent's returns placed on it. A version whose return moves it up a band is worth taking to the canary; it is not yet a won rate. `GET /api/rl/packs//score-vs-outcome` is the same numbers. ## Policies **Versions** lists every policy version with its lineage, provider (mock, vLLM endpoint, OpenAI, Anthropic), evaluation results, and a **Make live** button. **v1 · baseline** is the hand-written start every tenant gets; **v2 · trained** and later come from Train. Only a version that won its evaluation should go live; the button does not stop you, the numbers should. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.