> ## Documentation Index > Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # RL Arena > Listen, teach, ship: post-training a call agent with human feedback > [!WARNING] > > The Arena is a working preview. The game loop, the scoring and the reply API are real; the learning underneath is > prompt-level while the maths of the reinforcement-learning step (the reward model and the policy update) is being > worked out. Use the numbers to explore the loop, not to make decisions yet. ## Worlds: one per tenant The Arena opens on a list of **worlds**. A world is one tenant's arena: its real calls, an agent with a job, customers who answer, a reward that scores every reply, and its own versions of the agent. Where a culture pack has tenants, each tenant is its own world, so a property-sales tenant and a rent-split qualification tenant have separate calls, versions, scores and objectives rather than being two tabs of one arena. **Create a world** makes another, cloning an existing pack's rubric, guardrails and personas so it can run an episode immediately, or starting from a generic rubric you then edit under Environment. Inside a world, three features. The **Arena path** is the loop: calls in, hot spots, duel, train, evaluate. **Environment** is the reward and the world's dynamics: the rubric and guardrails the judge scores with, the objective, the judge, the outcomes map, and the simulated customers. **Policies** is every version of the agent: where it came from, its checkpoint, what it runs on, what v1 starts from, and where one is written by hand. In RL terms the Environment is r(s, a) and the Policies are π(a | s); the API still calls a world an `environment` (`/api/rl/environments`). Every card names the four parts that make it a decision process before anyone plays: | part | what it is here | | - | - | | **Agent** | the policy: the prompt (and later the weights) that writes the next reply | | **State** | the dialogue and the stage, which the agent sees; trust, the drawn objection and the persona, which it does not | | **Action** | one reply in the pack's language, constrained by the guardrails | | **Reward** | the judge's weighted rubric score, 0 to 100 per agent turn | ## The model behind a world **Model** on an environment writes the decision process down and lets you change it. It is a finite-horizon, partially observed process: the agent sees the conversation, while trust, the objection and the persona are hidden, which is what makes it a POMDP rather than a plain MDP. * **Transition**: the customer answers from the persona and the stage, then trust moves with the reply just scored, `trust ← clip(trust + (r − 50)/100 × 0.6, 0, 1)`. The stage advances one step. Seeded, so two versions meet the same customers. * **Horizon `H`**: agent turns per episode, four by default. * **Discount `γ`**: 1.0 by default, which is well defined because the horizon is finite. * **Return `G = Σ γᵗ·rₜ + γᴴ·terminal − c·violations`**: the one number an optimiser would maximise. The terminal reward is what the call was worth in the end, the part a per-turn rubric never captures: a reply can score well and still lose the customer. The violation penalty `c` is charged per guardrail breach. Changing the objective recomputes the return of existing episodes; their per-turn rewards are untouched. > [!NOTE] > > The **decision engines** below learn exactly this return. What ships to a live agent today is still the > preference-based step, a contextual bandit rather than a policy gradient, because the engines are not yet > reachable from the Arena's pages. So there is still no credit assignment across turns in production: a reply > that sets up a booking two turns later gets no credit for it. The return above is what a value function is > trained to predict, which is why it is defined first. ## Decision engines An engine takes what happened and returns a move for every state: `stage:trust:objection` in, one of the eight moves out. That mapping is the decision engine, in the form the Arena can render, compare and promote. Fourteen are registered, across three kinds of evidence. **Which kind** matters more than which algorithm, because it decides whether an engine can run at all on a given pack: | Learns from | Engines | What that means | | - | - | - | | Stored episodes | Q-learning, SARSA, Monte Carlo, TD(λ), value iteration, policy iteration | Offline and cheap. Cannot try a move nobody ever tried | | Preference pairs | DPO, Bradley-Terry reward model | Offline. A pair means one move beat another from a state, decided by judged return | | A live simulator | REINFORCE, PPO, online Q-learning, online SARSA, model-based, R-max | Explores, and costs one real episode per simulated call | An engine that is handed evidence it cannot use refuses before it starts, saying which kind it needed and how to get some: rather than returning an empty result or failing somewhere inside an algorithm. > [!NOTE] > > Two engines re-solve the whole model every time they learn something: **R-max** and **model-based learning**. > That is correct on the small gridworlds they were written for and quadratic in states here, where a pack has > 176\. Both estimate their cost first and refuse with the number and the knob that would change it, because an > engine that cannot finish must not look like one still running. R-max also takes its own discount, since its > optimistic value is `R_max/(1−γ)` and the Arena's γ is deliberately 1. ### Preferences, at two levels The Arena already learns from human preferences, and these engines do too, and they are **not the same preferences**. A person picking between two replies is choosing text, and those pairs ship to a GPU job for a real DPO weight update on the language model. The engines here work one level up, over the eight discrete moves, where a preference means one move beat another from a state and the evidence is the judged return. Nothing in a stored human preference records which move produced the winning reply, so the two are kept apart rather than merged: the move-level engine decides what to do, the text-level DPO decides how to say it. ### Adding one An engine is one file. It declares what it learns from and its hyperparameters *as data*, then implements `fit`. Because the parameters are data rather than a function signature, the API lists them and a form can be drawn for an engine nothing has heard of, so a new algorithm needs no change to the API or the web app. ## The loop The RL Arena is where the call agent gets better, and it is played rather than administered: reinforcement learning from human feedback as seven missions. Open **Data Lab → RL Arena** in the dashboard; the **Play** tab is the game board and its Home guide ticks the missions off as you go. A longer walkthrough for a first-time reader is in the repository at `docs/rl-tutorial.md`. ## The seven missions, and the RL behind each | # | in the game | what you do | in RL terms | | - | - | - | - | | 1 | Bring calls in | real conversations arrive from your call platform, a dataset or the demo corpus; every reply is scored | episodes and a reward signal from the real environment | | 2 | Find the hot spots | thousands of moments are grouped into a few dozen situations, weakest first, on a map | clustering states, prioritising where the policy is weak | | 3 | Duel | one moment, two hidden and shuffled replies; pick the better, tick why, +10 XP | pairwise human preferences, the training signal of RLHF | | 4 | Level up (**train**) | your picks become a new, numbered version of the agent | a policy update: prompt-level now, DPO with LoRA for weights | | 5 | Boss fight (**evaluate**) | new vs old on the same customers and seeds: win rate, radar, regressions | held-out evaluation with fixed seeds | | 6 | Crown the champion (**deploy**) | only a version that won goes live; this is the deploy step, there is no separate button | promotion gate | | 7 | Connect (**serve**) | your call platform asks the champion for its next reply; every reply comes back as a live call | the deployed policy, with online data collection | Every mission card shows its plain name under the game name (train, evaluate, deploy, serve), and clicking the card runs it: training and deploying are missions 4 and 6 on the **Play** tab, not separate screens. Level is the agent's version number, XP is ten per pick, "in a row" counts picks without leaving the Duel (the flame lights at 3, roars at 5 and 10, with a banner at each), the day streak counts days in a row with at least one pick (it breaks after a day without one, so the hub nudges you to duel today), and a rule violation is a reply that used a phrase the culture pack forbids. ## The loop 1. **Listen.** Real calls arrive through a [webhook, a dataset or paste](https://levers9.mintlify.site/lab/call-logs.md). Every agent reply is scored on the market's rubric by a judge, and thousands of moments are grouped into a few dozen [scenarios](https://levers9.mintlify.site/lab/scenarios.md) so you see where the agent struggles. 2. **Teach.** For one moment, two drafts side by side. Pick the better one and tick why. Twenty picks take ten minutes; each is a lesson. See [Compare & approve](https://levers9.mintlify.site/lab/train-and-evaluate.md#compare). 3. **Ship.** [Train](https://levers9.mintlify.site/lab/train-and-evaluate.md#train) turns the picks into a new version. [Evaluate](https://levers9.mintlify.site/lab/train-and-evaluate.md#evaluate) runs it against the old one on the same customers. Make it live only when it wins. ## The tabs The Arena is played, not administered. **Play** shows the agent's level (its version), XP (one pick is 10 XP), a streak, and six missions in order with a ring each; the big Play button starts whichever mission is next. | tab | what it is for | | - | - | | Play | level, XP, the six missions: bring calls in, find the hot spots, duel, level up, boss fight, crown the champion | | Calls | real calls in, each turn scored; webhook setup | | Hot spots | the map of situations across all calls, ranked by where the agent is weakest | | Duel | one moment, two replies, pick the better one with 1 / 2 or the arrow keys, tick why, Enter for the next. Opened from a hot spot or a call it works through that set; opened on its own it works through the review queue, the forty weakest real replies from the top hot spots. The header says which moment of how many, from which call and turn and hot spot; Up next lists them and jumps to any. Simulated customers only while a world has no calls | | Boss fight | the new version against the old one on the same customers | | Arena (advanced) | simulated customers talk to a policy; watch, fork, replay | | Versions (advanced) | policy versions with lineage; make one live | | Culture pack (advanced) | the rubric, guardrails and personas for this market | | Compare (classic), Details (advanced) | the earlier form-based Compare tab and the numbers-first Home | ## Connect: the inference layer Mission 7 opens the **Connect** dialog: the endpoint, the key, an example, what the champion has served (replies, average score, violations), and a box to talk to the live agent as a customer. The call platform posts the conversation so far and gets the crowned version's next reply, judged and rule-checked: ```bash theme={null} curl -X POST https://levers9-api.uselevers.com/api/rl/agent//reply \ -H "X-Levers-Webhook-Key: " -H "Content-Type: application/json" \ -d '{"call_id": "CALL-0142", "turns": [{"role": "customer", "text": "..."}]}' # {"reply": "...", "policy": {"name": "v2 · trained", "version": 2}, "judge": {"total": 76, ...}, "violations": [], "latency_ms": 1400} ``` Every exchange is filed as a live call (source `live`) under the tenant, so production traffic feeds Calls, Hot spots and the next Duel. `GET /api/rl/agent//stats` returns what has been served, by version. Send the key in the `X-Levers-Webhook-Key` header, never in the URL. A key in the URL (`?key=`) ends up in load balancer and proxy logs. It still works for now, and every request that uses it is logged (without the key) so the dashboard's operator can find the sender; once nobody uses it, `LEVERS9_RL_WEBHOOK_QUERY_KEY=false` turns it off. A key that has been in a URL can be replaced with `POST /api/rl/webhook-info/rotate` `{"pack": ""}`, by an admin of the workspace signed in to the dashboard. The old key stops working at once, so tell the call platforms that use it first. ## Who can use the Arena Everything in the Arena (its calls, versions, judge, webhook keys, the Decision API, Models and Compare) lives in one workspace. When the dashboard's operator sets `LEVERS9_ARENA_WORKSPACES` to that workspace's id, only its members get the Arena: anyone else gets a 403 that names the workspace to switch to, the Lab hides the Arena for them (`GET /auth/me` says `"arena": false`), and Insights doesn't list the Arena's tables. The call webhook and the agent routes aren't affected: a call platform has no session, and they keep checking the pack key. With the setting empty, or sign-in off, everyone gets the Arena, as before. Jobs are filed under the workspace that started them. `GET /api/rl/jobs` lists your workspace's own, and a job of another workspace is a 404. Jobs from before that are filed under the Arena's workspace when the dashboard starts. ## Endpoints the Arena calls A version's endpoint and the judge's are addresses you type, and the dashboard calls them from inside the cluster. So they're checked, when you save them and again when they're called: a deployment on the platform's own gateway (`…/asgi//v` or `…/endpoint//v`), OpenAI and Anthropic, hosts the operator lists in `LEVERS9_PROVIDER_HOSTS`, and any other host over https whose addresses are all public pass. A private, loopback, link-local or cluster address doesn't, and neither does the controlplane's own API. Right now the dashboard only logs what it would refuse (`LEVERS9_EGRESS_GUARD=observe`); with `enforce` a refused address is a 422 that says why, and nothing is sent to it. A version you point at another host leaves its token behind unless you give a new one with it. Moving between two deployments on the same host, or between two addresses of the platform's gateway, keeps it. ## Vocabulary | word | meaning | | - | - | | **episode** | one conversation: turns, a score per reply, an outcome (viewing booked / follow-up / dropped) | | **reward** | the score of one reply, 0 to 100, from the rubric | | **judge** | whatever produces scores: cue-based rules until a model key is configured, then a model reading the whole conversation. Reviewers can overrule it | | **preference** | a reviewer's pick between two replies for one situation, plus the reasons ticked | | **policy version** | one agent version; keeps its parent and the number of preferences it learned from | | **scenario** | a group of similar situations across many calls, found automatically | | **snapshot** | anything frozen: a preference version (TSV on the volume), a policy version, a forked conversation | | **tenant** | one client; its own culture pack, webhook key and filters | ## The Arena tab Simulated customers with personas, moods and hidden needs talk to a policy. A customer's trust moves with each judged reply and decides the outcome. Start an episode, watch it turn by turn with the score and the rubric reasons for each reply, and **fork** it at any turn to continue with another version and see where the two diverge. Real calls can be turned into episodes too, so the synthetic customer continues from where the real one hung up. Without model keys the arena runs a deterministic mock: the customer follows a staged script and reply quality depends on which rubric skills the policy's prompt carries, so a trained version visibly improves. With `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` in the dashboard API's environment, or a vLLM endpoint, policies and the judge use real models; the UI shows which is active. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.