> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Decision models

> Register a Hugging Face decision model, run it over real calls as a GPU job, and read its report live on the Decision Engine page

The **Decision Engine** page (Data lab, then Decision Engine, at `/lab/engines`) has two sections:

* **Models**: trained decision models. You do not fit them; you measure them.
* **Algorithms**: the decision engines that learn a state to move table from a world's evidence
  ([Bring your own engine](https://levers9.mintlify.site/lab/your-own-engine.md)).

A **decision model** is a classifier that answers typed questions about a state. There are three families, and
each run loads or calls the model with its family's own backend:

| family | backend | models | what it is |
| - | - | - | - |
| `typed-decision` | [`laya`](https://pypi.org/project/laya/) | [`convaiinnovations/laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual) | typed-question classifier, a probability for every option |
| `kev` | [`kev`](https://github.com/jaredpalmer/kev) (`kev.serve.Server`, installed from git) | `jaredpalmer/kev-0.8b`, `kev-4b`, `kev-9b` | decision models on Qwen3.5 with the System One API shape, Apache-2.0 |
| `jev` | [`typesafe-sdk`](https://docs.typesafe.ai/api) | `jev` (TypeSafe's current public model, `jev-latest`) | TypeSafe's hosted System One API; **external**: see [External models](https://levers9.mintlify.site/lab/external-models.md) |

Jev is the one model that is not loaded on your cards: it is called over the internet, in a CPU job, with the
workspace secret `TYPESAFE_API_KEY`. It is a third-party service hosted outside the Kingdom, so it never
receives real customer calls unless the tenant has allowed it, with a reference to the client's written
permission ([External models](https://levers9.mintlify.site/lab/external-models.md)).

```python theme={null}
import laya

agent = laya.load("convaiinnovations/laya-multilingual")
out = agent.predict({"transcript": ["agent: ...", "customer: ..."]}, questions, max_len=2048)
out["answers"]["outcome"]   # {"choice": "booked", "probabilities": {...}, "answer_confidence": 0.62}
```

Kev answers the same question in the same shape (its stated confidence is called `confidence`), so a run reports
the same numbers for both. One difference to keep in mind: Kev was trained on states of up to 384 tokens, and
a call is about two thousand. A run passes the whole call as the state and the report says how long the calls
were and what share went past 384 tokens, so a weak Kev number reads as what it is: a model outside its
training length.

### Lev: our own fine-tuned decision model

A **Lev** is Levers' own decision model: a fine-tuned checkpoint in a folder on a volume, registered in your
workspace as `lev-<tenant>-<version>` (for example `lev-acme-v1`). Its base (Laya or Kev) is shown beside it. No
Lev ships with the dashboard: each one is registered per workspace from its checkpoint folder, and how well it
does is whatever your own runs measure on your own held-out calls.

A Lev comes with the exact question it was trained on, and a run asks that question, never the workspace
default. The question is read from the checkpoint's `rl_agent_config.json` when its `training` block records
it, either as `training.questions` (`{"acme": {"question", "options", "labels", "won"}}`) or as
`training.question` plus `training.options`; the job reads it again at run time, so the checkpoint is the
authority. A checkpoint whose config records no question can have its training text pinned when you register
it (`questions` on `POST /api/rl/models`). A Lev is asked only about the tenants it was trained for.

A Lev can also know its held-out set (`held_out` at registration: dataset, size, seed). Run it on anything
else (another seed, another size) and the report warns that the sample may contain calls it was trained on; the
Run panel has a button that puts the held-out set back.

A run asks the model about a sample of real calls and compares every answer with the label the platform
already wrote on the call. It tells you, in about a minute of GPU time, whether a model is worth using,
worth fine-tuning, or neither.

> [!NOTE]
>
> The platform's label was written by a model, not a person. What a run measures is **agreement**, not
> accuracy. It is still the fastest honest answer to "is this model any use on my calls?".

For labels a person has confirmed, build a gold set. The `decision` dataset kind in the Data Lab is made for it:
reviewers answer your questions row by row and mark the events behind each answer
([Decision gold sets](https://levers9.mintlify.site/lab/decision-gold-sets.md)).

## Register a model

Laya multilingual, Kev 0.8B, 4B and 9B, and Jev are registered already. To add another, open **Add a model** in
the Models section:

* **Hugging Face Hub**: paste the id (`owner/name`). The family is `kev` for `jaredpalmer/kev-*` and Laya
  otherwise, unless you pick one. The id is checked on the Hub before it is stored, so a typo fails at once and
  not after a GPU has started. A private or gated repo needs the cluster's `HF_TOKEN` to see it.
* **Checkpoint on a volume (Lev)**: the volume (`laya-lab`), the checkpoint folder and the tenant it was trained
  for. The folder is read: `rl_agent_config.json` makes it a Laya fine-tune, `head.pt` a Kev one. It is
  registered as the next `lev-<tenant>-v<N>` unless you give a version.

A base model is asked your workspace's own outcome question for each world, built from that world's data when
you run it: "How did the call in `transcript` end?", with one option per platform label found on the world's
real calls (the 20 most common), and the labels its **Outcomes** card calls won counted as wins. For a world
whose calls are labelled Booked, Hot Lead, Warm Lead and Not Interested, the options are `booked`, `hot_lead`,
`warm_lead` and `not_interested`. It is also asked the customer's overall sentiment (positive, neutral,
negative). A call whose label maps to no option is left out and counted. A world with fewer than two labels on
its real calls has no question until it has, or until you pin one (`questions` on `POST /api/rl/models`).

## Pick a dataset

The Run panel lists what a run can read, built each time you open it:

* **Call-log exports** on the `call-logs` volume, one folder per tenant. An undated file
  (`acme/acme_calls.jsonl.gz`) shows as *ACME · full history*; a dated one (`acme/acme_calls_20260929.jsonl.gz`)
  as *ACME · export of 29 Sep 2026*, so a later week sits beside the history under its own name. Until a run has
  counted a file, the picker shows its file name; the first run that reads it counts it, and the count shows afterwards.
* **Arena calls**: every world's calls in the Arena registry, with their labels counted now.

The sample is stratified by default: an equal share per label, and what a small label cannot fill goes to the
others. Untick it for a uniform draw that keeps the export's own mix. The same seed draws the same calls.

> [!WARNING]
>
> Only the transcript text reaches the model: one string per turn (Laya keeps the end of a call longer than its
> window; Kev is given the whole call). Phone numbers
> and every other field of an export row stay on the volume, and a run's stored answers carry no transcript.

## Run it and watch the report

**Run** starts a headless levers9 job in your workspace (`lab/rl/model_run.py`, app `lab-models`) on two RTX
PRO 6000 cards by default. It keeps going if you close the page, and a restart of the dashboard picks it up
again from its platform task. The report fills in as it labels, polled every second:

1. **Stages**: load and sample, load the model, label calls, report.
2. **Progress**: calls done, calls a second, time left.
3. **Agreement so far** against chance (one over the number of options), won against not won, sentiment.
4. **Confusion**: rows are the platform's label, columns the model's answer, growing as calls are labelled.
   A model that answers the same thing for everything shows as one bright column.
5. **Confidence against correctness**: how often calls in each band of stated confidence actually agreed,
   with the **85% review threshold**: below it a call would go to a person.
6. **Calibration**: one temperature fitted on half the calls and tested on the other half. A fitted
   temperature far above 1 means the model is better calibrated only because it admits it does not know.
7. **Verdict**: *not usable without training* (under 1.5 times chance), *weak: fine-tune before use* (under
   60%), or *promising: calibrate on the gold set*.

**Full log** opens the job container's own output, which prints the same report as aligned tables.

Finished runs are kept under **Past runs** with the same report. The per-call answers (call id, the platform's
label, the model's choice and probabilities) are written to the `model-runs` volume as
`<model>/<run>/answers.jsonl.gz`.

For scale: Laya labels a few thousand calls in about a minute on an RTX PRO 6000, roughly 13 ms a call.

## Compare models

**Compare models**, above the Run panel, puts two or more models on the **same calls** and one leaderboard.
Pick the models, pick the dataset, and run:

* **The same sample for every model.** The calls are drawn the way a run draws them (grouped by label, the
  same seed), inside each job, and each job reports a fingerprint of the call ids so the page can check they
  match. With a Lev in the comparison, its question is what the calls are grouped by and, unless you pick
  otherwise, the dataset, size and seed are its recorded **held-out set**, so the draw is exactly the calls it
  never saw in training. If the sample is anything else, the comparison warns that the Lev may have been
  trained on some of these calls.
* **Each model is asked the question it was trained on.** A Lev asks its own wording; a base model asks the
  workspace's outcome question for the world. Each answer is scored against that model's own mapping of the
  platform label, so the numbers compare; when wordings differ the report says so.
* **Where it runs.** Laya, Kev and every Lev run one after another in one GPU job (two RTX PRO 6000 cards by
  default, for the reason under [Which card](#which-card)); Jev runs in its own CPU job, up to 8 requests in
  flight, each retried with backoff.

The leaderboard fills in while the jobs label. Per model:

| column | what it is |
| - | - |
| Outcome agreement | answers that match the platform's label, on the balanced (stratified) sample; a bar with the chance line on it |
| Real mix | the same agreement weighted to the label mix of the whole dataset, so a label that is rare in real life counts as rarely |
| Won vs not | whether the model and the platform agree on won against not won |
| Calibration | calibration error as shipped, then after one fitted temperature |
| Over 85% | the share of calls the model is at least 85% sure of (they would skip review), and how often those are right |
| p50 / p95 | per-call latency: GPU time per call for a model on the cards, the round trip for Jev |
| Per 1,000 calls | cost: GPU seconds (and dollars when `LEVERS9_COMPARE_GPU_HOUR_USD` is set), or Jev's input tokens (and dollars when `LEVERS9_JEV_PRICE_PER_M_INPUT` is set) |
| Verdict | the same three verdicts as a run |

Below it: **recall by label** (of the calls under each label, the share each model got right) and, when both
jobs have finished, **how often each two models give the same answer** (the same option when they share
options, else the same side of won against not won).

Finished comparisons are kept under **Past comparisons** (the leaderboard, never a transcript). The per-call
answers are on the `model-runs` volume as `compare/<id>/<model>.jsonl.gz`.

The **scripted** dataset, *Decide page scenarios*, holds the made-up calls from the Decide page in English
and Saudi Arabic, each with the ending a person would give it. It is small (eight calls), needs no permission
to send to an external model, and is how Jev can be tried before a tenant allows it.

## From a terminal

```bash theme={null}
levers9 model ls
levers9 model add jaredpalmer/kev-4b
levers9 model add --volume laya-lab --path models/acme-ft --tenant acme      # a Lev: lev-acme-v1
levers9 model datasets
levers9 model run laya-multilingual --dataset acme --n 5000       # follows the job and prints the verdict
levers9 model run laya-multilingual --dataset arena:acme --no-wait
levers9 model runs
levers9 model compare --models lev-acme-v1,kev-4b,laya-multilingual,jev --dataset acme   # a Lev alone defaults to its held-out set
levers9 model compare --models kev-4b,jev --dataset scripted:decide
levers9 model compares
levers9 model external-allow --tenant acme --reference "signed letter ACME-2026-14"
```

A run or comparison started here is the same job the page draws, so it shows up there live.

## The API

| call | what it does |
| - | - |
| `GET /api/rl/models` | the registry, with each model's questions, run count and last result |
| `POST /api/rl/models` | register `{hf_id, family?, label?, questions?}`, or a Lev `{volume, path, tenant, version?, questions?, held_out?}`; 422 when the Hub does not have it or the folder is not a checkpoint |
| `DELETE /api/rl/models/{id}` | forget a model; its runs are kept |
| `GET /api/rl/datasets` | the datasets a run can read |
| `POST /api/rl/models/{id}/run` | `{dataset, n, seed, stratify, gpu?, gpu_count?}`; answers a ledger job of kind `model-run` |
| `GET /api/rl/jobs/{job}` | the job, with its live `report`, the last 200 `log_tail` lines, `task_id` and `container_id` |
| `GET /api/rl/models/runs`, `GET /api/rl/models/runs/{run}` | finished runs and one run's full report |
| `POST /api/rl/models/compare` | `{models: [ids], dataset?, n?, seed?, stratify?, tenant?}`; answers a ledger job of kind `model-compare` whose `report.models` fills per model as it labels; 422 from the external-model guard |
| `GET /api/rl/models/compares`, `GET /api/rl/models/compares/{id}` | finished comparisons and one comparison's leaderboard, recall grid and pairwise agreement |
| `GET /api/rl/external-models?tenant=`, `PUT /api/rl/external-models` | a tenant's external-model setting; see [External models](https://levers9.mintlify.site/lab/external-models.md) |

## Which card

The card and the count default to `LEVERS9_MODEL_RUN_GPU` (`RTXPro6000`) and `LEVERS9_MODEL_RUN_GPU_COUNT` (`2`)
on the dashboard API. Two cards, not one, on purpose: on the attached RTX PRO 6000 pool a one-card job can land on
a one-card node that never starts it, and it sits PENDING while its container reads RUNNING. The model needs
one card; the second keeps the job off that node. See [Multi-GPU](https://levers9.mintlify.site/guides/multi-gpu.md) for how cards are chosen.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.