> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Decision gold sets

> Build and annotate a human gold set for a decision model: answers to typed questions, the events that caused them, counterfactuals, and the review of a logged decision

A **decision model** answers typed questions (yes or no, one of a few options, a level on a scale) and gives
each answer a probability. It does not write text. It is the kind of model served on the
[Decision API](https://levers9.mintlify.site/lab/decision-api.md). A **gold set** is a set of examples whose labels a person has confirmed: you
measure a model against it, and with enough rows you train on it.

The `decision` dataset kind (shown as **Decision model**) is where that gold set is made. Its rows are about
**leads**: a lead is a customer being followed up, such as someone asking about a property or a policyholder
whose insurance is due for renewal. A row is one **decision point** of a lead: the moment something had to be
decided about it. It holds:

* the lead's **journey**: its past events, oldest first;
* the new call;
* optionally a **logged decision**: the action an engine took at that moment, with its propensity (the
  probability the engine gave that action) and its outcome (whether the lead then succeeded). The engine is
  whatever picks the next action for a lead: a policy, a set of rules, or the Decision API's
  [decision step](https://levers9.mintlify.site/lab/recommendation.md).

A reviewer does four things with a row:

1. answers each question;
2. marks the events that caused each answer;
3. says how an answer would change without an event;
4. reviews the logged decision.

The lifecycle is the one every [dataset](https://levers9.mintlify.site/lab/datasets.md) has: import, optional pre-label, annotate, export.

## How it differs from the language-model kind

| | `llm` (Language model) | `decision` (Decision model) |
| - | - | - |
| a row | a conversation | one decision point of a lead: its journey, the new call, optionally a logged decision |
| what a person writes | the target reply, as free text | one option per typed question, and the events that caused it |
| pre-label | a language model drafts the reply | a Decision API drafts every answer |
| what leaves | TSV | TSV, the gold set as JSON lines, the questions as a schema |
| the model it is for | one that writes the next reply | one that scores fixed options |

Choose `decision` when the model picks from fixed options: where a lead stands, whether price is still an
objection, which action to take next. Choose `llm` when the model writes.

## The questions

The questions are **System One questions**: the same JSON the [Decision API](https://levers9.mintlify.site/lab/decision-api.md) takes in
`questions`. Each one has an id, a type, the question in words and its options.

| type | the answer | its options |
| - | - | - |
| `choice` | one of a few options | `criteria`, as `{key: description}`, two or more |
| `noul` | yes or no | `true` and `false` |
| `score` | a level on an ordered scale | `criteria`, a list of levels from lowest to highest; the options are `0`, `1`, `2` and so on |

A questions file for a property sales lead:

```json theme={null}
{
  "stage": {
    "type": "choice",
    "question": "Where does this lead stand after the call?",
    "criteria": {
      "ready_to_reserve": "asked to reserve, to pay a deposit, or for a written offer",
      "wants_viewing": "asked for a viewing or agreed to one",
      "needs_information": "asked for details and committed to nothing",
      "not_interested": "said no, or asked not to be called again"
    }
  },
  "price_objection": {
    "type": "noul",
    "question": "Is price still an open objection for this lead?"
  },
  "urgency": {
    "type": "score",
    "question": "How soon does the lead want to move?",
    "criteria": ["no date in mind", "within a few months", "within this month"]
  }
}
```

The words of the question go in `question`. `instructions`, the name the Decision API's own examples use, is
read the same way, so a questions file you already send to the Decision API works as it is.

Write each description as the test a reviewer applies, not as a restatement of the key. Two reviewers who read
"asked to reserve, to pay a deposit, or for a written offer" agree more often than two who read "ready".

The questions are part of the dataset's config, as `config.task.questions`, and a decision dataset cannot be
built without them.

### The actions

A row may carry a decision the engine logged. The **actions** are what that decision chose from: a JSON object
of `{key: description}`, with `no_action` among them. They are optional, stored as `config.task.actions`, and they
are what a reviewer picks from when saying what they would have done.

```json theme={null}
{
  "send_offer": "send a written price offer for the unit they viewed",
  "hand_to_rep": "hand the lead to a human sales rep",
  "call_again": "make an automated follow-up call",
  "no_action": "do nothing for now"
}
```

## The record format

The import reads JSON lines (one record per line), records printed over several lines one after another, or a
file holding a JSON array of records. A webhook takes the same record as its body. One complete record, spread
over several lines here for reading:

```json theme={null}
{
  "id": "lead-2041-d3",
  "lead_id": "lead-2041",
  "decision_time": "2026-09-24T16:30:00+03:00",
  "segments": [
    {"role": "agent", "text": "Good afternoon. I am calling about the two-bedroom unit you viewed last week."},
    {"role": "customer", "text": "Yes, we liked it. The price was the problem, but your colleague sent a new payment plan."},
    {"role": "agent", "text": "Does the new plan work for you?"},
    {"role": "customer", "text": "It does. Please send the offer in writing so I can show my brother."}
  ],
  "journey": [
    {"id": "ev-1", "time": "2026-09-10T11:02:00+03:00", "source": "CRM", "type": "stage",
     "text": "Inbound lead from the website, asked about two-bedroom units", "provenance": "confirmed"},
    {"id": "ev-2", "time": "2026-09-14T18:40:00+03:00", "source": "AI call", "type": "label",
     "text": "First call: price objection, asked for a lower down payment", "provenance": "model"},
    {"id": "ev-3", "time": "2026-09-20T10:00:00+03:00", "source": "calendar", "type": "booking",
     "text": "Viewing attended, unit B-204", "provenance": "confirmed"},
    {"id": "ev-4", "time": "2026-09-22T09:15:00+03:00", "source": "call center", "type": "message",
     "text": "Rep sent a revised payment plan: 10% down, 24 months"},
    {"id": "ev-5", "time": "2026-09-25T12:00:00+03:00", "source": "payments", "type": "payment",
     "text": "Deposit received for unit B-204", "provenance": "confirmed"}
  ],
  "decision": {"action": "send_offer", "propensity": 0.9333, "policy": "engine-v3",
               "outcome": null, "time": "2026-09-24T16:31:00+03:00"},
  "labels": {"stage": "ready_to_reserve"},
  "causes": {"stage": ["ev-3", "ev-4"]}
}
```

| field | what it is |
| - | - |
| `id` | the row's id, unique in the dataset. Without one the row is named after its lead |
| `call_id` or `lead_id` | the lead. Every decision point of one lead carries the same value |
| `decision_time` | when the decision was made, in ISO 8601. An event at or after it is not history |
| `segments` | the new input: the call as `[{role, text}]`. `turns` (a list of `"agent: ..."` and `"customer: ..."` lines, as the Decision API takes a call) or `context` or `text` (the call as plain text, one turn per line) are read too |
| `journey` | the history: the lead's events, oldest first, each `{id, time, source, type, text}` with an optional `provenance`. `events` is read as the same field. An event without an `id` is given one (`ev-1`, `ev-2`, and so on). An empty list is a lead with no history |
| `decision` | optional: the decision the engine logged, `{action, propensity, policy, outcome, time}` |
| `labels` | optional: answers you already have, as `{field: option}` |
| `causes` | optional: causes you already have, as `{field: [event ids]}` |
| `questions` | optional: System One questions the record brings with it. The questions set when the dataset is created take precedence |

A record needs a call, a history, or both. Some of these words need a definition:

* An **event** is one thing that happened to the lead: a stage change in the CRM, a call, a booking, a payment,
  a message. Its `source` says where it came from and its `type` what it is. Both are free text; the
  [Causality builder](https://levers9.mintlify.site/lab/decide-playground.md#causality-builder) uses the same fields.
* **Provenance** says how far to trust an event's text: `model` (a model wrote it), `confirmed` (a person
  checked it) or `corrected` (a person changed it).
* The **propensity** is a number between 0 and 1. An engine that sometimes tries an action other than its best
  one logs it with every decision ([Recommendations](https://levers9.mintlify.site/lab/recommendation.md#exploration-and-the-propensity)).
  `policy` names the engine or the version of it that decided.
* The logged **outcome** is `yes` or `no`. Leave it empty when it was not known when the record was written.

Two things in the example are worth a second look. `ev-5` is dated after the decision time, so it is not part
of the history: it cannot be marked as a cause, and it is left out of the exported journey. And the `labels` and
`causes` the record brings are imported as **drafts**, never as human labels. A row counts only once a reviewer
has saved it.

### Tables

A tabular source (TSV, CSV, parquet, a query) maps its columns to these roles:

| role | the column holds |
| - | - |
| `call_id` | the lead |
| `context` | the new call, as text |
| `journey` | the events, as JSON text in the cell |
| `decision_time` | the ISO 8601 time |
| `decision` | the logged decision, as JSON text in the cell |

### Sources

A decision dataset reads from every source that feeds the language-model kind: a folder on the `datasets` volume,
files dropped in the browser, S3, GCS, BigQuery, Hugging Face, a URL list, a webhook, or a query against your own
Postgres or MySQL. It can also be built from a dataset you already have: a language-model dataset, or the
validated rows of a speech-to-text dataset. Those give one row per call with an empty journey, which is the
right start when you have calls and no record of events yet.

## What a row holds

After the build, every decision point is one line of `rows.tsv` in the dataset's folder on the `datasets` volume:

| column | what it is |
| - | - |
| `id` | the row's id |
| `call_id` | the lead |
| `decision_time` | when the decision was made |
| `context` | the new call: its turns as `role: text`, on one line |
| `journey` | the lead's events, JSON on one line |
| `decision` | the logged decision, JSON on one line; empty when there is none |
| `draft_output` | the drafts for this row, JSON on one line |
| `provider` | who wrote the drafts |
| `status` | `pending`, `validated` or `flagged` |
| `split` | `train`, `val` or `test`. It is drawn from the lead, so every decision point of one lead lands in the same split |

## Create one

### From the dashboard

Open **Datasets**, then **New dataset**. Pick the kind **Decision model**, then a source, then whether a model
drafts the answers. On the last step, name the dataset and give its **Questions**: paste the JSON, or upload
the file. The dialog lists back each question it read, with its type and its number of options, so a wrong
paste shows before the build. **Actions** are optional: one per line, as `key: description`. The build runs on
the cluster and the page follows it, as for every kind.

### From the CLI

```bash theme={null}
levers9 dataset upload ./decisions.jsonl --to leads-gold        # onto the datasets volume, in uploads/leads-gold
levers9 dataset create leads-gold --kind decision --path uploads/leads-gold \
        --questions questions.json --actions actions.json
```

`--questions` is sent as `task.questions`; leave it out only when the records in the folder carry their own `questions`.
`--actions` is optional and is sent as
`task.actions`. The CLI builds from a folder on the `datasets` volume; pick any other source in the dialog.

For a table, name the columns with `--map`. `inspect` prints the columns and a sample first:

```bash theme={null}
levers9 dataset inspect uploads/renewals --kind decision
levers9 dataset create renewals-gold --kind decision --path uploads/renewals \
        --questions questions.json \
        --map call_id=PolicyNo --map context=CallText --map journey=Events \
        --map decision_time=DecidedAt --map decision=Logged
```

## Pre-label with the Decision API

Pre-labelling is optional. The provider for this kind is `decision_api`, with an `endpoint`: the URL of a
Decision API deployed in this workspace (the `endpoint` line of `levers9 decide status`). For every row it
drafts:

* every answer, with the model's confidence in it;
* the events the model read most as evidence for that answer, when the served model returns them;
* a chance per action, when the response carries one.

```bash theme={null}
levers9 dataset create leads-gold --kind decision --path uploads/leads-gold \
        --questions questions.json --actions actions.json \
        --provider decision_api --endpoint https://levers9.uselevers.com/asgi/lev/v1
```

To draft a dataset that already exists, use **Pre-process** on its page, or
`POST /api/datasets/{id}/prelabel` with `{"provider": {"type": "decision_api", "endpoint": "..."}}`. Only rows
without a draft are drafted unless you ask to overwrite, and a draft never overwrites a human label. The
provider `none` imports only: reviewers start from an empty row.

> [!WARNING]
>
> A draft pulls a reviewer toward the model's answer. If the gold set will measure the model that drafted it,
> the agreement comes out too high. For rows that will measure a model, import without pre-labelling, or draft
> with a different model.

## The annotation screen

Open the dataset. The row list is on the left, as for every kind ([Annotation](https://levers9.mintlify.site/lab/annotation.md)). The editor is
laid out the way a row is reviewed: reading first, answering second.

**Reading.** **The journey** lists the events before the decision time, oldest first; a lead with no history
says so. **The new call** follows it. Events at or after the decision time are listed apart and dimmed: they are
not part of the history and cannot be marked as causes.

**Answering.** **The questions** come one block each, and one question is active at a time. Then the logged
decision, then a rationale.

| part | what the reviewer does | saved as |
| - | - | - |
| Answers | picks one option for each question, or **None of these** when no option fits | `answers` |
| not sure | marks a question they are not sure of. An answer given with it is kept and counts for half in the export | `unsure` |
| Causes | with a question active, clicks the events in the journey that caused its answer: the ones without which they would have answered differently. Each lands under the question as a chip. No event marked means the answer comes from the call alone | `causes` |
| Counterfactual | on a cause's chip, picks what the answer would be without that event. Removing the cause removes it | `counterfactual` |
| The logged decision | reads the action the engine took and what was logged with it; sets the **reviewed outcome** as they find it (`yes`, `no` or `unknown`), whatever was logged; picks the action they would have taken. A row with no logged decision has no outcome to review | `outcome`, `action` |
| Rationale | writes, in a sentence or two, why. Optional | `rationale` |

**Drafts.** A model's draft is shown preselected, with its confidence and the probability it gave each option.
It is not the reviewer's answer until they confirm it: by clicking it, by pressing its number, or by validating
the row, which confirms every draft left as it was. The events the draft read most are offered as suggested
causes, and each action shows the draft's chance when there is one.

**Saving.** Edits are saved automatically a moment after the last one, and at once when you move to another
row. **Validate & next** moves the row from **pending** to **validated**; every question needs an answer or a
not sure mark first. **Flag & next** marks a row that is unclear or unusable. When the API refuses a label, its
sentence is shown at the top of the editor.

| keys | action |
| - | - |
| `1` to `9` | pick that option of the active question. The number after the last option is None of these |
| `Enter` | confirm the draft of the active question |
| `↑` `↓` | move between questions |
| `U` | not sure |
| `⌘ Enter` (Ctrl on Windows) | validate and go to the next row |
| `⌘ .` | flag and go to the next row |
| `⌘ ↑` `⌘ ↓` | previous or next row |

Statuses, filters, named links for outside reviewers and the NDA gate work as for every kind
([Sharing](https://levers9.mintlify.site/lab/sharing.md)).

### The labels

Every save appends to `labels.tsv`. Nothing is ever edited in place, and each line carries the reviewer's name.
A label's value is a string; the first four hold JSON. In them, a **field** is a question's id (`stage`,
`price_objection`) and an option is one of that question's option keys.

| label | value | meaning |
| - | - | - |
| `answers` | `{field: option_key}` | the answer to each question. `"__none__"` is none of these. A `noul` question takes `"true"` or `"false"`, a `score` question the level's number as a string |
| `unsure` | `[field]` | the questions the reviewer was not sure of |
| `causes` | `{field: [event_id]}` | the events that caused each answer |
| `counterfactual` | `[{field, event_id, answer}]` | without this event, the answer to this question would be this option |
| `outcome` | `yes`, `no` or `unknown` | the reviewed outcome of the logged decision |
| `action` | an action key | what the reviewer would have done |
| `rationale` | free text | why |

Through the API, one save is one request:

```bash theme={null}
curl -s -X POST $DASH/api/datasets/$ID/labels -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{
  "row_id": "lead-2041-d3",
  "status": "validated",
  "user": "reviewer-1",
  "labels": {
    "answers": "{\"stage\": \"ready_to_reserve\", \"price_objection\": \"false\", \"urgency\": \"1\"}",
    "unsure": "[\"urgency\"]",
    "causes": "{\"stage\": [\"ev-3\", \"ev-4\"], \"price_objection\": [\"ev-4\"]}",
    "counterfactual": "[{\"field\": \"price_objection\", \"event_id\": \"ev-4\", \"answer\": \"true\"}]",
    "outcome": "yes",
    "action": "send_offer",
    "rationale": "The revised plan removed the price objection. She asks for the offer in writing."
  }
}'
```

The API checks every save and answers `422` with a sentence when something does not fit: a field that is not
one of the dataset's questions, an option that is not one of that question's, an event id that is not in the
row's journey. An event at or after the decision time cannot be a cause.

## What each label is for

* **Answers** are what the model should say. In training they are the target for each question. In evaluation
  they are the yardstick: the share of rows where the model's answer is the reviewer's, for each question and
  for each option. An answer marked not sure counts for half.
* **Causes** tell you whether the model read the right events. A served model can return the events it weighed
  most for an answer ([explain](https://levers9.mintlify.site/lab/decision-api.md#explain-what-the-model-read)); compare them with the events
  the reviewer marked. A model that gets the answer right from the wrong events is right by luck.
* **Counterfactuals** become **twin records**: a copy of the row with one event removed, and the answer the
  reviewer said would follow. They test whether the answer follows its cause. Ask the model about the row and
  about its twin: if the answer does not change when the cause is gone, the model did not use it. What a model
  reads most is a clue, not proof; the twin is the test.
* **Outcome and propensity** are for learning what an action changes. The history up to the decision, the
  action and the reviewed outcome make one example of what happens after that action. Weighting each example by
  one over its propensity corrects for the engine having preferred some actions, and the same logs can then
  score a new policy before it runs. An `unknown` outcome is left out; it is not a failure.
* **The reviewer's action** measures agreement with a person in the **review lane**: the decisions the engine
  sends to a person because they are close or doubtful ([Recommendations](https://levers9.mintlify.site/lab/recommendation.md#review)). Where
  the engine's action and the reviewer's differ is where to look first.
* **The rationale** is for the people who read the disagreements. It is not a label a model learns from.

## Export

Three files leave a decision dataset.

| route | what it is |
| - | - |
| `GET /api/datasets/{id}/export.tsv` and `POST /api/datasets/{id}/import` | the spreadsheet round trip, as for every kind ([Annotation](https://levers9.mintlify.site/lab/annotation.md#spreadsheets)). The label columns hold the same values as `labels.tsv`. The file has no quoting, so the quotes inside a JSON cell are written with a backslash in front (`{\"stage\":\"qualified\"}`); the import reads that form, a spreadsheet's re-quoted form and plain JSON alike |
| `GET /api/datasets/{id}/export.jsonl?status=validated&twins=1` | the gold set: one JSON object per line |
| `GET /api/datasets/{id}/schema.json` | the questions as a decision-model schema |

```bash theme={null}
levers9 dataset export leads-gold --format jsonl --status validated --twins -o gold.jsonl
levers9 dataset export leads-gold --format schema -o schema.json
levers9 dataset export leads-gold --status validated -o leads-gold.tsv     # --format tsv is the default
```

Without `-o` the export goes to stdout, so it can be piped.

These are the owner's exports, and they hold the text as it is. If the dataset's
[policy](https://levers9.mintlify.site/lab/personal-data.md#who-reads-the-text-off-flag-and-mask) is Mask, a share link's gold set holds the masked
call, journey and rationale; add `?text=masked` to your own `export.jsonl` or `export.tsv` for the same file,
or `--text masked` to `levers9 dataset export`.

A decision dataset is not [published to a bucket](https://levers9.mintlify.site/lab/publishing.md): that copies recordings and a transcript
manifest, and this kind has neither. Download the gold set instead.

### The gold set

One line of `export.jsonl` for the record above, wrapped here for reading:

```json theme={null}
{"id": "lead-2041-d3", "call_id": "lead-2041", "tenant": "leads-gold",
 "decision_time": "2026-09-24T16:30:00+03:00",
 "segments": [{"role": "agent", "text": "Good afternoon. I am calling about the two-bedroom unit you viewed last week."},
              {"role": "customer", "text": "Yes, we liked it. The price was the problem, but your colleague sent a new payment plan."},
              {"role": "agent", "text": "Does the new plan work for you?"},
              {"role": "customer", "text": "It does. Please send the offer in writing so I can show my brother."}],
 "journey": [{"id": "ev-1", "time": "2026-09-10T11:02:00+03:00", "source": "CRM", "type": "stage", "text": "Inbound lead from the website, asked about two-bedroom units", "provenance": "confirmed"},
             {"id": "ev-2", "time": "2026-09-14T18:40:00+03:00", "source": "AI call", "type": "label", "text": "First call: price objection, asked for a lower down payment", "provenance": "model"},
             {"id": "ev-3", "time": "2026-09-20T10:00:00+03:00", "source": "calendar", "type": "booking", "text": "Viewing attended, unit B-204", "provenance": "confirmed"},
             {"id": "ev-4", "time": "2026-09-22T09:15:00+03:00", "source": "call center", "type": "message", "text": "Rep sent a revised payment plan: 10% down, 24 months"}],
 "labels": {"stage": "ready_to_reserve", "price_objection": "false", "urgency": "1"},
 "weights": {"urgency": 0.5},
 "cause": {"stage": ["ev-3", "ev-4"], "price_objection": ["ev-4"]},
 "decision": {"action": "send_offer", "propensity": 0.9333, "policy": "engine-v3", "outcome": "yes",
              "time": "2026-09-24T16:31:00+03:00", "logged_outcome": null},
 "reviewer_action": "send_offer",
 "rationale": "The revised plan removed the price objection. She asks for the offer in writing.",
 "split": "train", "annotators": ["reviewer-1"], "twin_of": null, "flip": null}
```

| field | what it is |
| - | - |
| `id`, `call_id`, `decision_time`, `segments` | the row, as imported |
| `tenant` | whose data this is: the dataset's name |
| `journey` | the events before the decision time. `ev-5` is gone |
| `labels` | the reviewer's answers, `{field: option}` |
| `weights` | `{field: 0.5}` for each answer marked not sure. A field that is not listed counts in full |
| `cause` | the events marked as causes, `{field: [event ids]}` |
| `decision` | the logged decision, or `null` when the row has none. `outcome` is the reviewed one (`yes`, `no`, or `null` when the reviewer set `unknown`); `logged_outcome` keeps the original |
| `reviewer_action` | the action the reviewer would have taken |
| `rationale` | the reviewer's sentence |
| `split` | the row's split |
| `annotators` | everyone who labelled the row |
| `twin_of`, `flip` | empty on an original record; set on a twin |

`status=validated` keeps the rows a person validated, which is what a gold set is. A row with no human answer
is never exported, whatever its status.

### Twins

With `twins=1` every counterfactual adds one more record: the same row with that event removed from the
journey. Its `twin_of` is the original's id, its `flip` names the event and the question, and its `labels` hold
that one question only, with the answer the reviewer gave for the world without the event. The twin of the
record above has the journey without `ev-4` and:

```json theme={null}
{"id": "lead-2041-d3~cf1", "twin_of": "lead-2041-d3", "flip": {"removed": "ev-4", "field": "price_objection"},
 "labels": {"price_objection": "true"}}
```

Keep a record and its twin on the same side of any split you make yourself: they share everything but one event.

### The schema

`schema.json` is the dataset's questions in the form a decision model is configured with: every question
becomes an `enum` field, whatever its type.

```json theme={null}
{
  "tenant": "leads-gold",
  "version": 1,
  "fields": {
    "stage": {"type": "enum", "question": "Where does this lead stand after the call?",
              "options": {"ready_to_reserve": {"description": ["asked to reserve, to pay a deposit, or for a written offer"]},
                          "wants_viewing": {"description": ["asked for a viewing or agreed to one"]},
                          "needs_information": {"description": ["asked for details and committed to nothing"]},
                          "not_interested": {"description": ["said no, or asked not to be called again"]}}},
    "price_objection": {"type": "enum", "question": "Is price still an open objection for this lead?",
                        "options": {"true": {"description": ["yes"]}, "false": {"description": ["no"]}}},
    "urgency": {"type": "enum", "question": "How soon does the lead want to move?",
                "options": {"0": {"description": ["no date in mind"]}, "1": {"description": ["within a few months"]},
                            "2": {"description": ["within this month"]}}}
  }
}
```

The option keys in the schema are the keys in `labels`, so the gold set and the schema always fit each other.

## How many rows, and which

* **Stratify by answer.** A random draw gives you hundreds of the common answer and five of the rare one, and
  the rare one is where a model fails. Use the drafts, or any label you already have, to pick about the same
  number of rows for each option of the question you care about most. The reviewer still decides the answer.
* **Include rows with no history.** A first contact has an empty journey. These rows show whether the model can
  answer from the call alone, and no event should be marked as a cause on them.
* **Include rows where the history contradicts the call.** The CRM says the lead is cold and the customer asks
  for a viewing. An old price objection was settled by a later message. These are the rows where reading the
  right events changes the answer, and where a counterfactual is worth the most.
* **Put two reviewers on a share of the rows**, one in ten to one in five, without either seeing the other's
  answers, and measure their agreement: for each question, the share of those rows where both picked the same
  option. `labels.tsv` keeps every save with the reviewer's name, so it can be computed from that file. The
  simplest way to keep the second reviewer blind is a second dataset built from the same records. Low agreement
  on a question usually means its option descriptions are unclear: fix them before labelling the rest. No model
  can be expected to agree with the gold set more than two people agree with each other.
* **Size sets what you can see.** On 200 rows an agreement number moves by five to seven points either way by
  chance; on 1,000 rows by two to three. A difference smaller than that between two models is a tie.
* **Keep the gold set out of training.** A lead in the gold set should not be in the data a model trains on.
  The `split` column is drawn from the lead for that reason.

Next: [Decision models](https://levers9.mintlify.site/lab/decision-models.md) to measure a model, the [Decision API](https://levers9.mintlify.site/lab/decision-api.md) to serve
one, and [Recommendations](https://levers9.mintlify.site/lab/recommendation.md) for what the logged decision and its propensity are.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.