> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Personal data

> Find names and numbers in a dataset's text, choose who reads them, and measure how much the scan misses

Call transcripts are full of personal data: the customer's name, a phone number read out digit by digit, an ID
number, an IBAN. The Data Lab deals with it in three steps, all on the **Anonymization** card of a dataset:

1. **Scan.** Find the personal data in every text of the dataset.
2. **Choose a policy.** Decide who reads the text as it is and who reads it masked.
3. **Check the scan.** Have people review a random sample of rows, so you know how much the scan gets wrong and
   how much it misses. Until you've done that, nobody knows how good the masking is.

Nothing here changes the dataset. The rows and labels stay as they were written; the scan and the review write
their own files beside them.

When the dataset is ready to train on, the same card writes it out with the personal data taken out, and a
report beside the file that says how well the scan was checked: see [Export for training](#export-for-training).

## What the scan reads

| kind | text that is scanned |
| - | - |
| `stt` | the draft transcript, and the transcript a person wrote |
| `tts` | the sentence, and what Whisper heard when it listened back to the clip |
| `llm` | the conversation, the model's draft, and the reply a person wrote |
| `decision` | the call, every text inside the journey, and the reviewer's rationale and action |

A label is read whenever it's text a person wrote. A label a person picks isn't text and isn't read: a
speaker's role, a clip's rating, and a decision dataset's answers, causes and outcome.

## What it finds, and how

The scan is four layers that read one text at a time, and a pass that reads the whole dataset first. Each
looks in its own way. Where two of them find the same thing, the
stronger one wins: a value the call is known by beats a number that proves itself with a checksum, which beats
a pattern, which beats a guess from the words around it. A language model can be asked for names on top of
these, as a last step: see [Asking a model for names](#asking-a-model-for-names).

* **Known values.** If the import mapped the customer's phone number and name (see
  [Known values](https://levers9.mintlify.site/lab/datasets.md#known-values-the-customers-phone-number-and-name)), the scan looks for exactly
  those in that call's text, in every form: with or without the country code, in Arabic-Indic digits, said as
  words, and a name through the ways Arabic is spelled or in any letter case. This is the layer that finds a
  name said with no cue beside it.
* **Numbers with a shape.** Saudi mobile numbers however they're written (`+9665…`, `009665…`, `05…`, with
  spaces or dashes, in Arabic-Indic digits), national ID and Iqama numbers, IBANs, card numbers, emails, URLs,
  plates, and any other run of six digits or more. A number of nine digits or more written in groups
  (`920 000 00 01`, `011...400...0000`) is found too: a phone when it has the shape of a Saudi landline, a
  unified number (9200) or a free one (800), a number otherwise. An `@handle` of three characters or more is
  masked as `[URL]`. An ID and a card are checked with Luhn and an IBAN with mod 97. A ten-digit number that looks like an ID but fails the check is still reported, with a lower score,
  because a transcript often gets one digit wrong. A long number with a currency or a unit right after it
  (`750000 ريال`, `120000 متر`) is a price or a size, and is left as it is.
* **Numbers said out loud.** On a call a customer dictates a phone or an ID number digit by digit. Five or
  more digit words in a row («صفر خمسة خمسة واحد …», `zero five five one …`) are a spoken number.
* **Names by what's next to them.** A word is taken for a name after a vocative or an honorific (`يا فهد`,
  `أستاذة نورة`, `أبو فهد`), when someone introduces themselves (`اسمي …`, `معك نورة من شركة …`,
  `my name is …`), and for a capitalised Latin word right after a greeting (`مرحبا FAHAD`). An ordinary
  phrase that only starts the same way, like `يا هلا`, `أنا أبي شقة` or `معك حق`, is not a name.

Each finding becomes a typed placeholder, so the sentence still reads: `[NAME]`, `[PHONE]`, `[ID]`, `[IBAN]`,
`[CARD]`, `[EMAIL]`, `[URL]`, `[PLATE]`, `[NUMBER]`.

### Names the dataset gives away

Some names can't be found by reading one row at a time. So before a scan reads the rows, it reads the whole
dataset once and learns three things.

**Where a script puts the name.** If one side of your calls works from a script, it says the customer's name
in the same spot of its opening, call after call. The scan finds that spot by counting: the words around it
stay the same and the word in it keeps changing. Whatever stands there in a row is treated as that row's own
known name and is looked for in every text of the row. You don't tell the scan which side is the caller. It
works that out from whose opening repeats.

**What people answer when asked who's speaking.** When the caller asks for a name, a short answer is taken as
one.

**Which words are just ordinary.** A word the other side says in half a percent of the calls or more is never
taken for a name by these steps. That's what keeps a customer's hello from being hidden as a name. A common
first name is the exception: however often it's said, it stays a name.

A name the scan knows in Latin letters, from a `known_name` column or from the script, is also found where the
transcript writes it in Arabic. FAHAD in the greeting and «معك فهد» later in the call are the same name, and
both are masked. This is a reading, not a spelling, so now and then it hides an ordinary word that happens to
fit. The report tells you how often: it searches every row for another row's name, which has no reason to be
there, and counts what that finds.

All of this needs transcripts with speaker lines (`Assistant:` and `User:`, `agent:` and `customer:`,
`العميل:` and `الموظف:`, `SPEAKER_00:`) and at least 200 conversations. A smaller dataset, or one with no
speaker lines, is scanned the way it was before, and the report says so. It doesn't learn from speech
datasets, where a row is one clip. A name learned from a row is used on that row only, and a script that
always says the name as two words or more isn't found.

On a hundred thousand real sales calls, counted without anyone reading them, the cue-word layer alone found
28 in 100 of the times a scripted greeting said the customer's name. The pass above found the rest.

> [!WARNING]
>
> Outside those cases the cue-word layer is still the weak one. A name said with no cue beside it, not mapped as
> a known value and not in a script's place, isn't found by any of these. A language model can find it: see
> [Asking a model for names](#asking-a-model-for-names). Either way the scan has to be
> [checked](#check-the-scan) before you rely on it.

## Running a scan

**Anonymize** on the card runs it, and so does **Pre-process** on the dataset page, where it follows drafting
so that it reads the new text. All of it runs on the platform and sends your text to no model. A small dataset
is scanned inside the dashboard and the report is there when the button comes back. A large one (more than
5,000 clips or sentences, or more than 500 conversations or decision points) is scanned on the cluster by the
same pipeline that built it: the card says the scan is running and shows the report when it's done, and you
can keep annotating meanwhile.

The card has two more options, both off by default and off again every time you open the page: **Also ask a
model on your own cluster to find names** and **Also ask an outside model to find names**. They're described
in [Asking a model for names](#asking-a-model-for-names). Only one of them can be on.

From a terminal the scan is `levers9 dataset pii scan`. It follows a scan that runs on the cluster and prints
the report when it's done: rows, rows with findings, findings by type and by how each was found, what the scan
[learned from the dataset as a whole](#names-the-dataset-gives-away), and where the text went. Where the scan
learned something, the report also tells the rows with somebody's personal data from the rows where the caller
only says its own name.

```bash theme={null}
levers9 dataset pii scan calls-sept             # --no-wait starts a big one and comes straight back
levers9 dataset pii report calls-sept           # the last report again; --json for the API's answer
levers9 dataset pii scan calls-sept --names-model names-qwen     # also ask a model on your own cluster for names
levers9 dataset pii scan demo-calls --outside-model
```

`--names-model` and `--outside-model` are the two boxes on the card, and a scan takes one of them at most.
`--names-model` asks a model you serve on levers9: the text stays on your cluster and is masked before the
model reads it. `--outside-model` asks Anthropic or OpenAI, which is sent the rows as they are, so the command
prints one warning line first. Leave it off for real customer data. Both are described in
[Asking a model for names](#asking-a-model-for-names). With neither, the command asks no model.

Over the API the scan is `POST /api/datasets/{id}/pii/scan` (`POST /api/datasets/{id}/anonymize` is the same
route under its first name), and `GET /api/datasets/{id}/pii` returns the last report and how the latest scan
is going. `{"runner": "job"}` sends a scan to the cluster whatever the size (`--on-cluster` from a terminal).

## Asking a model for names

The layers above all miss one kind of name: a name said with nothing beside it. «فهد، وش عندكم؟» holds a name
to anyone who reads it, and to none of the rules. A language model reads it the way a person does. A scan can
ask one, in one of two ways.

| | a model on your own cluster | an outside model |
| - | - | - |
| what it is | a language model you serve on levers9, from your [Endpoints](https://levers9.mintlify.site/dashboard/endpoints.md) page | Anthropic or OpenAI, whichever the dashboard has a key for |
| what it's sent | each text **after** the scan has masked what it found, so `[PHONE]`, `[ID]` and `[NAME]` stand where the numbers and names already caught were | the text of your rows as it is |
| where the text goes | nowhere new: to your own endpoint, through the platform's own gateway | out of the platform |
| real customer data | yes | no |

### A model on your own cluster

On the card, tick **Also ask a model on your own cluster to find names**, pick one of the models you're
running, and scan. The text stays on your cluster.

From a terminal it's `--names-model` with the endpoint's name as the Endpoints page lists it, or its URL:

```bash theme={null}
levers9 dataset pii scan calls-sept --names-model names-qwen
levers9 dataset pii scan calls-sept --names-model last          # the model that read this dataset last
```

The command says before it asks that the text stays on your cluster, and the report it prints says so again
from what the scan recorded: which model read the text, how many texts it read and gave no answer for, the
names it returned, kept and dropped (and why), and the findings it added that no other layer had.

The model comes last, and here's what happens:

1. The scan runs as it always does: known values, numbers, spoken numbers, cue words, and what the dataset
   gives away.
2. Each text is masked with what that found, and the masked text is what the model reads. A phone number, an
   ID or a name that was already found is never sent, even to your own cluster.
3. The model answers with a list of the names it sees.
4. Each answer is checked before it's believed (see below). A name that passes is masked wherever it stands in
   that text. It's also looked for in the row's other texts the way a known name is, so a name the model reads
   once in the call is hidden in the draft and in what a person wrote for that row too.

The model's findings never change another layer's. Where the two meet, the other layer's finding stays as it
is and the model's keeps only the part nobody else covered. So a finding marked `model` is exactly what the
model added.

A model makes things up, and it takes a district for a person. So an answer is kept only if it stands in the
text it was asked about, character for character, as a word of its own; is four words or fewer; isn't a
placeholder like `[NAME]`; isn't a city, or something that starts with a word like `شركة`, `حي` or `مشروع`;
and isn't made of words that are never a name, of the dataset's ordinary words or of the script's own words.
Everything dropped is counted in the report by the reason it was dropped for. A common first name is never
dropped for being said often.

**Picking a model.** Any chat model you serve on levers9 that reads Arabic works: deploy one from the
Endpoints page or with `levers9 deploy --llm`. The scan sends up to 4,000 characters of text and a prompt of
about a page in one request, so the model needs a context of 8,000 tokens or more. Bigger models read
dialect better and answer slower. We haven't measured which model finds the most names, so start with the
largest one you already run and [check it](#check-the-scan).

**What it costs in time.** The model reads every text, so this is much slower than a plain scan: one request
for each text, and more for a long one, which is read in pieces of 4,000 characters that overlap a little. A
small dataset, scanned inside the dashboard, sends 8 requests at a time. A large one is scanned on the cluster
500 rows to a container, four containers at a time, so the endpoint gets about 32 requests at once however
big the dataset is. Plan for a second or so a text on one GPU: minutes for a few hundred calls, hours for a
hundred thousand. That figure isn't measured yet; the card shows how far the scan is.

If the endpoint is idle (scaled to zero), a scan on the cluster waits up to ten minutes for it to start. A
scan inside the dashboard waits a minute and a half and then says the model didn't answer; send the endpoint
one request first, or run the scan on the cluster: `--on-cluster` from a terminal, `{"runner": "job"}` over the API.

**When a request fails.** It's tried twice more. After that the text keeps what the other layers found, and
the report says how many texts got no answer. If the model answers nothing at all, the scan stops with an
error and the files of the scan before it are left as they were. A wrong address or a model the endpoint
doesn't serve fails on the first request, before any row is read.

**It still has to be checked.** A model misses names too, and the checks above drop a few real ones: a
one-word name that's written like an ordinary word, as «علي» is like «على», is dropped. So review a sample
after a scan with a model as you would after any scan. The review says which model the scan it measured
asked, so you can scan once without a model and once with one, against the same reviews, and see what the
model added to recall for names. Do it in that order: a plain scan made after a scan with the model keeps
the model's names, and the review says so ("asked no model, and kept the names names-qwen found on 7 Oct"),
so its numbers are the model's and not a scan without one.

### What a later scan keeps

Most scans ask no model: a plain **Scan again**, the scan in **Pre-process**, the one that follows a build or
a draft run under Mask, the one that choosing Mask starts, and `levers9 dataset pii scan` with no option. None
of them gives a reader back a name a model had hidden.

Every scan reads what the scan before it found, and for each text that hasn't changed it keeps the findings a
model made, beside what it finds now. The masked text of an unchanged row is the same after the scan as
before it, to the letter. This needs no model and no network, so it can't fail, and it works the same way
whether the scan runs inside the dashboard or on the cluster.

Three things to know about it:

* **A text that changed keeps nothing.** A draft written again or a label typed since is another text, and
  the places of the old findings mean nothing in it. A scan without the model reads it with the other layers
  only. So does a row that's new. The card says how many rows that is: "412 rows keep names a model found on
  7 Oct. 38 rows are new or changed since then and no model has read them. Scan with the model again to cover
  them."
* **A scan with the model starts over for the texts it reads.** Its new answer replaces what a model found
  there before. For a text it got no answer for, it keeps what was there.
* **A known value still wins.** If the import now knows the name, or another layer now finds it, the finding
  is that layer's and is counted once.

What a reviewer settled still comes before all of this, as it always has.

The scans that start on their own never ask a model, even on a dataset a model has read. They'd be starting
a GPU endpoint nobody asked for, and failing whenever it's asleep. Keeping the earlier findings is what
protects the reader until you scan with the model again.

**The dataset remembers its model.** When a scan with a model on your own cluster ends well, the dataset
keeps which endpoint and model it was (`pii.names_model` in its config and in `meta.json`, beside the
policy). The card then says "Last read by names-qwen on 7 Oct" under the choice and picks that model first
when you tick the box. That's all it does: no scan asks a model because one is remembered. An outside model
is never remembered.

From a terminal, `levers9 dataset pii report` and `levers9 dataset pii policy` print the same two things: how
many rows keep a model's names and how many no model has read, and which model read the dataset last.
`levers9 dataset pii scan NAME --names-model last` asks that model again, which is how you cover the rows that
are new or changed.

### An outside model

**Also ask an outside model to find names** sends the text of your rows, as it is, to Anthropic or OpenAI.
Leave it off for real customer data.

### The request and the report

A model on an endpoint is `names_model` in the body of the scan:

```json theme={null}
{"names_model": {"type": "endpoint", "endpoint": "https://<your gateway>/asgi/<name>/v1/v1", "model": "<the name it serves under>"}}
```

`endpoint` is the endpoint's URL with `/v1` at the end (the base an OpenAI client would use), or just its name
as the Endpoints page lists it. `model` can be left out for an endpoint the page lists.

The endpoint counts as your own when it's one of the endpoints your workspace serves, as the Endpoints page
lists them: you name it, or you give a URL that is that endpoint's. Then it needs nothing else, and it's
called with your workspace's token. Being on the platform's gateway isn't enough by itself: the gateway serves
every workspace's endpoints, and your token and text go to yours only. If the list of your endpoints can't be
read at that moment, the scan is refused and nothing is sent. Any other address is an outside model: without `"allow_external": true` it isn't asked, nothing is sent, the scan still runs, and the report
says in `names_model_skipped` that sending text outside was not allowed. With it, the endpoint is asked from
the dashboard, reads the same masked text, and is sent no token. Because it's asked from inside the cluster, its
address is checked first: it has to be https and public, never a private, loopback or link-local address or the
cluster's own. Once the dashboard's operator turns that check on (`LEVERS9_EGRESS_GUARD=enforce`) such an address
is refused with a 422 that says why, and nothing is sent; until then it's only logged. From a terminal that yes is
`--allow-outside`, beside `--names-model URL` and `--model NAME`: the command prints one warning line first,
and without the flag an address that isn't yours is never asked.

Anthropic or OpenAI takes two fields, and both have to be true: `{"use_model": true, "allow_external":
true}`. With `use_model` alone nothing is sent and the report says so. A key on the dashboard changes nothing
by itself. `names_model` and `use_model` in one request is refused with 422: a scan asks one model.

The report says what happened:

| key | what it says |
| - | - |
| `names_model` | which model read the text in this scan and where it runs: `endpoint:<name>` for one on your own cluster, `outside:<name>` for an endpoint anywhere else, `anthropic` or `openai`, or nothing |
| `names_model_sent` | what that model was sent: `masked` for a model on an endpoint, `original` for Anthropic or OpenAI, nothing when no model was asked |
| `text_left_the_platform` | `true` when text of the dataset left the platform in this scan: any model but one on your own cluster. `false` for no model and for `endpoint:<name>` |
| `names_model_skipped` | why a model that was asked for wasn't used |
| `names_by_model` | for a model on an endpoint, in numbers: `texts` asked about, `texts_failed`, `texts_skipped` (nothing left to read once masked), `requests`, `requests_failed`, `names_returned`, `names_kept`, `names_dropped` by reason, and `findings_added`, the findings on something no other layer had found |
| `names_by_model.carried` | what this scan kept of an earlier scan's model findings: `texts`, the `rows` they're in, `findings`, and whose they are when that's known (`model`, and `at`, when the scan that asked it ran) |
| `names_by_model.never_read` | `texts` no model has read as they are now, and the `rows` that hold one: new or changed since, or all of them when no model was ever asked |

```json theme={null}
{"names_model": null, "names_model_sent": null, "text_left_the_platform": false,
 "names_by_model": {"state": "off", "where": null, "texts": 0, "texts_failed": 0, "texts_skipped": 0, "requests": 0, "requests_failed": 0,
                    "names_returned": 0, "names_kept": 0, "names_dropped": {}, "findings_added": 0,
                    "carried": {"texts": 520, "rows": 412, "findings": 610, "model": "endpoint:names-qwen", "at": "2026-10-07T09:00:00Z"},
                    "never_read": {"texts": 41, "rows": 38}}}
```

That's a plain scan made after a scan with a model: it asked nobody, sent nothing anywhere, and kept 610
findings a model had made in 412 rows. `text_left_the_platform` is about this scan only; what an earlier scan
sent is in that scan's report. `GET /api/datasets/{id}/pii` adds `names_model_sent` and
`text_left_the_platform` to a report made before those keys existed, read off its `names_model` the same way,
so every report answers the question alike.

Findings a model made have `source` `model` in `spans.jsonl`, and the line of a text a model has read says
which model (`model`). No name the model gave, and nothing else it wrote, is in any file.

## Who reads the text: Off, Flag and Mask

A scan on its own changes what nobody reads. What a reader gets is the dataset's **policy**:

| policy | what it does |
| - | - |
| **Off** | Everyone who can open the dataset reads the text as it is. Every dataset starts here. |
| **Flag rows with personal data** | Nothing is hidden. Each row in the workspace, yours and a link's, says how many findings it has, and the TSV export gains two columns, `pii_findings` and `pii_types`. |
| **Mask personal data for everyone but me** | Every reader except you, at the dataset's own pages, gets the masked text and never the original. |

Under **Mask**, these read the masked text:

* a [share link](https://levers9.mintlify.site/lab/sharing.md): its rows, the labels it shows back, its TSV export and, for a decision
  dataset, its gold set;
* the manifest of a [published](https://levers9.mintlify.site/lab/publishing.md) dataset;
* the model that drafts in **Pre-process**: a language model, or the Decision API;
* [Insights](https://levers9.mintlify.site/lab/insights.md): every text column of the dataset's table;
* a dataset built from this one (the **Existing Lab dataset** source), and the calls the RL Arena takes from it.

These still read the original:

* **You.** Signed in, you read the text as it is on the dataset page and in the label log, and your own
  **Export TSV** and gold set hold the original under every policy. Add `?text=masked` to either export for
  the masked copy, which is the file a link gets. The card has a link for it.
* **A reviewer of the sample.** Someone checking the scan has to see what it missed, so the review shows the
  sampled rows as they are. See [Who can review](#who-can-review).

**Mask never falls back to the original.** The masked text is made from the findings over the text as it is
now, and only when that text is the one that was read (a hash of each text is kept with its findings). A draft
that was written again, or a row that was never scanned, reaches those readers as
`held back until the personal data scan has run`. A stage that would turn those words into data stops short of
it: a publish fails before it uploads anything, a row like that isn't drafted, and it's left out of a dataset
built from this one. The card says how many rows are held back, and scanning again releases them.

Three things keep that rare:

* choosing Mask on a dataset that was never scanned starts the scan;
* a build or a draft run of a Mask dataset, started from the dashboard or its API, ends with a scan whether
  or not you asked for one (run from `lab/ingest` by hand, follow it with `--pii`);
* a label is read for personal data as it's saved, by the scan's own rules and with no outside model. So an
  annotator sees their label back masked right away, and it never waits for the next scan. The label is stored
  as it was typed.

An annotator on a link works over masked text. A `[NAME]` they leave in a transcript is what's saved. An
exported TSV of masked text can be imported back safely: a cell nobody edited isn't taken for a label.

From a terminal the policy is `levers9 dataset pii policy`. With no value it prints the policy and how many
rows are flagged and how many aren't scanned yet. Setting Mask on a dataset that was never scanned starts the
scan, and the command follows it and prints its report. `levers9 dataset export` with `--text masked`
downloads the masked copy.

```bash theme={null}
levers9 dataset pii policy calls-sept           # the policy, and how far the scan covers the rows
levers9 dataset pii policy calls-sept mask      # --no-wait leaves a scan it started to run
levers9 dataset export calls-sept --text masked -o calls-sept-masked.tsv
```

Over the API, `PUT /api/datasets/{id}/pii/policy` with `{"policy": "off" | "flag" | "mask"}` sets the policy
and `GET /api/datasets/{id}/pii/policy` reads it. `?coverage=1` adds how many rows are flagged and how many
are held back. The policy is kept in the dataset's config as `pii.policy` and written into `meta.json` on the
volume, which is where the pipeline's containers read it.

## Check the scan

Mask is as good as the scan, and a scan nobody has checked is a guess. The check is a review: people read a
sample of rows as they are, with what the scan found marked, and settle each one. Two things come out of it.
You get numbers for how good the scan is. And every text a person has settled is masked with what they
settled from then on, not with what the scan found.

Everything below is on the card. From a terminal it's `levers9 dataset pii review`:

```bash theme={null}
levers9 dataset pii review draw calls-sept                 # the sample: 300 rows, or --size N; fixed until --redraw
levers9 dataset pii review status calls-sept               # how far the review is, then the table with its ranges
levers9 dataset pii review link calls-sept --for Fatima    # a review link, which shows original text
```

`draw` says so plainly when the dataset hasn't been scanned or already has a sample. `status` prints the
table described under [What the numbers mean](#what-the-numbers-mean) once enough rows are reviewed, and
before that how many more it takes; `--json` gives the API's answer. Above the table it says which names model
the measured scan asked, and whether it kept a model's names from an earlier scan, so two measurements can be
compared. `link` takes `--view-only` and
`--days N`, and prints one warning line before it makes the link. `levers9 dataset links calls-sept` lists a
dataset's links and says which of them open the review. The rows themselves are reviewed on the dataset page
or through a review link.

### Draw a sample

**Draw a sample** on the card picks rows at random from the whole dataset: 300 unless you say otherwise, and
every row when the dataset has fewer. Every row has the same chance, whether the scan found something in it or
not. That matters. A sample taken from the rows the scan flagged could tell you how often a finding is right,
but never how much the scan missed, because a miss is a row it didn't flag.

The sample is fixed once it's drawn, so everyone reviews the same rows and the numbers are of one sample. The
rows are reviewed in the order they were drawn, which means the first fifty reviewed are a random fifty too.
**Draw again** draws a new sample and keeps every review already made.

A dataset has to be scanned before a sample can be drawn.

### Review a row

**Review the sample** opens the rows one at a time. Each text of the row is shown as it is, with the scan's
findings as coloured marks that say their type. For each row you:

* **take off** a mark that isn't personal data (an order number the scan took for a number to hide);
* **change the type** of a mark that's right about the place and wrong about what it is;
* **mark** anything the scan missed: select the text and pick a type;
* **confirm the row**. A row with nothing to find is confirmed as it is, and so is a row with no text.

It works with the mouse or the keyboard. Arrows move through the text and Shift with an arrow selects. A
letter marks the selection (`N` name, `P` phone number, `I` national ID, `B` IBAN, `C` card number, `E` email,
`U` web address, `L` plate, `M` long number), Backspace takes off the mark the caret is in, and Ctrl or ⌘ with
Enter confirms the row. Arabic runs right to left and the page handles that: what you select is what gets
marked, in a line that mixes Arabic, Latin and digits too.

If a text changes while you have its row open (someone saved a label, or a draft was written again), the row
isn't saved. The page shows it as it is now and you confirm it again. A row that changes after it was reviewed
is listed as needing another look.

### Who can review

* **You**, on the dataset page.
* **Someone with a review link.** **Review link** on the card makes one per person. It's a different kind of
  link from the ones in the Share dialog:

  * it shows the sampled rows **as they are**, names and numbers included, even when the policy is Mask;
  * it opens the sample and nothing else: no other row, no audio, no export, no import;
  * the person **always** accepts an agreement before they see a row. If the dataset has an
    [agreement](https://levers9.mintlify.site/lab/sharing.md#nda) written, it's that one, whether or not you require it of other links. If it
    has none, it's a short default confidentiality text;
  * you can take it back on the card at any time.

  Hand a review link only to someone you'd let read those rows.

### What the numbers mean

Once enough rows are reviewed, the card shows a table: one line per type, and one for all types together.

| column | what it says |
| - | - |
| **Precision** | Of what the scan marked, how much really was personal data of that type. Low precision means the scan hides things it didn't need to. |
| **Recall** | Of the personal data that was there, how much the scan found. Low recall means names and numbers get through. |
| **Right** | findings a reviewer kept |
| **Wrong** | findings a reviewer took off or gave another type |
| **Missed** | things a reviewer marked that the scan hadn't |

Recall is the one that matters for Mask: what the scan misses is what a link is shown.

**Every number comes with a range, and the range is the honest part.** The sample is a few hundred rows out of
many, so 80% in the sample doesn't mean 80% in the dataset. The range (a 95% Wilson interval) is where the
true figure most likely is: if you drew and reviewed samples like this one again and again, about 95 in 100 of
the ranges would hold it.

How wide the range is depends on how many findings the rate was worked out from, not on how many rows you
reviewed. If the scan found 28 of 100 names that were really there, recall is 28% and the range is 20% to 37%.
If it found 3 of 10, you'd also read about 30%, but the range is 11% to 60%: a sample that small can't tell a
scan that misses half the names from one that misses nine in ten. So a sample of 300 rows with only ten IBANs
in it measures phone numbers well and IBANs hardly at all. If a range is too wide to act on, draw a bigger
sample.

A few more things to know when you read the table:

* **The numbers are of the scan, not of the review.** They compare what the scan found on its own with what
  reviewers settled. After a new scan they're worked out again against the same reviews, so you can see
  whether a change to the scan helped. The line under the table says which model, if any, that scan asked
  for names, and whether it kept names a model found in an earlier scan, so a scan with a model and one
  without can be told apart.
* **A wrong type counts twice.** A national ID the scan called a phone number is one wrong for "Phone number"
  and one missed for "National ID". The line under the table counts it as found, because under Mask it's
  hidden either way (as `[PHONE]` where `[ID]` would be right). That line is the share of personal data Mask
  actually hides.
* **A finding counts as right when it overlaps the reviewed one** and has the same type. A mark one word too
  wide still hides the number.
* **Not measured yet.** The table appears when 50 sampled rows are reviewed, or all of them when the sample is
  smaller. Before that the card says how many are done.
* A text that changed after its review, or one the scan hasn't read as it is now, isn't counted.

### What a review changes right away

For a text a person has reviewed, what they settled is the truth. From the moment a row is confirmed, every
reader of masked text gets those texts masked with the reviewed findings: a share link's rows and exports, a
published manifest, the model that drafts, Insights, a dataset built from this one. A name the scan missed and
a reviewer marked is `[NAME]` for all of them. A finding the reviewer took off is no longer hidden.

A later scan doesn't undo this. The scan writes its own findings again, the reviews stay where they are, and
the reviews still win. A review holds for the text it was made on: if that text is edited afterwards, the
review no longer applies to it, and the new text is held back until it's scanned or reviewed again.

Reviewing only fixes the rows in the sample. It tells you about the rest of the dataset; it doesn't clean it.

### Over the API

| route | what it does |
| - | - |
| `POST /api/datasets/{id}/pii/review` | draw the sample. Body: `{"size": 300, "seed": 7, "redraw": false}`, all optional. 409 when the dataset hasn't been scanned, or when a sample exists and `redraw` isn't set |
| `GET /api/datasets/{id}/pii/review` | the measurement: `sample` (size, reviewed, by whom), `by_type`, `all` and `any_type` with `tp`, `fp`, `fn`, `precision`, `recall` and their ranges (`precision_ci`, `recall_ci`), `rows_with_nothing`, `measured` with a `reason` when it's false, and `scan`: the report version and time of the scan that was measured, `names_model`, the model it asked for names (empty when it asked none), and `names_carried`, the model findings it kept from an earlier scan (`model`, `at`, `texts`, `findings`; empty when it kept none) |
| `GET /api/datasets/{id}/pii/review/rows` | a page of the sample to review (`offset`, `limit`): each row's texts as they are, with `sha256`, the scan's findings (`scan`) and what was reviewed before (`reviewed`) |
| `POST /api/datasets/{id}/pii/review/rows` | confirm one row: `{"row_id": "…", "texts": [{"field": "draft_text", "sha256": "…", "spans": [{"start": 4, "end": 9, "type": "name"}]}]}`. Every text of the row has to be there |
| `GET /api/share/{token}/pii/review` | the same page, for a review link |
| `POST /api/share/{token}/pii/review` | confirm one row through a review link |

Offsets count characters from the start of the text, the way Python does: an emoji is one character. A type is
one of `name`, `phone`, `national_id`, `iban`, `card`, `email`, `url`, `plate`, `number`. A review is refused
with 422 when a finding falls outside the text, two findings overlap or a type isn't known, and with 409
(`this text changed since you opened it`) when the `sha256` you send is no longer the text's.

A review link is made with `POST /api/datasets/{id}/shares` and `{"label": "Fatima", "task": "pii"}`. The
other routes of a link (rows, labels, files, exports, import) answer 403 for it, and the two review routes
answer 403 for any other link.

## Export for training

Mask decides what a reader of the dataset is shown. Training is another question: you want a file you can hand
to a training run, with the personal data taken out of it for good. **Export for training** on the card writes
that file, and a report beside it that says how the file was made and how well the scan behind it was checked.

The export is made from the findings a share link is masked with today: what a reviewer settled comes first,
then what the scan found, the names a model found and the ones a later scan kept among them. So an export never
shows more than a link does. It also means an export is exactly as good as the scan, which is why its report
opens with whether anybody measured the scan.

From a terminal it's `levers9 dataset pii export`:

```bash theme={null}
levers9 dataset pii export calls-sept --require-measured -o calls-sept.jsonl    # made-up names and numbers
levers9 dataset pii export calls-sept --render mask                             # placeholders
levers9 dataset pii exports calls-sept                                          # what was made, newest first
levers9 dataset pii export-delete calls-sept deid-surrogate-20261007T101500Z.jsonl
```

The first thing `export` prints is whether the scan was measured, then the recall for names with its range,
the rows it wrote and the ones it left out with why, where the file and its report are on the volume, and what
the file doesn't cover, in the report's own sentences. It reads the review before it starts, so the measurement
it copies is of the scan as it stands. `-o FILE` also saves the file where you say, with its report beside it
(`calls-sept.report.json` beside `calls-sept.jsonl`), and never writes over a file unless you add `--force`.
The options are the request's: `--status`, `--split`, `--leave-out-held-back`, `--require-measured`,
`--min-recall name=0.98`, `--by point`, and `--new-salt`, which asks before it acts. A large dataset is written
on the cluster and the command follows it; `--no-wait` comes straight back.

### Two ways to write a finding out

| | Made-up names and numbers | Placeholders |
| - | - | - |
| in the API | `"render": "surrogate"`, the default | `"render": "mask"` |
| what a finding becomes | a made-up value of the same kind, written the way the original was | its placeholder: `[NAME]`, `[PHONE]`, `[ID]` |
| «معك نورة، رقمي 0500000001» | «معك ريما، رقمي 0500047213» | «معك \[NAME]، رقمي \[PHONE]» |
| what a model learns from it | to say names and numbers the way people do, and none of them is anybody's | to say `[NAME]` |
| pick it for | a model that talks, writes or transcribes | a model that should write placeholders itself, or a file people will read |

A made-up value follows three rules:

* **The same person keeps the same made-up name all through a call.** Every row with the same `call_id` is one
  call, and a row with no call id is a call of its own. A number said three times in a call is the same made-up
  number three times, in the groups it was said in each time (`+966 50 000 0001` stays four groups). Each word
  of a name has its own made-up word, so a full name, the first name alone and the family name alone are still
  one person. In another call the same person gets another made-up name, so two calls can't be joined by it.
* **It's written the way the original was.** Arabic-Indic digits stay Arabic-Indic. A number that was dictated
  digit by digit stays words. A Latin name in capitals stays in capitals, and «أبو فهد» becomes «أبو» and
  another name.
* **It looks made up to anyone who checks.** A phone is `05000` and five digits, a national ID has `000` after
  its first digit and a check digit that holds, an IBAN belongs to bank `00`, a card starts `000000`, and an
  email or a link is at `example.com`.

Two things to know before you train on made-up values. Names are drawn from a short list, about forty first
names and ten family names in each script, so a model sees far less variety in names than real calls have. And
a name written in Latin letters and the same name written in Arabic get two different made-up names: FAHAD in
a greeting and «فهد» later in the call don't stay one person in the export.

### The salt

A made-up value isn't stored anywhere. It's worked out every time from four things: the dataset's **salt**,
the row's call, the finding's type and the value itself. The salt is 32 random bytes, made the first time a
dataset needs one. It's what keeps anybody from working backwards: without it, nobody can test whether a
made-up name stands for a given real one.

* **Where it's kept.** On the dashboard's own disk, in a file beside its list of datasets
  (`~/.levers9/dashboard/dataset-salts.json`, readable by the dashboard's own user and nobody else). It isn't
  in the dataset's config or its `meta.json`, it's never written to the `datasets` volume, no route of the API
  returns it, and it isn't in an export or its report. A large export is written on the cluster, and that one
  run is handed the salt for as long as it lasts.
* **Why it's kept.** Two exports made under the same salt line up: the same person in the same call has the
  same made-up name in both. The report says when the salt was made (`surrogates.salt_made_at`), so you can
  tell whether two files line up by comparing that.
* **A new one.** `{"new_salt": true}` in the request makes a new salt before the export is written. The answer
  says so under `new_salt`: surrogate exports made before no longer line up with new ones. Do that when you
  want a release nobody can hold against an earlier one. The new salt is made when the export starts, so it
  stays the dataset's salt even if that export is then refused for a row held back.
* **Losing it.** If the salt is lost (the dashboard's disk is replaced, or the dataset is removed from the Data
  Lab and added again), nothing leaks and the exports you already made stay as good as they were. The next
  export gets a new salt and other made-up values, and that's all. Removing a dataset from the Data Lab removes
  its salt.

An export with placeholders needs no salt, and none is made for it.

### The file

An export is two files under the dataset's `exports/` folder on the `datasets` volume, named by the render and
the time in UTC:

```
calls-sept/exports/deid-surrogate-20261007T101500Z.jsonl
calls-sept/exports/deid-surrogate-20261007T101500Z.report.json
```

The `.jsonl` has one line per row, in a shape a trainer of that kind reads as it is:

| kind | a line |
| - | - |
| `llm` | `{id, call_id, split, messages: [{role, content}], reply, pii}` |
| `stt` | `{id, call_id, split, audio_path, text, pii}` |
| `tts` | `{id, split, text, pii}` |
| `decision` | the [gold set](https://levers9.mintlify.site/lab/decision-gold-sets.md#export)'s record, with its texts rendered, and `pii` |

```json theme={null}
{"id": "call-0142", "call_id": "call-0142", "split": "train", "messages": [{"role": "assistant", "content": "السلام عليكم HISHAM معك ريما"}, {"role": "user", "content": "هلا، رقمي 0500047213"}], "reply": "تمام يا HISHAM", "pii": {"findings": 4, "types": ["name", "phone"]}}
```

* **`messages`** are the turns of the conversation, cut at its speaker lines (`Assistant:` and `User:`,
  `agent:` and `customer:`, `العميل:` and `الموظف:`). The side that works for you is written as `assistant`:
  assistant, agent, AI, bot, rep, operator, advisor, consultant, and «الموظف», «المساعد», «الوكيل»,
  «المندوب», «المستشار», «البائع», «المسوق». The other side is `user`: user, customer, client, caller, human,
  and «العميل», «المستخدم», «المتصل», «الزبون». A numbered speaker (`SPEAKER_00`) doesn't say which side it is,
  so the first one to talk in a conversation is written as `assistant` and every other one as `user`, unless a
  speaker of that conversation is already the assistant by its own word. A `system:` line keeps the role
  `system`, and so does anything that stands before the first speaker line. A text with no speaker lines is
  one `user` message. Turns of the same speaker in a row are left as they are, not joined.
* **`reply`** is the reply a person wrote, or the model's draft when nobody wrote one. A row with neither has
  no `reply`.
* **`text`** of a clip is the transcript a person wrote, or the draft when nobody wrote one.
* **`pii`** says how many findings were written out of that line and of which types, so you can count them or
  leave such rows out.
* A decision point's record has its call (`segments`), every event of its history (`journey`) and the
  reviewer's `rationale` and `reviewer_action` rendered. Everything else of it is as the gold set has it.

A row with no text to train on is left out: a clip with neither a transcript nor a draft, a conversation with
no text, a decision point nobody answered. The report counts them.

You can keep only some rows: `"status": "validated"` for the rows a person validated, and
`"splits": ["train", "val"]` for the splits you name.

### The report beside it

The `.report.json` is how the file says of itself how it was made. It holds counts, names of models and
sentences. It holds no value that was found, no made-up value, and nothing of the salt but when it was made.

| key | what it says |
| - | - |
| `measured`, `note`, `names` | first, whether the scan was measured, in one sentence, and how many of the names it found. For a scan nobody checked: `"measured": false`, `"note": "Not measured. Nobody has drawn a sample of this dataset and reviewed it yet."` |
| `render`, `at`, `file`, `dataset`, `kind`, `options` | which render, when, and what was asked for |
| `rows` | `in_dataset`, `written`, `by_split`, and `left_out` by why: `status` and `split` (not asked for), `held_back`, `no_text` or `not_answered` |
| `findings` | how many findings were written out, in how many rows, by type |
| `roles` | for conversations: how many turns each speaker word was written as, and the rule for numbered speakers |
| `surrogates` | for made-up values: what "the same" is scoped to, and `salt_made_at` |
| `scan` | the scan the export rests on: its `version` and `at`, the names model it asked, what it `carried` over from an earlier scan's model, and how many texts no model has read (`never_read`) |
| `review` | the review's measurement as it stands, copied and not worked out again: `measured` and `reason`, precision and recall per type with their ranges (`by_type`), all types together (`all`), the figure that counts a value as found whatever its type (`any_type`), and how many sampled rows were reviewed. Who reviewed them is left out. `of_this_scan` says whether it measured the scan the export rests on |
| `checks` | what was asked of the measurement and how each type stood |
| `audio_deidentified` | always `false` |
| `not_covered` | what the export doesn't cover, in sentences |

The measurement counts only when it's of the scan the export rests on. After a new scan, the review's numbers
are still the earlier scan's until the review is read again, which works them out for the new one. The card
does that before every export, and so does `levers9 dataset pii export`. Over the API, call
`GET /api/datasets/{id}/pii/review` first. An export made in between says "Not measured" and why.

### What it refuses

An export of a scan nobody measured **is made**. It just says so first, in the answer and in the report, and
the file's name is the same as any other's. These are the cases where no file is written, each with a plain
sentence (`409`):

* **The dataset was never scanned.** There are no findings to write it with.
* **Something is writing the dataset**: a build, a draft run, a scan or another export.
* **A row holds text no scan has read as it is now**, because a draft was written again or a label was typed
  since. Nobody knows what's in it, so by default the export stops and says how many rows that is: scan again
  and export after. Or send `"held_back": "leave_out"` and the other rows are written, with the count in the
  report. The text of such a row is never written, and neither are the words a link is shown in its place.
* **`"require_measured": true`** and the scan isn't measured.
* **`"min_recall": {"name": 0.98, "phone": 0.98}`** and the review's recall for one of those types is under
  its number. The number is held against the **lower end** of the 95% range, which is the careful reading:
  93 names found of 100 is 93%, but the range starts at 86%, so `0.9` refuses it. `"by": "point"` holds it
  against the estimate itself. The refusal names each type that fell short with its figure. A type with nothing
  to find in the sample has no recall, so it passes, and the report says so. Besides the types of a finding you
  can name `all` and `any_type`.
* **No row is left to write** after what you asked for.

A small dataset (the same sizes as [the scan](#running-a-scan)) is written inside the dashboard, and the answer
holds the report. A large one is written on the cluster by one container that reads the rows from the volume:
the answer says the export is running, the card shows it when it's done, and a refusal that needs the rows (a
row held back, nothing to write) shows up as the run's state. Both write the same bytes for the same dataset,
salt and time.

### Over the API

| route | what it does |
| - | - |
| `POST /api/datasets/{id}/pii/export` | make an export. Body, all optional: `{"render": "surrogate" \| "mask", "status": "validated", "splits": ["train"], "held_back": "refuse" \| "leave_out", "require_measured": false, "min_recall": {"name": 0.98}, "by": "lower" \| "point", "new_salt": false}`. The answer opens with `measured`, `note` and `names`, then `state`, `file`, `path`, `report_path`, and for a small dataset `rows`, `report` (the report's content) and `download` |
| `GET /api/datasets/{id}/pii/exports` | what was made, newest first: `exports` (each with `path`, `render`, `at`, `rows`, `left_out`, `measured`, `note`, `names`, `download` and `report_download`), `state` (how the latest run is going: running, done, refused, error or interrupted) and `salt_made_at` |
| `DELETE /api/datasets/{id}/pii/exports/{file}` | remove one export, its file and its report |

All three are the owner's. A file is downloaded through the owner's own route for the volume,
`GET /api/volumes/datasets/download?path=<path>`, or from the Storage page. No share link can make an export,
list them or fetch one: a link's file route opens a row's clip and nothing else.

### What an export doesn't cover

* **What the scan didn't find.** It's still in the text. The report says how much the review found that to be,
  or that nobody has measured it.
* **Audio.** `audio_path` in a speech export points at the recording as it was made. A clip still says the
  names and numbers its text no longer has, so in a row with findings the text and the audio don't match. If
  you train speech to text on this file, leave those rows out (`pii.findings` above zero) or train on text you
  don't need the audio for.
* **Ids and call ids.** They're written as they are. Don't use a phone number or a name as one.
* **What's said around a finding.** A made-up name hides the real one, but a rare fact in the text, like a
  job, a street or a date, can still point at a person.
* **What isn't text.** For a decision dataset: event ids, times, sources and types, the answers, the causes
  and the logged decision. The people who answered a record are written as `annotator-1`, `annotator-2` and
  so on, the same person the same within one file, so their names don't travel with it.
* **The folder.** Like every file on the `datasets` volume, an export can be read by anyone in your workspace.

## The files

Everything the scan and the review write is under the dataset's `pii/` folder on the `datasets` volume. None of
these files holds a value that was found: they say where a finding is and what it is. An
[export for training](#the-file) is under `exports/`, beside that folder.

| file | what it holds |
| - | - |
| `pii/spans.jsonl` | the scan's findings, one line per text: the row, the field, a hash of the text, and each finding as `{start, end, type, score, source}`. A line of a text a names model has read as it is now also has `model`, which model that was. The next scan reads this file before it writes its own, to keep the findings a model made |
| `pii/rows.masked.tsv` | `row_id`, `field`, `findings`, `types`, `text`: each text masked with the scan's own findings. It doesn't include what a review settled; for a masked copy that does, use `export.tsv?text=masked` |
| `pii/report.json` | rows scanned, rows with a finding, counts by type, by field and by source, and when. `learned` says what the dataset taught the scan, in numbers only: each place a script puts the name, how many rows gave a name in an answer, how many Latin names were found in Arabic and how much of that is chance, and how many words count as ordinary. `script_names` counts the names the caller itself says in a fifth of all conversations, like its own: they're masked too, and `rows_with_nothing_else` says how many rows hold only that, so you can tell rows with a customer's data from rows where the caller just says who it is. A report with `version` below 4 has neither; scan again to get them. `names_model` and `names_by_model` say which model was asked for names and what came of it (see [the request and the report](#the-request-and-the-report)). A report with `version` below 5 has no `names_by_model`: no model on an endpoint was asked, and a `names_model` it has is Anthropic or OpenAI, which was sent the rows as they are |
| `pii/labels.spans.jsonl` | only under Mask: the same lines as `spans.jsonl`, for the labels saved since the last scan |
| `pii/known.jsonl` | the phone numbers and names each call is known by, from the import. This one does hold values, and nothing but the scan reads it |
| `pii/review/sample.json` | the sampled row ids in the order drawn, the seed, when, and the version of the scan report it was drawn under |
| `pii/review/reviews.jsonl` | what reviewers settled, one line per row and text, only ever added to: `{row_id, field, sha256, chars, spans: [{start, end, type}], user, at}`. The newest line for a row and text is the one that counts |
| `pii/review/report.json` | the measurement, as it was last worked out |

A `field` is a column of the row (`draft_text`, `text`, `verify_text`, `context`, `draft_output`), a label a
person wrote (`label.transcript`, `label.reply`, `label.rationale`, `label.action`), or one text of a journey
event (`journey.<event id>.text`). `source` says how the scan made a finding: `known`, `checksum`, `pattern`,
`spoken`, `context` or `model`.

[Insights](https://levers9.mintlify.site/lab/insights.md) shows three columns from all this: `pii_count`, `pii_types` and `anonymized_text`.
An earlier version of the scan kept the values it found in `pii.jsonl`; the next scan of a dataset removes
that file.

## What Mask doesn't cover

* **What the scan doesn't find.** This is the big one. A name said with no cue is shown, and Mask hides only
  what was found or what a reviewer marked. [Check the scan](#check-the-scan) to learn how much that is. If
  recall is too low for your data, map the
  [known values](https://levers9.mintlify.site/lab/datasets.md#known-values-the-customers-phone-number-and-name) at import, or
  [ask a model on your own cluster for names](#a-model-on-your-own-cluster). A model misses names too, so
  check again after.
* **Audio.** Mask covers text. Audio still plays as recorded: the clip a link plays is the recording, speech
  goes to a transcription model as audio, and a text-to-speech build sends each sentence to its voice as
  written, because the clip is the sentence.
* **What isn't text.** Ids, call ids, a clip's path, times, speaker and role, status and split, and a journey
  event's id, time, source and type are never masked. So don't use a phone number as a call id.
* **The volume.** The policy governs the Lab's own readers, not the files. Anyone in the workspace can still
  open `rows.tsv` and `labels.tsv` on the Storage page, mount the `datasets` volume in a job, or query a file
  as `file:datasets/...` in Insights.
* **What already left.** Calls the RL Arena took from the dataset before you chose Mask, a file published
  before, an export someone downloaded: the policy applies from the moment it's set.
* **The review sample, for its reviewers.** A review link reads the sampled rows as they are. That's a few
  hundred rows, behind an agreement, and it's the one place besides your own pages where the original is shown.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.