> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Import audio or text, optionally pre-label with a model, then annotate by hand

The Data Lab builds datasets for speech, language and decision models, and for labelling sounds by ear. A dataset's life is **import → (optional)
pre-label → annotate**. Everything lands as TSV on the `datasets` volume, so it is readable by any tool, and
the dashboard's **Datasets** page is the front end for the pipeline in `lab/ingest/`.

## Kinds

| kind | rows | what people label | pre-label providers (optional) |
| - | - | - | - |
| `stt` | audio clips, 10 to 40 s, cut from recordings | transcript and speaker role | none (annotate from scratch), Whisper on the cluster, Azure Speech, Google Speech, ElevenLabs Scribe |
| `tts` | text with a synthesised clip; Whisper checks it back and records the word error rate | clip quality | Kokoro on the cluster, ElevenLabs, Azure Neural, Google TTS (a voice is required, the clip is the example) |
| `llm` | conversations | the target reply or a label | none, a vLLM endpoint on the cluster, Anthropic, OpenAI |
| `decision` | decision points of a lead: its journey of past events, the new call, and optionally a decision the engine logged | an answer to each typed question, the events that caused it, how it would change without an event, and a review of the logged decision | none, a Decision API in the workspace |
| `sound` | short audio clips, each with an optional mark on the sound in question | one answer per clip, picked from a list you write | none: a sound is labelled by ear |

The `decision` kind is for a human gold set for a decision model, not for a language model. It has its own
record format, annotation screen and exports: see [Decision gold sets](https://levers9.mintlify.site/lab/decision-gold-sets.md).

The `sound` kind is for questions that can only be answered by listening, such as whether a caller's sound is
an interruption or a "mm-hm". It has its own folder format and annotation screen: see
[Sound datasets](https://levers9.mintlify.site/lab/sound-datasets.md).

## Sample data

**Sample data** on the Datasets page plants a short call with its clips and builds it as an ordinary dataset, so
annotation, sharing and export can be tried before anyone has uploaded a recording. Two sets ship with the
platform: a six-turn Saudi real-estate call in Arabic and a four-turn English support call, both synthetic, with
their exact transcript known.

The clips land on the `datasets` volume under `uploads/<name>` and are chunked like any recording, so nothing
about the result is special: the same rows, the same annotation screen, the same TSV. Tick **Fill the drafts
with the known transcript** to see what pre-labelling gives you; leave it off and every row starts empty, which
is the thing to try first.

## Import

**New dataset** asks for a name, the kind, and a source:

* a **volume** folder (drop recordings in with `levers9 cp` or the Storage page);
* an existing **Lab dataset** (to make a TTS, LLM or decision set from transcribed calls);
* **S3**, **GCS**, **BigQuery**, **Hugging Face**, Mozilla Data Collective, or a **URL list**;
* a **webhook**: your call platform posts items to the dataset's URL with its key, and they collect in a folder
  under `uploads/` on your own `datasets` volume (`uploads/<name>/webhook` unless you name another folder under
  `uploads/`) until you press **Run again**. Items are written to your workspace's volume, never to anyone else's.
  A dataset with no owner on record (one made before datasets carried their workspace, which the dashboard
  couldn't place) keeps receiving the old way until a platform admin gives it to its workspace; once the
  deployment turns on `LEVERS9_WEBHOOK_OWNER_STRICT`, such a dataset answers its sender 409 instead.

Audio is normalised and chunked; optional diarisation splits speakers. Progress streams to the page as the
pipeline runs on the cluster, and the dataset shows its row count, statuses and the config it was built with.
**Cancel** stops a build; **Rerun** builds again with the same config.

## Pre-processing

**Pre-process** on the dataset page runs the passes that prepare rows before anyone annotates them: drafting
every row with a model, finding personal data in them, or both. Both are optional, both can be run again, and
neither overwrites a human label. Asked for together, the scan runs when drafting finishes, so it reads the
new text rather than the text it replaced. You can close the browser meanwhile.

### Pre-labelling

Pre-labelling is an add-on, never a requirement: a provider drafts a first version of every row so annotators
correct instead of typing. Choose it at build time, or later with **Pre-process** on the dataset
page; only rows without a draft are drafted unless you ask to overwrite. Whisper and Kokoro run on the cluster's
GPU pool (pick the GPU); hosted providers run in a CPU container with the credentials below. Drafts never
overwrite a human label.

### Anonymization

The personal-data scan is the other half of **Pre-process**, and can also be run from the Anonymization card.
It reads every text a row has (drafts, what people wrote, a conversation, a journey) and finds phone numbers
written or said out loud, national ID and Iqama numbers, IBANs, card numbers, emails, URLs, plates, long
numbers and names. Each finding becomes a typed placeholder such as `[NAME]` or `[PHONE]`, so the sentence
still reads. Nothing of the dataset is changed, and the scan sends your text to no model unless you tick the
box that says so.

The scan only finds. Four more things follow from it, and they have a page of their own,
[Personal data](https://levers9.mintlify.site/lab/personal-data.md):

* **what it finds and how**, the files it writes and the API: [the scan](https://levers9.mintlify.site/lab/personal-data.md#what-it-finds-and-how);
* **who reads the text as it is**: the policy below;
* **how good it is**: a [review](https://levers9.mintlify.site/lab/personal-data.md#check-the-scan) of a random sample of rows, which gives
  you the scan's precision and recall and fixes the masking of every row reviewed;
* **a file to train on**: [Export for training](https://levers9.mintlify.site/lab/personal-data.md#export-for-training) writes the dataset out
  with every finding swapped for a made-up value or a placeholder, and a report beside it that says how well
  the scan was checked.

### Who reads the text: Off, Flag and Mask

A scan on its own changes what nobody reads. What a reader gets is the dataset's **policy**, which you choose
on the Anonymization card:

| policy | what it does |
| - | - |
| **Off** | Everyone who can open the dataset reads the text as it is. Every dataset starts here. |
| **Flag rows with personal data** | Nothing is hidden. Each row says how many findings the scan has in it. |
| **Mask personal data for everyone but me** | Share links, published files, draft models, Insights and datasets built from this one get the masked text. You still read the original on the dataset page and in your own exports. |

Under Mask a text the scan hasn't read as it is now is held back from those readers and never shown as it
is. Mask covers text: audio still plays as recorded. And Mask is as good as the scan, which misses a name said
with no cue beside it, so [check the scan](https://levers9.mintlify.site/lab/personal-data.md#check-the-scan) before you rely on it.

The whole of it, with what each reader gets and what Mask doesn't cover, is on
[Personal data](https://levers9.mintlify.site/lab/personal-data.md#who-reads-the-text-off-flag-and-mask).

## Formats the import understands

A tabular source is read by `lab/ingest/corpus.py`, which the uploader's preview also uses, so what the dialog
shows you is what the import reads:

| format | what is recognised |
| - | - |
| `.tsv` `.csv` | the delimiter is sniffed (tab, comma, semicolon, pipe), with a header or without one. A quoted cell may run over several lines |
| `.jsonl` `.ndjson` | one object per line; `{"messages": [...]}` chat data becomes a conversation |
| `.json` | an array of objects, or `{"data": [...]}` / `{"rows": [...]}` |
| `.txt` | one sentence, or one audio path, per line |
| `.parquet` | columns from the file's own footer |

Corpora name the same things differently, so known shapes are recognised by name: a NeMo manifest
(`audio_filepath` + `text`), Common Voice (`path` + `sentence`), Hugging Face audiofolder metadata (`file_name` +
`transcription`), LJSpeech's headerless `id|text|normalised`, a SADA-style CSV. Each column gets a suggested
role: audio, transcript, call id, context or speaker.

The suggestion is only where the dialog starts. **Nothing is imported on a guess:** when a folder holds two
exports with different columns, the dialog says so and each gets its own mapping, and you can correct any column
before the build. A manifest that names its own clips is imported with its transcripts, not as bare audio.

A value is imported whole, however long it is. The dialog shows the first 160 characters of each cell, and that
is only the preview. A transcript with a turn per line inside one quoted cell keeps its turns: a row is one
line of a TSV, so each line break becomes a space, and the turns are still found by the role they start with
(`Assistant:`, `User:`, `agent:`, `customer:`).

### Known values: the customer's phone number and name

A call export often has columns that say who the call was with. Map them in the dialog as **Customer's phone
number** and **Customer's name** (the roles `known_phone` and `known_name`). A column named for it, such as
`phone`, `mobile`, `customer_phone`, `customer_name` or `first_name`, is suggested for you.

These columns are not imported as rows. Their values are filed in `pii/known.jsonl`, one line per call, and
the only thing that reads that file is the [personal-data scan](https://levers9.mintlify.site/lab/personal-data.md), which looks for them in
the call's text. They are not shown on any page, not in any export, not sent when you publish, and a share link
can't open the file. A cell may hold several numbers (`0500000001 / 0500000002`), and over the API a role may
name several columns: `"known_name": ["first_name", "last_name"]`.

From a terminal they're two more roles for `--map` on `levers9 dataset create`, and a role given twice names
two columns: `--map known_phone=Mobile --map known_name=first_name --map known_name=last_name`.
`levers9 dataset inspect` prints them with the other flags it suggests.

## A big corpus: upload one archive

Thousands of small uploads is the slow and fragile way in. Put the corpus in a **`.zip`** and upload that
instead — one transfer, which resumes if it breaks:

```bash theme={null}
levers9 dataset upload ./sada-2022.zip --to calls-sept
```

The archive is **not unpacked**. A `.zip` on the volume is read as though it were a folder, so
`uploads/calls-sept/sada-2022.zip/batch_1/6k_SBA_100_0.wav` is just a path: members are listed from the
archive's index and each recording is read when the import reaches it. Nothing is copied, nothing is stored
twice, and the dataset builds straight out of the archive.

Mixing works too — the manifest loose and the audio zipped, or several archives side by side. An audio file
the manifest names is looked for where the manifest says, then beside it, then by name anywhere in the
folder including inside the archives, so an export whose paths do not quite match how it was unpacked still
imports.

> [!WARNING]
>
> A manifest on its own imports nothing: `train.csv` naming `batch_1/…wav` needs those recordings in the
> folder (or in an archive in it). The New dataset dialog says so before you build, and the build says which
> file it could not find rather than failing on a bare path.

## Credentials

Sources and providers read workspace [secrets](https://levers9.mintlify.site/dashboard/storage-and-secrets.md#secrets). Each declares the
variable names it expects; a dataset can map a name to any secret in the workspace, so one client's ElevenLabs
key is never used for another client's build.

## From the command line

`levers9 dataset` does the whole round trip without leaving the terminal — the same datasets the page shows:

```bash theme={null}
levers9 dataset upload ./recordings --to calls-sept      # onto the datasets volume
levers9 dataset inspect uploads/calls-sept               # columns, a sample, a suggested role per column
levers9 dataset create calls-sept --kind stt --path uploads/calls-sept --map text=GroundTruthText
levers9 dataset ls
```

`inspect` prints the `--map` flags it suggests; check the sample, then paste them into `create`. See the
[CLI guide](https://levers9.mintlify.site/guides/cli.md#datasets).

`levers9 dataset export NAME` downloads the rows with their latest labels as TSV, to stdout or to a file with
`-o`. For a decision dataset, `--format jsonl` downloads the gold set and `--format schema` its questions as a
schema ([Decision gold sets](https://levers9.mintlify.site/lab/decision-gold-sets.md#export)).

The pipeline itself also runs from its own folder, which is what both the page and the CLI drive:

```bash theme={null}
cd lab/ingest
python app.py --name calls-sept   --kind stt --source volume --path uploads/calls                      # import + chunk
python app.py --name calls-sept   --prelabel --provider whisper --model large-v3-turbo --gpu L4      # drafts later
python app.py --name calls-voice  --kind tts --source dataset --dataset calls-sept --provider azure --voice ar-SA-HamedNeural
python app.py --name calls-replies --kind llm --source dataset --dataset calls-sept --provider anthropic --model claude-sonnet-5
python app.py --name calls-sept   --publish --bucket my-staging-datasets --only-validated     # send it out
python app.py --name calls-sept   --pii                                                              # find personal data in it
```

`levers9 dataset pii scan calls-sept` runs the personal-data scan through the dashboard, as the dataset page
does, `levers9 dataset pii report calls-sept` prints its last report, and `levers9 dataset pii policy calls-sept`
shows or sets the policy: see [Personal data](https://levers9.mintlify.site/lab/personal-data.md). The
[review of the scan](https://levers9.mintlify.site/lab/personal-data.md#check-the-scan) is `levers9 dataset pii review`: `draw` draws the
sample, `status` prints how far the review is and what it says of the scan, and `link` makes a link for a
reviewer. The rows themselves are reviewed on the dataset page or through that link.
`levers9 dataset export` is the owner's export, so it holds the original text. Add `--text masked` for the
masked copy. A file to train on, with the personal data taken out for good, is
`levers9 dataset pii export`: see [Export for training](https://levers9.mintlify.site/lab/personal-data.md#export-for-training).

## Publishing

A finished dataset can be copied to a Cloud Storage bucket for a training pipeline that lives elsewhere, with a
service-account key held as a workspace secret: [Publishing to a bucket](https://levers9.mintlify.site/lab/publishing.md).

Next: [Annotation](https://levers9.mintlify.site/lab/annotation.md), then [Sharing](https://levers9.mintlify.site/lab/sharing.md) to bring in annotators, [Personal data](https://levers9.mintlify.site/lab/personal-data.md) before the text goes anywhere, and [Insights](https://levers9.mintlify.site/lab/insights.md) to explore what you built.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.