datasets volume, so it is readable by any tool, and
the dashboard’s Datasets page is the front end for the pipeline in lab/ingest/.
Kinds
The
decision kind is for a human gold set for a decision model, not for a language model. It has its own
record format, annotation screen and exports: see Decision gold sets.
The sound kind is for questions that can only be answered by listening, such as whether a caller’s sound is
an interruption or a “mm-hm”. It has its own folder format and annotation screen: see
Sound datasets.
Sample data
Sample data on the Datasets page plants a short call with its clips and builds it as an ordinary dataset, so annotation, sharing and export can be tried before anyone has uploaded a recording. Two sets ship with the platform: a six-turn Saudi real-estate call in Arabic and a four-turn English support call, both synthetic, with their exact transcript known. The clips land on thedatasets volume under uploads/<name> and are chunked like any recording, so nothing
about the result is special: the same rows, the same annotation screen, the same TSV. Tick Fill the drafts
with the known transcript to see what pre-labelling gives you; leave it off and every row starts empty, which
is the thing to try first.
Import
New dataset asks for a name, the kind, and a source:- a volume folder (drop recordings in with
levers9 cpor the Storage page); - an existing Lab dataset (to make a TTS, LLM or decision set from transcribed calls);
- S3, GCS, BigQuery, Hugging Face, Mozilla Data Collective, or a URL list;
- a webhook: your call platform posts items to the dataset’s URL with its key, and they collect in a folder
under
uploads/on your owndatasetsvolume (uploads/<name>/webhookunless you name another folder underuploads/) until you press Run again. Items are written to your workspace’s volume, never to anyone else’s. A dataset with no owner on record (one made before datasets carried their workspace, which the dashboard couldn’t place) keeps receiving the old way until a platform admin gives it to its workspace; once the deployment turns onLEVERS9_WEBHOOK_OWNER_STRICT, such a dataset answers its sender 409 instead.
Pre-processing
Pre-process on the dataset page runs the passes that prepare rows before anyone annotates them: drafting every row with a model, finding personal data in them, or both. Both are optional, both can be run again, and neither overwrites a human label. Asked for together, the scan runs when drafting finishes, so it reads the new text rather than the text it replaced. You can close the browser meanwhile.Pre-labelling
Pre-labelling is an add-on, never a requirement: a provider drafts a first version of every row so annotators correct instead of typing. Choose it at build time, or later with Pre-process on the dataset page; only rows without a draft are drafted unless you ask to overwrite. Whisper and Kokoro run on the cluster’s GPU pool (pick the GPU); hosted providers run in a CPU container with the credentials below. Drafts never overwrite a human label.Anonymization
The personal-data scan is the other half of Pre-process, and can also be run from the Anonymization card. It reads every text a row has (drafts, what people wrote, a conversation, a journey) and finds phone numbers written or said out loud, national ID and Iqama numbers, IBANs, card numbers, emails, URLs, plates, long numbers and names. Each finding becomes a typed placeholder such as[NAME] or [PHONE], so the sentence
still reads. Nothing of the dataset is changed, and the scan sends your text to no model unless you tick the
box that says so.
The scan only finds. Four more things follow from it, and they have a page of their own,
Personal data:
- what it finds and how, the files it writes and the API: the scan;
- who reads the text as it is: the policy below;
- how good it is: a review of a random sample of rows, which gives you the scan’s precision and recall and fixes the masking of every row reviewed;
- a file to train on: Export for training writes the dataset out with every finding swapped for a made-up value or a placeholder, and a report beside it that says how well the scan was checked.
Who reads the text: Off, Flag and Mask
A scan on its own changes what nobody reads. What a reader gets is the dataset’s policy, which you choose on the Anonymization card:
Under Mask a text the scan hasn’t read as it is now is held back from those readers and never shown as it
is. Mask covers text: audio still plays as recorded. And Mask is as good as the scan, which misses a name said
with no cue beside it, so check the scan before you rely on it.
The whole of it, with what each reader gets and what Mask doesn’t cover, is on
Personal data.
Formats the import understands
A tabular source is read bylab/ingest/corpus.py, which the uploader’s preview also uses, so what the dialog
shows you is what the import reads:
Corpora name the same things differently, so known shapes are recognised by name: a NeMo manifest
(
audio_filepath + text), Common Voice (path + sentence), Hugging Face audiofolder metadata (file_name +
transcription), LJSpeech’s headerless id|text|normalised, a SADA-style CSV. Each column gets a suggested
role: audio, transcript, call id, context or speaker.
The suggestion is only where the dialog starts. Nothing is imported on a guess: when a folder holds two
exports with different columns, the dialog says so and each gets its own mapping, and you can correct any column
before the build. A manifest that names its own clips is imported with its transcripts, not as bare audio.
A value is imported whole, however long it is. The dialog shows the first 160 characters of each cell, and that
is only the preview. A transcript with a turn per line inside one quoted cell keeps its turns: a row is one
line of a TSV, so each line break becomes a space, and the turns are still found by the role they start with
(Assistant:, User:, agent:, customer:).
Known values: the customer’s phone number and name
A call export often has columns that say who the call was with. Map them in the dialog as Customer’s phone number and Customer’s name (the rolesknown_phone and known_name). A column named for it, such as
phone, mobile, customer_phone, customer_name or first_name, is suggested for you.
These columns are not imported as rows. Their values are filed in pii/known.jsonl, one line per call, and
the only thing that reads that file is the personal-data scan, which looks for them in
the call’s text. They are not shown on any page, not in any export, not sent when you publish, and a share link
can’t open the file. A cell may hold several numbers (0500000001 / 0500000002), and over the API a role may
name several columns: "known_name": ["first_name", "last_name"].
From a terminal they’re two more roles for --map on levers9 dataset create, and a role given twice names
two columns: --map known_phone=Mobile --map known_name=first_name --map known_name=last_name.
levers9 dataset inspect prints them with the other flags it suggests.
A big corpus: upload one archive
Thousands of small uploads is the slow and fragile way in. Put the corpus in a.zip and upload that
instead — one transfer, which resumes if it breaks:
.zip on the volume is read as though it were a folder, so
uploads/calls-sept/sada-2022.zip/batch_1/6k_SBA_100_0.wav is just a path: members are listed from the
archive’s index and each recording is read when the import reaches it. Nothing is copied, nothing is stored
twice, and the dataset builds straight out of the archive.
Mixing works too — the manifest loose and the audio zipped, or several archives side by side. An audio file
the manifest names is looked for where the manifest says, then beside it, then by name anywhere in the
folder including inside the archives, so an export whose paths do not quite match how it was unpacked still
imports.
Credentials
Sources and providers read workspace secrets. Each declares the variable names it expects; a dataset can map a name to any secret in the workspace, so one client’s ElevenLabs key is never used for another client’s build.From the command line
levers9 dataset does the whole round trip without leaving the terminal — the same datasets the page shows:
inspect prints the --map flags it suggests; check the sample, then paste them into create. See the
CLI guide.
levers9 dataset export NAME downloads the rows with their latest labels as TSV, to stdout or to a file with
-o. For a decision dataset, --format jsonl downloads the gold set and --format schema its questions as a
schema (Decision gold sets).
The pipeline itself also runs from its own folder, which is what both the page and the CLI drive:
levers9 dataset pii scan calls-sept runs the personal-data scan through the dashboard, as the dataset page
does, levers9 dataset pii report calls-sept prints its last report, and levers9 dataset pii policy calls-sept
shows or sets the policy: see Personal data. The
review of the scan is levers9 dataset pii review: draw draws the
sample, status prints how far the review is and what it says of the scan, and link makes a link for a
reviewer. The rows themselves are reviewed on the dataset page or through that link.
levers9 dataset export is the owner’s export, so it holds the original text. Add --text masked for the
masked copy. A file to train on, with the personal data taken out for good, is
levers9 dataset pii export: see Export for training.