Sound datasets

Some questions about audio can only be answered by listening. Is this sound an interruption, or someone saying “mm-hm”? Is this clip speech or noise? Whose voice is it? The sound dataset kind (shown as Sound classification) is for those. A row is one clip. An annotator plays it and picks one answer from a list you write, with a single key. Nothing drafts a sound dataset, and nothing a model said about a clip is shown to the annotator: the labels are meant to be made by ear alone. The one thing shown besides the clip is a row’s text, when you give it one, such as a transcript of the clip.

What you write: the question and its answers

A sound dataset has one question, asked about every clip, and two to nine answers to choose from.

{
  "question": "What is this sound?",
  "options": [
    {"label": "Backchannel", "hint": "The caller shows they are listening and does not want to speak."},
    {"label": "Interruption", "hint": "The caller cuts in to take the turn."},
    {"label": "Noise", "hint": "Not speech: a cough, a bang, line noise."}
  ],
  "listen_s": 1,
  "tracks": {"left": "agent", "right": "caller", "mark": "right"}
}

field	what it is
`question`	what the annotator is asked about each clip
`options`	the answers, in the order of their keys: the first is `1`, the second `2`. Each has a `label` and, optionally, a `hint` shown under it. A `key` is made from the label (`another_voice` from “Another voice”) unless you give one; the key is what is saved
`listen_s`	how many seconds of the sound have to be played before an answer is taken. `1` when left out, `0` to take an answer at once
`tracks`	optional, for stereo clips: what the `left` and `right` channels are, and `mark`, which of the two the marked sound is on (`left` or `right`). Each channel is drawn as a track under its name and can be silenced while listening. The marked one is drawn on top with the box on it; without `mark` the box is on both

What you bring: clips, and optionally a manifest

Put the clips in a folder on the datasets volume. A clip is kept as it is: its channels and its sample rate are what the annotator hears. A clip is usually longer than the sound it is about, so that the annotator hears what came before. A manifest in the same folder says where in each clip the sound in question is, and the page marks that part of the player. It is a .jsonl, .tsv or .csv file with a row per clip:

column	what it is
`audio`	the clip’s file, relative to the manifest. Required
`id`	the row’s id. The file’s name without its extension when left out
`call_id`	what the clip was cut from
`mark_start_s`	where the sound in question starts in the clip, in seconds
`mark_end_s`	where it ends
`split`	`train`, `val` or `test`. Worked out from `call_id` when left out
`text`	optional: what the annotator reads under the clip, such as its transcript. Line breaks are kept, and up to 20,000 characters

{"audio": "clips/s00001.wav", "id": "s00001", "mark_start_s": 1.5, "mark_end_s": 1.93, "split": "train"}
{"audio": "clips/s00002.wav", "id": "s00002", "mark_start_s": 1.5, "mark_end_s": 2.84, "split": "test"}

Rows keep the manifest’s order, and that is the order annotators meet the clips in. Shuffle the manifest if the clips should not arrive grouped by call or by kind. Without a manifest every audio file in the folder is a row, in name order, with no mark.

Keep what a model or a rule said about each clip out of the manifest, the ids and the text. A row’s columns are sent to the annotator’s page. Keep that table elsewhere and join it to the answers by row id afterwards.

Build it

In the dashboard: Datasets → New dataset → Sound classification, pick the folder, then write the question and the answers, one per line as Label | hint. From a terminal, with the task in a file:

levers9 dataset upload ./clips --to interruptions
levers9 dataset create interruptions --kind sound --path uploads/interruptions --task task.json

The annotation screen

The clip is drawn as its tracks, one for each channel: how loud each side is from moment to moment, with a white box on the sound in question and a ruler that counts seconds from where that sound starts. On a call recording that is the caller on one track and the agent on the other, so the annotator sees what the other side was doing when the sound was made. A click on the picture plays from there. Under the picture is the row’s text, if it has one. Each line is shown in its own direction, so an Arabic line reads right to left next to a line such as PAUSE at 3.84 s, and a long text scrolls rather than being cut. The clip plays as soon as a row opens (autoplay is on for this kind, and each annotator can turn it off). The next three clips are fetched while the open one is listened to, so moving on does not wait on the network, and a clip gone back to is not fetched again. Play plays it again from its start, Only the sound plays the marked part with a third of a second either side, and under Hear each track can be silenced: with one track on, it plays in both ears, which is how a quiet sound under a loud voice is heard. An answer is taken only once the sound has been played from its start for listen_s seconds, or to its end when it is shorter. A play that starts inside the sound or after it does not count. An answer already given can be changed without listening again. Picking an answer saves it, sets the row to validated and moves to the next clip, so a clip costs one key. Above the picture is how often each answer has been given so far in the whole dataset.

keys	action
`1` to `9`	pick that answer and go to the next clip
`Space`	play the clip again from its start, or pause it
`X`	play only the sound
a track’s first letter	silence that track, or bring it back: `C` and `A` for a caller and an agent. A track has a key when its name starts with a Latin letter that the other track and the keys in this table do not use
`←` `→`	back to the previous clip, skip to the next
`R`	play from the start
`Backspace`	take this clip’s answer back; the row is to do again
`⌘ .` (Ctrl on Windows)	flag a clip that is broken or cannot be judged at all, and go on

Every key works on an Arabic keyboard layout too: it is read by its place on the keyboard as well as by the character it types. If “cannot tell” is a real answer for your question, make it one of the options. Flag is for a clip that did not play.

Reword the task later

The question, the options’ labels and hints, the listening time and the tracks can be changed after the dataset is built, while people are labelling. The options keep their keys and their order, because the answers already given are stored by key and picked by place:

curl -X PUT "$LEVERS9_API/api/datasets/$ID/task" -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" --data @task.json        # {"task": {...}}, each option with its "key"

A task that drops, adds, renames or reorders an option is refused. No row and no answer is touched, and an annotator’s open link shows the new wording when they next load it.

Give the rows a text later

A built dataset’s rows can be given a text, or a new one, without building it again. Write a file with a JSON line per row:

{"id": "s00001", "text": "0.00 caller: ...\nPAUSE at 3.84 s"}
{"id": "s00002", "text": ""}

levers9 dataset set-text interruptions texts.jsonl

The API takes the same as {"texts": [...]}:

curl -X PATCH "$LEVERS9_API/api/datasets/$ID/text" -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" --data '{"texts": [{"id": "s00001", "text": "..."}]}'

Only the rows in the file change, and an empty text takes a row’s text away. The clips, the ids, their order, the task and every answer stay as they are. Every id must be a row of the dataset and appear once: otherwise nothing is written and the answer names the ids it did not know. It counts the rows given, how many changed and how many rows now have a text. Under the mask personal data policy the text is scanned like a transcript, and a link shows it masked. A text set after the last scan is held back from links until you scan again (Personal data). A sound dataset is shared like any other: Share gives each annotator a link that opens this dataset’s annotation screen and nothing else, with no account, and behind an agreement if the dataset has one (Sharing). The link plays this dataset’s clips and no other file on the volume. Each answer is recorded with the annotator’s name.

Check how steady the answers are

Two things you can do with the manifest alone:

Repeats. List some clips twice, under different ids and far apart. An annotator who gives the same clip two different answers tells you how far to trust a single answer.
Overlap. Put clips you have already labelled yourself among the new ones. Their answers measure how well the annotator agrees with you.

In both cases the ids must not give it away, and the table that says which rows are repeats stays with you.

Bring answers you already have

Clips labelled somewhere else, on a page of your own or in a spreadsheet, keep their answers. Build the dataset from the clips, then import a TSV with a row per answered clip: id, choice (the key of the answer) and validated (1). Import TSV on the dataset page takes it, and so does the API, which also records whose answers they are:

curl -X POST "$LEVERS9_API/api/datasets/$ID/import?user=alfian" \
     -H "Authorization: Bearer $TOKEN" --data-binary @answers.tsv

Every answer in the file is checked against the dataset’s options before anything is written. One that is not an option refuses the whole file, with the row it is on. A row the file leaves out, or leaves blank, stays as it is, and the same file sent twice adds nothing.

Get the answers out

Export TSV (or levers9 dataset export NAME) has a row per clip with its choice (the key of the answer picked), and validated and flagged. Its text column holds each row’s text on one line, with every line break written as \n and a backslash as \\. labels.tsv on the volume is the full log: every answer with who gave it and when, including answers that were later changed.

Not yet

Publishing to a bucket. The published manifest is built around transcripts. Export the dataset instead.
Pre-labelling. Nothing drafts a sound dataset.
Two annotators on the same clip. A row holds one current answer: a second annotator’s answer replaces the first as the row’s answer, though both stay in labels.tsv. To compare two people, give each their own dataset built from the same folder.

Get started

Build with the SDK

Dashboard

Data Lab

RL Arena

Operate

What you write: the question and its answers

What you bring: clips, and optionally a manifest

Build it

The annotation screen

Reword the task later

Give the rows a text later

Check how steady the answers are

Bring answers you already have

Get the answers out

Not yet

​What you write: the question and its answers

​What you bring: clips, and optionally a manifest

​Build it

​The annotation screen

​Reword the task later

​Give the rows a text later

​Share it with annotators

​Check how steady the answers are

​Bring answers you already have

​Get the answers out

​Not yet

What you write: the question and its answers

What you bring: clips, and optionally a manifest

Build it

The annotation screen

Reword the task later

Give the rows a text later

Share it with annotators

Check how steady the answers are

Bring answers you already have

Get the answers out

Not yet