> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Publishing to a bucket

> Send a finished dataset to Google Cloud Storage with a service-account key held as a workspace secret

A dataset that people have annotated usually has to reach a training pipeline that lives in a cloud project.
**Publish** copies one dataset out of the Lab into a Cloud Storage bucket: the recordings, and a manifest that
names them. Nothing on the `datasets` volume changes, and the dataset stays annotatable afterwards.

A `decision` dataset is the exception. It holds no recordings and no transcript manifest, so it is not
published to a bucket: download its gold set instead ([Decision gold sets](https://levers9.mintlify.site/lab/decision-gold-sets.md#export)).

There are two kinds of destination, and the difference is only where the credential lives.

## A bucket Levers operates — nothing to set up

Pick it and publish. No key, no secret, no first-time setup: the dashboard holds the credential and signs
**one short-lived upload URL per object**, and the pipeline container uploads straight to Cloud Storage with
those. A platform admin configures which buckets appear (`LEVERS9_PUBLISH_DESTINATIONS` on the dashboard).

This is deliberate, not just convenient. A workspace secret is readable by whoever owns that workspace — they
can attach it to a job of their own and print it — so a key for a bucket holding *every* tenant's training data
must not live there. What the container gets instead is a ticket scoped to one dataset's folder: it cannot
write to another dataset, cannot climb out of its folder, and stops working when the run is over.

```bash theme={null}
levers9 dataset publish calls-sept --to staging --only-validated
```

A shared bucket is one set of folders for every workspace, so a folder belongs to the first workspace that
publishes to it. Publishing again from that workspace works as before. Another workspace that picks the same
folder name gets a 409 saying the folder belongs to another workspace; pick another folder name with `--folder`
(or in the Publish dialog). Folders published before owners were recorded went to the workspace that published
them first, and a platform admin can hand a folder to another workspace from **Admin → unowned**.

The tickets are signed with the dashboard's `LEVERS9_AUTH_SECRET`. A dashboard without one (a local one with
sign-in off) offers no shared buckets at all, and says so if you ask for one.

## Your own bucket — a key you control

For a client's project, or anywhere Levers does not operate the bucket, the credential is a
**service-account key stored as a workspace secret**. It is read inside the pipeline container that does the
upload, never by the dashboard, and never written to the dataset's config — the config records only which
secret to read. The rest of this page is that path.

### 1. A service account that may write to the bucket

Give it object access to that one bucket, nothing else:

```bash theme={null}
PROJECT=my-project-staging
BUCKET=my-staging-datasets

gcloud iam service-accounts create levers-lab-publisher \
  --project "$PROJECT" --display-name "Levers Lab dataset publisher"

# object read/write on this bucket only — no project-wide storage role
gcloud storage buckets add-iam-policy-binding "gs://$BUCKET" \
  --member "serviceAccount:levers-lab-publisher@$PROJECT.iam.gserviceaccount.com" \
  --role roles/storage.objectAdmin

gcloud iam service-accounts keys create /tmp/levers-lab-publisher.json \
  --iam-account "levers-lab-publisher@$PROJECT.iam.gserviceaccount.com"
```

`objectAdmin` rather than `objectCreator` because re-publishing overwrites the manifest. If nothing may ever be
overwritten, use `objectCreator` and publish with a fresh folder name each time.

### 2. The key as a workspace secret

The whole key file is the secret's value, under the name the pipeline reads, `SERVICE_ACCOUNT_JSON`:

```bash theme={null}
levers9 secret create SERVICE_ACCOUNT_JSON "$(cat /tmp/levers-lab-publisher.json)"
rm /tmp/levers-lab-publisher.json
```

Or paste it on the dashboard's **Secrets** page. Either way, delete the local file afterwards.

> [!NOTE]
>
> Publishing to more than one project? Store each key under its own name (`STAGING_SA`,
> `ACME_PROD_SA`) and bind it per dataset with `secret_map`, exactly as sources and providers do. The pipeline
> still asks for `SERVICE_ACCOUNT_JSON`; the map says which secret answers.

### 3. Publish

From the dataset page, **Publish** asks for the bucket, a folder name (the dataset's name by default), which
secret holds the key, and whether to send only the rows a human has validated.

From a terminal, with the CLI:

```bash theme={null}
levers9 dataset publish calls-sept --bucket my-staging-datasets                    # everything
levers9 dataset publish calls-sept --bucket my-staging-datasets --only-validated   # signed-off rows only
levers9 dataset publish calls-sept --bucket my-staging-datasets --folder calls-september-2026
levers9 dataset publish calls-sept --bucket acme-corpora --secret ACME_PROD_SA            # another project's key
```

It prints each stage as the run goes, then the manifest's path. Or run the pipeline directly from its own
folder, which is the same code:

```bash theme={null}
cd lab/ingest
python app.py --name calls-sept --publish --bucket my-staging-datasets --only-validated
```

Or against the dashboard API:

```bash theme={null}
curl -X POST "$DASHBOARD/api/datasets/$DATASET_ID/publish" \
  -H "Authorization: Bearer $LEVERS9_TOKEN" -H 'Content-Type: application/json' \
  -d '{"destination": {"type": "gcs", "bucket": "my-staging-datasets", "only_validated": true},
       "secret_map": {"SERVICE_ACCOUNT_JSON": "STAGING_SA"}}'
```

## More than one bucket: dev, staging, prod

The layout below is the same in every bucket, so a second destination is setup only — no different
commands. Repeat steps 1 and 2 against that project and store its key under its own secret name:

```bash theme={null}
# dev
gcloud iam service-accounts create levers-lab-publisher --project my-project-dev --display-name "Levers Lab dataset publisher"
gcloud storage buckets add-iam-policy-binding gs://my-dev-datasets --project my-project-dev \
  --member serviceAccount:levers-lab-publisher@my-project-dev.iam.gserviceaccount.com --role roles/storage.objectAdmin
gcloud iam service-accounts keys create /tmp/dev-key.json --project my-project-dev \
  --iam-account levers-lab-publisher@my-project-dev.iam.gserviceaccount.com

levers9 secret create DEV_SA "$(cat /tmp/dev-key.json)" && rm /tmp/dev-key.json
levers9 dataset publish calls-sept --bucket my-dev-datasets --secret DEV_SA
```

A project with no datasets bucket yet gets one configured the way the others are (same region, uniform
bucket-level access, public access prevention) before you point a publish at it:

```bash theme={null}
gcloud storage buckets create gs://my-prod-datasets --project my-project-prod \
  --location ME-CENTRAL2 --uniform-bucket-level-access --public-access-prevention
```

In the dashboard, the Publish dialog lists every bucket this workspace has published to before (only this
workspace's: other workspaces' buckets and datasets aren't listed). Picking one
fills in the bucket **and** the secret that opened it, which is the pairing that is easy to get wrong by
hand; a bucket whose secret has since been deleted is marked rather than silently offered. The list comes
from the runs' own receipts, so a bucket appears once someone has published to it and never needs
registering.

> [!WARNING]
>
> Nothing distinguishes a prod bucket from a dev one except its name, and `--only-validated` is opt-in
> everywhere. Publishing without it sends model drafts too — fine for staging, worth thinking about for a
> corpus other people train on.

## What lands in the bucket

```
raw/raw_audio/<folder>/<call_id>.wav      the recordings (a synthesised clip per row, for TTS)
raw/structured/<folder>/<folder>.csv      one line per row, naming the audio above
```

Those two prefixes are the staging bucket's pipeline entry points (an Eventarc trigger runs the
`audio-pipeline` workflow for objects under them), so the manifest is uploaded **last**: by the time
anything downstream reads a line, the audio that line names is already there. The dev and prod buckets have
no such trigger today, but the layout is deliberately identical so a dataset moves between them unchanged.

The manifest keeps the column names the structured folder already uses (`FileName`, `SegmentID`,
`SegmentStart`, `SegmentEnd`, `Speaker`, `GroundTruthText`, `ProcessedText`, …), so a reader written for the
datasets already in that folder reads this one too. Four columns are ours and sit at the end: `Split`,
`Status`, `CallID` and `Provider` — a reader that selects by name ignores them, and a person opening the CSV can
see which rows a human signed off.

For a speech dataset, one recording is uploaded once even though many rows are chunks of it; each row's
`SegmentStart` and `SegmentEnd` cut it. `GroundTruthText` is the transcript a person confirmed, falling back to
the model's draft when there is none — so publishing early gives drafts, and publishing again after annotation
replaces them.

If the dataset's [policy](https://levers9.mintlify.site/lab/personal-data.md#who-reads-the-text-off-flag-and-mask) is **Mask**, the manifest
carries the masked text: `[NAME]`, `[PHONE]` and the rest in place of what the scan found, or of what a
reviewer settled where a row was [reviewed](https://levers9.mintlify.site/lab/personal-data.md#check-the-scan). The recordings go as
recorded, because Mask covers text and not audio. A manifest is read by a training pipeline, so a publish
doesn't write a line it can't vouch for: if a row's text changed since the last scan, the run stops before it
uploads anything and asks you to scan again.

Publish isn't the only way out. For a file to train on with made-up names and numbers in place of the real
ones, and a report that says how well the scan was checked, see
[Export for training](https://levers9.mintlify.site/lab/personal-data.md#export-for-training). From a terminal that's
`levers9 dataset pii export`.

## Interrupted uploads

Uploads are resumable, which matters at corpus scale: a dataset is thousands of files and can be tens of
gigabytes. Each file goes up as a GCS resumable session, and if a transfer breaks the next attempt asks the
bucket how many bytes it actually holds and continues from there rather than starting the file again. Session
URIs are written beside the dataset before the first byte, so a container that is killed — evicted, restarted,
cancelled — resumes mid-file. If a run fails, run it again: it picks up where it stopped.

## Publishing again

Publishing is safe to repeat. Audio already in the bucket is left alone (the run reports how many were skipped)
and the manifest is rewritten, which is what you want after more rows are validated. Pass `overwrite` to send the
audio again as well. Each run appends a receipt to the dataset's `meta.json` — bucket, folder, counts, and when —
so the dataset page can show where its rows went.

## When it fails

| what you see | what it means |
| - | - |
| `missing secrets: SERVICE_ACCOUNT_JSON` | the secret does not exist in this workspace; create it in step 2 |
| `has no key configured yet; ask a platform admin` | a shared destination is listed but its key is not on the dashboard |
| `this upload ticket is not valid (or has expired)` | the run took longer than the ticket's life; publish again, it resumes |
| `SERVICE_ACCOUNT_JSON is not JSON` | the secret holds a token or a path, not the key file's contents |
| `... is a authorized_user credential` | that is a `gcloud auth` file, not a service-account key |
| `403 does not have storage.objects.create` | the binding in step 1 is missing, or on the wrong bucket |
| `no validated rows yet` | `only_validated` was asked for and nobody has signed a row off |
| `rows hold text the personal-data scan has not read as it is now` | the dataset's policy is Mask and some text changed since the last scan; run the scan from the Anonymization card, then publish again |

## The bucket's own README

Each dataset bucket carries a `README.md` describing this layout for people who open the bucket rather than
the dashboard. The canonical copy is `lab/ingest/BUCKET_LAYOUT.md` in the repo — edit it there and upload,
so the buckets and the docs cannot drift:

```bash theme={null}
for P in my-project-dev my-project-staging; do
  B=$(gcloud storage ls --project $P | grep -o 'gs://my-[a-z]*-datasets/')
  gcloud storage cp lab/ingest/BUCKET_LAYOUT.md "${B}README.md" --project $P
done
```

Next: [Annotation](https://levers9.mintlify.site/lab/annotation.md) to get rows validated first, or [Insights](https://levers9.mintlify.site/lab/insights.md) to check what you
are about to send.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.