> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Logs

> How a container's output reaches your terminal and the dashboard, and what happens when something in between fails

Everything a job, endpoint or box prints is captured once, on the worker that runs it, and every reader
gets the whole of it: the CLI that called the job, the dashboard, a stream that reconnects after a
dropped connection. Nothing is shared out between readers and nothing is skipped on a reconnect.

## The path

```mermaid theme={null}
flowchart LR
  C[container stdout / stderr] --> W[worker<br/>one log file per container<br/>one line = one numbered record]
  W -- "since line N" --> CP[controlplane<br/>attach / job invoke streams]
  CP --> CLI[levers9 CLI<br/>python app.py]
  CP --> D[dashboard<br/>log spool on disk]
  D --> B[browser<br/>live view · history · download]
```

1. **The worker writes every line to the container's log file** as it arrives, and numbers it. The file
   is the source of truth; nothing is held in memory for readers.
2. **A reader tails that file from a line number.** The first attach starts at line one; a reader that
   comes back after a drop sends the last line it saw and gets what it missed. Two readers, ten readers,
   each see everything.
3. **The controlplane passes the line number through** on `AttachToContainer` and `JobInvoke`, so the CLI
   and the dashboard both resume rather than replay.
4. **The dashboard spools to disk** and records its position next to the file. A finished container's
   output stays readable after the container is gone.

> [!NOTE]
>
> Before this, the worker kept output in one in-memory buffer that handed each line to whichever reader
> asked first. With the CLI and the dashboard both attached, each saw half the lines (step 1, 6, 31 in
> one, step 16, 36, 46 in the other), and a reader that arrived late saw nothing. That is what the numbered
> file replaced.

## Reading logs

* **CLI**: `python app.py` streams the job's output as it runs. If the connection drops, the CLI reattaches
  to the same task and continues from the last line it printed.
* **Dashboard**: a call's drawer shows its output live and keeps it afterwards; **Full view** opens the
  whole stream; **Download** gives the entire spool file, not just the last screenful.
* **A long run opens at its newest part.** A training job can leave hundreds of thousands of lines, so the
  viewer starts from the newest 64 KB and keeps the newest 3,000 lines once they are parsed, which is what
  paints at once. It says which of the two bounds is in force: **showing the newest part of 344 KB · show
  everything** re-reads from the first line, and **showing the newest 3,000 lines · download all of it**
  hands over the whole spool file. Neither is a gap in the capture: the spool has every line either way,
  and `Download` always gives the lot.
* **Before the first line**, the viewer says what the container is doing (waiting to be scheduled,
  running but nothing printed yet). A container that is loading its blueprint or importing a large library
  can be quiet for a minute or two; that is the container, not the stream.

## How long logs are kept

**A year**, on the dashboard's own volume, set by `LEVERS9_LOG_RETENTION_DAYS`. A container's spool is
removed once it is older than that, and the spool of a container still being captured is never removed
however old its file looks — an always-on job that has been quiet carries the timestamp of its last line.

`LEVERS9_LOG_SPOOL_MAX_MB` (8 GB) is a backstop rather than a policy: if the spool grows past it, the oldest
go early and the dashboard log says so, because a spool removed because the volume was filling is a different
fact from one that aged out.

> [!NOTE]
>
> This used to be a count — the newest 500 spools, whatever span that covered. So how far back the logs went
> depended on how many containers had run rather than on time: a busy week bought a few days and a quiet one
> bought a month, with no way to tell which you had. The prune also ran only when the process started, so the
> spool was cut back at each deploy rather than ageing out, which is why logs seemed to disappear in steps.

A single container's own spool is capped separately: past 5 MB it is trimmed to its newest 2 MB, with a line
written into the file so the gap is visible rather than silent.

## Keeping them longer than that

The spool is one disk attached to one pod. It survives a restart and not the loss of that disk, and a year of
retention on a single disk is still a single disk. So a workspace can ask for its finished logs to be kept:
**Members → Keep container logs in this workspace**.

With it on, a container's output is copied to a `container-logs` **volume in that workspace** once the
container finishes, and stays there after the local spool ages out — the log view falls back to the archived
copy without anyone asking it to. With it off, logs live on the dashboard's disk for the retention window
above and are then gone.

> [!NOTE]
>
> It is off until a workspace admin turns it on, and turning it on **creates the volume**. That is storage the
> workspace owns and pays for, holding every container's output indefinitely — a decision to take rather than
> inherit, which is why there is no deployment-wide default that does it for you.

A volume rather than a bucket of the dashboard's own, for two reasons. The dashboard already holds a
controlplane token and already writes volumes, so this needs no new credential, no IAM role and no key that
could leak. And a volume belongs to a workspace, so the isolation is the storage's own: the archived copy is
fetched with the reader's token, and a token that is not that workspace's does not find the file.

## What can go wrong

| scenario | what happens | what you see |
| - | - | - |
| The CLI's connection to the controlplane drops mid-job (load balancer rollout, laptop sleep, network blip) | The task keeps running; the controlplane holds it for a two-minute grace period. The CLI reattaches with the task id and the last line it printed. | A one-line warning, then the output continues from the next line. No repeats. |
| The controlplane restarts | Same as above for every open stream: the worker keeps writing the file, readers reattach and resume. | A short pause, then output continues. |
| The dashboard restarts while a container is running | The spool sees a file that is not marked finished and reattaches from its recorded line. | `[reconnected to the log stream; resuming after line N]` in the log, then the rest. |
| A reader attaches after the container has already exited, worker still up | The worker serves the file to the end and closes. | The full output, then the exit line. |
| A reader attaches after the container exited **and** the worker is gone (scaled down, node reclaimed) | The file went with the worker. The dashboard's spool is the copy that survives, if it was attached while the container ran. | The spool's copy, or `[container gone before the stream could resume; output after line N was not captured]` if the dashboard's own stream had dropped before the end. |
| The dashboard is not running while a container runs | Nobody spooled it. The CLI that called it still saw everything. | "No logs captured for this container: it ran while the dashboard was offline." |
| A container prints more than the worker's per-hour line budget | Lines past the budget are dropped by the worker with one note in the log; the budget is `worker.containerLogLinesPerHour` (default 6000) and is the one deliberately lossy setting in this path. | `Rate limit exceeded, logging at reduced rate, some logs will be dropped`, once. Raise the setting for chatty training loops. |
| A line is torn (the reader catches a record mid-write) | The reader waits for the rest of the line; nothing is emitted until the record is complete. | Nothing; lines never come through split. |
| The stream between controlplane and worker drops (a tunnelled cluster, a worker pod restarting) | The controlplane's stream ends; the CLI and the spool reattach and ask from their last line. A restarted worker has no file for containers it did not run, so those are gone. | Continues, or the "not captured" note above. |
| An always-on endpoint logs for days | The dashboard trims its spool to the newest few megabytes and says so; the worker's file grows until the container ends. | `[older output trimmed]` at the top of the dashboard's copy. |

## Where to look on a worker

Each container's file is `/var/log/worker/<container id>.log` on the worker that ran it: one JSON record
per line, with `msg`, `time`, `container_id` and `task_id`. `levers9 worker list` shows which worker.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.