Progress Channel

---
name: progress-channel
description: One visible channel for every long-running process — local sweeps, backgrounded commands, and work handed to external systems (a CI run, a media-server queue, a long download). The server that serves the live progress page IS the tracker — a stdlib Python daemon holding live jobs in memory, auto-spawned by the first producer — with learned ETAs, pre-start duration forecasts, and stall/orphan detection from each job's own history. Use when starting any operation expected to exceed ~10 seconds, when backgrounding a command, when triggering work in another system, or when the user asks "are we there yet", "how long will this take", "what's still running", or "track this progress".
---

**Codex:** Read [the client path](references/clients.md) before applying this workflow.


# progress-channel

Work that takes minutes is usually reported by whatever the agent happens to
print — a carriage-returned counter in one terminal. That fails three ways:
the only way to learn the state is to re-ask (pure overhead), the counter can
lie (a stale fragment reads as a stall; submission-order iteration pins it at
0), and work handed to another system has no representation at all. This
skill makes a shared progress channel the default for all three.

## The discipline

- **Register anything expected to exceed ~10s** — a sweep, a backgrounded
  command, a call that hands work to another system and returns early.
  `progress forecast <name>` answers "how long has this taken before" *prior*
  to starting; if history says minutes, register.
- **Never answer a progress question by re-polling and narrating.** The
  answer is the channel: the live page at `http://127.0.0.1:7717/`, `watch`
  in a terminal, or `list` once. Register, then get on with something else.
- **External work gets a `mirror` watcher** — one small process polling the
  foreign system into the same channel, exiting when the work is idle. One
  list, whatever the source.

## Architecture

The thing that serves the progress page **is** the tracker. A stdlib-only
Python daemon (`http.server`, no dependencies) binds `127.0.0.1:7717`
(`$PROGRESS_PORT` overrides), holds live jobs **in memory** — single writer,
so there is no store or locking at all — and serves the HTML view at `/`,
JSON at `/jobs`, and forecasts at `/forecast?name=`.

- **Producers auto-spawn it**: the first `Job` brings the daemon up if it
  isn't running; binding the port is the single-instance lease. Nothing to
  install or supervise.
- **Only the learning data is persisted**: finished runs append to
  `~/.claude/progress/history.jsonl` (`cat`/`jq`-able; `$PROGRESS_HOME`
  overrides the directory). Live state is memory-only on purpose — if the
  daemon restarts, producers re-register on their next flush; a reboot kills
  the jobs anyway.
- **A job never fails because the tracker is sick**: if the daemon can't be
  reached or spawned, the job degrades to a warn-once untracked no-op.

## Library

```python
# <plugin>/scripts/progress.py — stdlib only; import via the plugin path
with Job('video integrity', total=10453) as j:
    for item in items:
        j.step(detail=item, ok=1)        # or truncated=1, unreadable=1, ...
```

- The job is a **context manager**: an exception reports `failed` with the
  error; only a SIGKILL-class death goes silent, and the daemon's sweep
  catches those by checking the producer's pid directly (it is local — no
  heartbeat protocol). Never report progress around it.
- `step()` accepts categorical counters (`ok=812 truncated=3`) — sweep
  output is categorical, and forcing it into fixed fields is what stops a
  channel from being generic.
- Flushes are **throttled** (every 50 steps or 1s, plus on exit) — a
  10k-item loop is a handful of POSTs, not 10k.
- `set_total(n)` when the size is discovered mid-run; omit `total` when
  genuinely unknown (the view shows a count, not a lying percentage).

## CLI

```bash
python3 <plugin>/scripts/progress.py list              # one-shot view (all states)
python3 <plugin>/scripts/progress.py watch             # live TUI
python3 <plugin>/scripts/progress.py forecast <name>   # pre-start estimate
python3 <plugin>/scripts/progress.py run --name build --total 1 -- make all
python3 <plugin>/scripts/progress.py mirror --name 'immich metadata' \
    --source immich-jobs --poll-cmd '<status cmd>' --interval 30
python3 <plugin>/scripts/progress.py daemon            # foreground (debug)
python3 <plugin>/scripts/statusline.py                # status-line rows (stdin: session JSON)
<cmd> 2>&1 | <plugin>/scripts/progress_tap.py <name>  # measured bar from a build's own output
python3 <plugin>/scripts/progress.py prune             # compact history.jsonl + drop names idle >7d
```

`run` wraps any command as a job (duration feeds the forecast even without
per-item granularity). `mirror`'s `--poll-cmd` is a user-supplied shell
command printing `<done> <total>`, `<done>`, or `idle`; the watcher exits
after `--idle-after` consecutive idle polls, and its live job is the lease —
a second mirror for the same `--source` refuses to start while the first is
alive.

## From any script: start / step / finish

Anything that can run a command can be a producer — no Python import needed:

```bash
P='python3 <plugin>/scripts/progress.py'
T=$($P start --name 'photo import' --total 800)
trap '$P finish $T --fail "aborted at $f"' ERR
for f in *.jpg; do
    convert "$f" ...
    $P step $T --count ok=1 --detail "$f"     # -n N, --done N, --total N
done
$P finish $T                                   # or: --fail "why" / --cancel
```

If the loop lives inside a shell **function**, use an `EXIT` trap instead
(`trap 'rc=$?; [ $rc -ne 0 ] && $P finish $T --fail "exit $rc"' EXIT`): an
`ERR` trap does not fire inside a function unless `set -E` is on, so the script
exits through `set -e` and the job is left for the orphan sweep.

`start` prints a token; the cross-invocation state (count, counters, step
gaps) lives in a token file under `~/.claude/progress/tokens/`, so `step` is
stateless for the script and keeps the same 1s POST throttle as the library.
Liveness anchors to the **calling script's pid** (`--pid` overrides), so a
script that dies without `finish` is swept as orphaned like any other
producer.

If you wrap the CLI in a shell **function** and call it inside `$( )`, add
`--pid $$` to `start`. The CLI's parent is then a short-lived subshell, not
your script, and without the flag the job is swept as orphaned within seconds
while the script is still running.

## Integration examples

`examples/` (beside `scripts/`) holds small runnable producers — copy one and
replace the `sleep` with the work. Each runs the same with or without the
channel. Read [examples/README.md](../../examples/README.md) and point the user
at the one that matches what they have: a shell loop, a nested shell pipeline,
Python, Node, Go, a Makefile, or a build tool (`taps.md`). Any other language
uses the same three CLI calls: `start` prints a token, `step`, `finish`.

## Sub-jobs: a pipeline and its current step

A job can run inside another. Every surface shows it nested under its parent:

```
⏳ pipeline rome (italy) #119 ██████████████▏░░░  79% 26/34
⏳   ↳ ground-round ███████████████▉░░  89% 549/619
```

```python
with Job('pipeline rome', total=34) as p:
    for step in steps:
        with p.child(step.name, total=step.size) as c:
            ...
```

```bash
T=$(progress start --name 'pipeline rome' --total 34)
export PROGRESS_PARENT=$T              # every sub-script below nests itself
C=$(progress start --name ground-round --total 619)   # or: --parent $T
progress finish $T                      # also cancels any child still open
```

- **Never bake the indent into the name** (`'  ↳ ground-round'`). That was the
  manual pattern this replaces, and it broke three ways: the child jumped
  above its parent whenever it stepped last, cancelling the parent left the
  child behind, and the arrowed name learned no ETA from standalone runs of the
  same step. A child keeps its **plain** name, so its history is shared.
- `$PROGRESS_PARENT` reaches Python producers too, so a `progress run` or a
  tap launched by a pipeline step nests without any flag. Leave it unset (or
  empty) for top-level work.
- **The parent bar rolls up.** A counted parent with running, *counted*
  children reads `(done + child fractions) / total`, mode `items+sub` — 79%
  above, not a bar stuck at 76% for the whole step. Estimated children
  (time/eta/creep) never move a counted bar.
- **Time left rolls up too:** the running step's own time-left plus the
  remaining whole steps at the parent's per-step rate. The parent alone would
  price the in-flight step as untouched. A step with no estimate yet leaves
  the parent's own figure rather than a partial sum that reads as precise.
- **Closing cascades from the producer.** A parent's `with` block, or
  `finish` on its token, closes open children first — `cancelled`, naming the
  parent. A child whose parent ended without cascading (a raw POST client)
  stays visible at the top level, labelled `parent … ended`, never hidden.
- **Mirrors** get sub-steps from an optional second `--poll-cmd` line,
  `<done> <total> <name>`. A new name finishes the previous child as done.
- The status line keeps its 3-row cap: every top-level job gets a row first,
  children take what's left, and a child without a row is folded onto its
  parent's line. Any depth is stored; the status line shows two levels, the
  page and `list` show all.

## Scoping: whose job is this?

Every producer records the Claude Code session that owns it, for free —
Claude Code exports `CLAUDE_CODE_SESSION_ID` into every tool process, so
nothing has to be plumbed through. `CLAUDE_CODE_AGENT` (set only inside a
subagent) is recorded too, so a subagent's work shows under the session that
spawned it while still being attributable to the agent.

```bash
curl -s 'localhost:7717/jobs?session=<id>'    # one session's work
```

The page takes the same `?session=` parameter and its session column links to
it. Work started outside a session — cron, a bare shell — carries no session
and appears only in the unfiltered view; the filter is an exact match, because
a per-session view that quietly included machine-wide work would be worse than
useless.

## What `/jobs` returns, and for how long

**`/jobs` returns live work only.** That is the whole point of an ambient
surface: it answers "what is happening now", not "what has ever happened".

- `?state=all` — everything still retained
- `?state=done,failed` — an explicit comma-separated set
- absent — `running` + `stalled`, **plus** anything finished within the linger
  window, so a job is visibly seen reaching 100% instead of blinking out

Deliberate surfaces are not filtered: `list`, `watch` and the MCP
`progress_list` all ask for `?state=all`, because you went looking.

Two clocks, deliberately separate:

| | default | env | meaning |
|---|---|---|---|
| linger | 20 s | `PROGRESS_DONE_LINGER` | how long a finished row stays in the **live view** |
| removal | 24 h | `PROGRESS_FINISHED_HOURS` | when a done/failed row is **dropped** |
| orphan removal | 30 min | `PROGRESS_ORPHAN_MINUTES` | orphans are already dead — a shorter leash |
| hard cap | 60 | `PROGRESS_FINISHED_MAX` | bounds a pathological day |

## Time-based progress

Not all work counts items. A bar fills from the best evidence available, and
**says which**, so an estimate is never mistaken for a measurement:

| mode | fills against | when |
|---|---|---|
| `items` | `done/total` | a total is known — the only measured mode |
| `time` | elapsed / declared duration | `Job(expect_seconds=)` / `start --seconds` |
| `eta` | elapsed / median of this job's past runs | history exists for the name |
| `eta~` | elapsed / median of *similar* jobs' runs | name never ran, but its **stem** did |
| `creep` | `1-exp(-t/90)`, capped 95 % | nothing to measure against |

`eta~` is the borrowed estimate: the name is normalized to a stem (lowercase,
path-ish tokens dropped, digits stripped — `make /home/a` ≈ `make /home/b`,
still ≠ `pytest`) and matched against same-stem runs in the same project
first, then anywhere. History records carry `project`/`session`/`agent` for
this; matching uses only the stable keys (stem, project) — session ids are
recorded for audit, never matched on.

`time` and `eta` cap at 99 %: a job that overruns its estimate must not read as
finished, because that is exactly when you want to look at it. The page hatches
estimated bars; the status line prints the mode.

## Heartbeats: `ping` and the stopped watchdog

`step` advances a count and is throttled. **`ping` says "still alive" and always
posts immediately** — a heartbeat a throttle might swallow is not a heartbeat.

```bash
T=$($P start --name 'remote build' --seconds 300 --timeout 60)
while ...; do $P ping $T; done              # time-based: just a heartbeat
$P ping $T -n 5                              # or carry items along
$P ping $T --total 900 --seconds 600         # override the shape mid-run
```

`--timeout N` is a contract the producer opts into: **no step or ping for N
seconds and the job is declared `stopped`** and retired. This is for producers
whose pid says nothing useful — a remote job, a shell that forks, anything
polled rather than owned. It outranks the learned `stalled` heuristic, which
stays advisory and self-resolving.

## In the Claude Code window

Two surfaces show this session's live jobs, and they never both draw the same
job. Read [references/in-the-window.md](references/in-the-window.md) before
setting either up or explaining one.

- **The band above the prompt** (0.7.0, Claude Code only) — a mod that ships
  with the plugin. No setup; up to six rows. `/progress-bar` toggles it, and
  `on|off|auto` sets the mode. `auto`, the default, hides the band while a
  status line is polling the daemon.
- **The status line** — `scripts/statusline.py`, wired in
  `~/.claude/settings.json` with `refreshInterval: 1`. Works on any client with
  a status line. When the user says "set up the progress status line", make the
  edit for them, following the reference.

With neither wired, point the user at the band first: it is already on.

## Tap: a measured bar from a build's own output

`scripts/progress_tap.py` is a transparent pipe filter: stdin is forwarded
byte-for-byte, and position lines the tool already prints become the bar — no
estimate needed. Maven's reactor prints `[3/15]`; git prints
`Receiving objects: 42% (12345/29292)`; pytest prints `[ 42%]`:

```bash
mvn -B verify 2>&1              | progress_tap.py 'gate: full' > build.log
git clone --progress <url> 2>&1 | progress_tap.py 'clone linux' --pattern git
pytest 2>&1                     | progress_tap.py 'tests' --pattern pytest
cargo build 2>&1                | progress_tap.py 'build' --pattern cargo
tool 2>&1 | progress_tap.py migrate --pattern 'count:^migrated' --total 800
```

Patterns, by what the tool prints:

- **measured** (done and total): `maven` (default) · `git` · `docker` · `ninja` · `ratio`
- **percent**: `pytest` · `cmake` · `rsync` · `percent`
- **counted** (one line per unit; pass `--total` if you can compute it):
  `gradle` · `cargo` · `go` · `jest` · `dotnet` · `terraform` · `ansible` ·
  `batch` · `count:<regex>`

`ratio` and `percent` are generic, for a tool with no preset. The command per
tool, the flag some need to print position into a pipe, and how to compute a
total are in [examples/taps.md](../../examples/taps.md). A tool with no position
output (`npm install`, `ffmpeg`) cannot be tapped — use `run`.

Two caveats
the callers must own: a pipeline's `$?` is the **tap's** — read
`${PIPESTATUS[0]}` for the build's verdict — and git prints no position in a
pipe unless `--progress` is passed. If the channel is down the tap degrades to
`cat` in silence; a progress bar is never worth a broken build.

## Notifications

An executable at `~/.claude/progress/notify` is the entire configuration:
the daemon runs it detached on every `done` / `failed` / `orphaned` /
`stalled` transition (stalls fire once per episode; activity resets it),
with the event in env vars — `PROGRESS_EVENT`, `PROGRESS_NAME`,
`PROGRESS_STATUS`, `PROGRESS_ERROR`, `PROGRESS_SECONDS`, `PROGRESS_PROJECT`.
Point it at `notify-send`, an email sender, anything. The daemon never
waits on it and never fails because of it. No hook file, no notifications.

## MCP (sessions read the channel as tools)

`scripts/progress_mcp.py` is a stdio MCP server — a thin face on the same
daemon API: `progress_list`, `progress_forecast`, `progress_start`,
`progress_step`, `progress_finish`. Register it per project:

```json
{"mcpServers": {"progress": {"command": "python3",
    "args": ["<plugin>/scripts/progress_mcp.py"]}}}
```

## Advisory auto-registration (hook)

The plugin ships a `PreToolUse` hook on Bash that nudges — never rewrites —
when a command deserves tracking: either the channel's own history says
this command shape has a median over ~10s (the learned answer to "should
this be tracked"), or it matches a short list of famously long-running
commands / is being backgrounded. The suggestion names the exact `run`
wrapper to use; trivially short commands stay silent.

## What the view tells you (and why to trust it)

The viewer never takes `running` at face value — that is how counters lie:

- **orphaned** ☠ — the producer's pid is dead (killed too hard for the
  context manager). The daemon's sweep adjudicates these actively.
- **stalled** ⏸ — silent longer than 3× the job's *own* learned p95
  inter-step gap (never under 30s). A job whose gaps are always long is not
  stalled — that is exactly why the threshold is learned per job, and why a
  job with no history is never called stalled.
- **eta** — before ~10% progress the estimate comes from the median per-item
  rate of past runs of the same shape (name + kind + order-of-magnitude of
  total, so videos and thumbnails under one name don't blend); past it, the
  current run's own observed rate takes over. First run of a shape shows no
  estimate rather than extrapolating.

History keeps the last 20 runs per shape — an ETA input, not an archive —
and **forgets dead names**: a job name with no run in the last 7 days loses
every row (estimate, stall threshold, sparkline) at daemon startup, on a
daily sweep pass, and on `prune`. Per-name, not per-row, so one fresh run
keeps a job's whole learning window. `PROGRESS_HISTORY_DAYS` widens it for
monthly jobs.

**Upgrades restart the daemon by handshake**: `/health` reports the plugin
version, and a producer from a *newer* install asks the old daemon to
`/shutdown` (SIGTERM fallback for pre-0.4 daemons) and respawns the new
code — no stale daemon holds the port across a plugin update. Only strictly
newer evicts, so a dev checkout never bullies an installed daemon, and live
jobs survive because every producer re-registers on its next flush.
The page's history section adds per-job duration sparklines and a **trend**
tag ("slowing +40%") when recent same-shape runs drift from the older
baseline; `forecast` prints the same trend. `run` also tees the wrapped
command's last output lines into the row, so a failed job shows *why* on
the page.

## Verify it's working

`python3 <plugin>/scripts/test-harness.py` (123 checks: real daemon on an
ephemeral port, SIGKILL orphan sweep, restart re-registration, degraded
mode, shell start/step/finish, notify hook, trend, MCP handshake, advisory
hook, status line, eta~ stem matching, retention, upgrade handshake, the
tap's passthrough/maven/git parsing, statusline_seen discovery, the
whats-new notice, the wrap script) — or register a trivial job and open
the page: the row appears at registration, not completion.