Job Observability

---
name: job-observability
description: Observe automated jobs through trigger, logical operation identity, queue/wait time, attempts, execution, side effects, checkpoints, failures, completion, and stale/stuck detection.
---
# Job Observability

Use when scheduled, queued, event-triggered, or background automation needs to be diagnosable and operable at scale.

## Procedure
1. Define the logical job/operation identity separately from individual attempts so retries, node relocation, and replay can be correlated to one intent.
2. Record trigger/source, tenant/resource scope, workflow/job type, revision/config version, enqueue/start/end time, executing node/worker when relevant, attempt count, and final state.
3. Measure queue/wait age, runtime, completion/failure rate, retry rate, backlog/depth, concurrency, and resource/dependency pressure that can explain delayed work.
4. Emit structured lifecycle events for accepted, claimed/started, checkpointed, retried, blocked/waiting, succeeded, failed, cancelled, expired, and reconciled states that exist in the automation.
5. Correlate external side effects using safe provider/request IDs without logging credentials or unnecessary sensitive payloads.
6. Detect stale/stuck work using explicit heartbeats, leases, deadlines, or progress checkpoints appropriate to the job model rather than one universal timeout.
7. Distinguish retry churn from healthy throughput. A job eventually succeeding after many attempts may still indicate a degraded dependency or broken policy.
8. Alert on actionable conditions: missed schedules/objectives, old queue age, repeated failure, retry exhaustion, stalled checkpoints, or systematic duplicate suppression/reconciliation anomalies.
9. Provide a queryable history that supports replay/recovery decisions and can answer what ran, what changed, which side effects occurred, and why the job ended in its current state.
10. Test telemetry during crash/restart, duplicate trigger, retry, stale worker, relocation to another Fleet node, partial side effect, and final reconciliation.

## Decision rules
- Attempt logs alone are not job history; preserve the logical operation across attempts.
- Metrics show population health while event/run records explain individual failures; use both where the scale warrants it.
- Do not mark a job successful merely because its process exited zero if its intended side effect or durable outcome is unverified.
- Fleet may own node/task placement, but automation telemetry should carry node/attempt identity so relocation and correlated node failures are diagnosable.

## Quality gate
Job observability is sufficient when operators can follow one logical job across triggers, attempts, nodes, checkpoints, and side effects; population-level backlog/failure health is visible; stuck work is detectable; alerts are actionable; and replay/recovery decisions can be made from recorded evidence.