Backup Job Health

---
name: "Backup Job Health"
description: >
  Portfolio-wide backup job health across whatever BCDR and SaaS-backup tools
  are connected: the two structurally different job models (image-based
  appliance backup vs. SaaS-data snapshot backup) and how to normalize them
  into one health record per protected unit, why a live consecutive-failure
  streak matters more than a trailing success rate, the missed-versus-failed
  distinction and its different root causes, and the two storage-trending
  risk patterns (approaching capacity, anomalous growth).
when_to_use: >-
  When assessing whether backup jobs are actually running successfully across
  a portfolio, rather than checking one client's one appliance in isolation.
  Use when: backup health, backup job failures, missed backups, backup
  success rate, failed backup jobs, backup storage trending, is this client's
  backups running, backup audit, backup job status.
---

# Backup Job Health

## Overview

"Backups are running" is the single most-assumed, least-verified fact in an MSP's
environment. A backup job that failed silently three nights ago looks identical, from
a distance, to one that's been succeeding for months — nobody notices until a restore
is needed and there's nothing current to restore from. This skill is the recurring,
portfolio-wide sweep that catches that gap before it becomes an incident: job success/
failure rate, missed-backup detection, and storage-consumption trending, rolled into
one normalized view across every backup and BCDR tool an org has connected.

This skill is about whether backups are *happening*. It is deliberately narrower than
disaster-recovery readiness as a whole — it does not assess whether a backup, once
taken, is actually recoverable (see `restore-test-verification`), and it does not
assess whether the retention window or cadence in place actually satisfies a
contracted requirement (see `retention-rpo-compliance`). Treat this as the first,
most frequent layer of the DR assurance stack: if jobs aren't running, nothing
downstream matters yet.

## Anti-triggers

- **Backup Radar's own health records** — ScalePad already aggregates and
  scores backup results across vendors; use `scalepad-backup-radar` when the
  question is what that API returns. This skill normalizes across every
  connected backup tool, including orgs that have no Backup Radar at all.
- **One platform's job, alert, or storage data** — use
  `datto-bcdr-api-patterns`, `datto-saas-protection-api-patterns`,
  `spanning-api-patterns`, or `unitrends-api-patterns` for a single vendor's
  request shapes and field names.

## Key Concepts

### Two fundamentally different job models

Backup and BCDR vendors protect data in two structurally different ways, and treating
them as the same "job" concept produces misleading comparisons:

- **Image-based appliance backup** (e.g. Datto BCDR / SIRIS / Alto, Unitrends) —
  protects physical or virtual servers/workstations by taking periodic block-level
  or image-level snapshots to a local appliance, then syncing offsite/to the cloud.
  A "job" here is a scheduled backup of one protected agent/asset on one appliance.
  Health signals include: last successful local backup, last successful offsite
  sync, and (for Datto BCDR specifically) screenshot/boot verification status —
  see `restore-test-verification`.
- **SaaS-data snapshot backup** (e.g. Datto SaaS Protection, Spanning) — protects
  cloud application data (Microsoft 365 mailboxes/OneDrive/SharePoint/Teams, Google
  Workspace, Salesforce) by taking periodic API-level snapshots of tenant data. A
  "job" here is a scheduled backup pass across a set of protected seats/users for a
  tenant. There is no "appliance" and no local/offsite sync distinction — health
  signals are seat coverage (are all licensed users actually being backed up) and
  per-run success/failure across the tenant.

Normalize both into a single health record per protected unit (appliance-agent pair,
or tenant-seat set) with the same fields — last successful run, run status, and
failure streak — even though the underlying job mechanics differ. Don't force a
one-size-fits-all metric like "backup window duration" that only makes sense for one
model.

### Job success/failure rate

For each protected unit, compute the success rate over a rolling window (default:
last 30 days unless the org has a documented preference) and, more importantly, the
**current consecutive-failure streak**. A unit with a 96% success rate over 30 days
but a live 4-night failure streak right now is a more urgent problem than a unit with
90% success and no current streak — trailing averages hide exactly the thing that
matters most: is it broken *right now*.

### Missed-backup detection

A missed backup is distinct from a failed backup: a failed backup ran and errored; a
missed backup never ran at all (no job execution recorded for the expected window).
Both matter, but they point to different root causes — a failed job usually means an
in-scope problem (disk full, credential expired, source unreachable), while a missed
job often means a scheduling, licensing, or connectivity problem that's more
structural (the agent/connector isn't checking in at all). Report them as separate
categories rather than merging them into one "unhealthy" bucket, since the
remediation path differs.

### Storage-consumption trending

Track local and offsite/cloud storage consumption per appliance (or per SaaS tenant,
where the vendor exposes storage/quota data) over time. Flag two distinct risk
patterns:

- **Approaching capacity** — an appliance trending toward its local storage limit,
  which risks retention truncation (older recovery points get purged early to make
  room) even while nightly jobs continue to report success. This is a silent
  retention risk — see `retention-rpo-compliance` for how a storage-forced retention
  cut interacts with a contracted retention requirement.
- **Anomalous growth** — a sudden, unexplained jump in daily change-rate/storage
  consumption, which can indicate anything from a legitimate data-growth event to
  ransomware encryption activity happening on the protected source. Anomalous growth
  is worth flagging even when it isn't yet a capacity problem, because of what it
  might indicate about the protected system.

### If no backup/BCDR tool is connected

State plainly that job health cannot be assessed: "No backup or BCDR connector is
connected through the gateway, so there's no backup job data to audit." Do not
fabricate success rates, job counts, or storage figures.

## Common Workflows

### Full portfolio sweep

1. Discover connected backup/BCDR tools via `conduit__search_tools` — don't assume
   which of Datto BCDR, Datto SaaS Protection, Spanning, or Unitrends (or others) are
   live for this org.
2. For each connected tool, pull the protected-unit list (appliances/agents for
   image-based tools; tenants/seats for SaaS-snapshot tools) and each unit's recent
   job history.
3. Normalize into one health record per protected unit: last successful run, current
   run status, consecutive-failure streak, and (where available) storage/quota state.
4. Bucket into: actively failing (current failure streak), missed (no run recorded
   for expected window), storage-at-risk, and healthy.
5. Report worst-first: longest active failure streak, then missed backups, then
   storage risk.

### Targeted client check

1. Resolve the client to its protected units across whatever connected tool(s) cover
   them (a client may have both an on-prem appliance and a SaaS-backup tenant).
2. Pull and normalize job history for just that client's units.
3. Report success/failure rate, active streaks, and storage state for that client
   only.

## Error Handling

- **No backup/BCDR connector connected:** stop and say so; do not fabricate job
  status.
- **A connected tool doesn't expose storage/quota data:** report job success/failure
  normally and mark the storage-trending section "unable to verify — connector does
  not expose storage data" rather than omitting it silently.
- **A protected unit exists in inventory but has no job history at all:** treat this
  as a missed backup, not a gap in the report — a never-run job is exactly the kind
  of silent failure this skill exists to catch.

## Best Practices

- State the rolling window used (default 30 days) explicitly in every report.

## Related Skills

- [Restore-Test Verification](../restore-test-verification/SKILL.md) — whether a
  successfully-run backup is actually recoverable; this skill only confirms the job
  ran, not that its output is usable.
- [Retention/RPO Compliance](../retention-rpo-compliance/SKILL.md) — whether the
  retention window and backup cadence in place satisfy a contracted requirement;
  relevant when storage-forced retention truncation is detected here.