Database Failover Runbook

---
name: database-failover-runbook
enabled: true
description: |
  Use when performing database failover runbook — step-by-step runbook for
  executing controlled and emergency database failovers. Covers pre-failover
  health checks, replication lag verification, connection draining, promotion of
  standby, application reconnection, and post-failover validation to minimize
  data loss and downtime.
required_connections:
  - prefix: aws
    label: "AWS (or cloud provider)"
  - prefix: pagerduty
    label: "PagerDuty (or alerting tool)"
config_fields:
  - key: database_cluster
    label: "Database Cluster Name"
    required: true
    placeholder: "e.g., prod-orders-postgres"
  - key: database_engine
    label: "Database Engine"
    required: true
    placeholder: "e.g., PostgreSQL 15, MySQL 8, Aurora"
  - key: failover_type
    label: "Failover Type"
    required: true
    placeholder: "e.g., planned, emergency"
features:
  - DEVOPS
  - RUNBOOK
---

# Database Failover Runbook

Execute a **{{ failover_type }}** failover for database cluster **{{ database_cluster }}** ({{ database_engine }}).

## Workflow

### Phase 1 — Pre-Failover Assessment

```
CLUSTER STATUS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Current primary node: ___
[ ] Standby/replica node(s): ___
[ ] Replication lag: ___ms (must be < 1s for planned failover)
[ ] Replication status: STREAMING / ASYNC / SYNC
[ ] Active connections on primary: ___
[ ] Long-running transactions: [ ] NONE  [ ] IDENTIFIED
[ ] Last successful backup: ___
```

### Phase 2 — Pre-Failover Checklist

```
READINESS CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Standby node is healthy and accepting read queries
[ ] Replication lag is within acceptable threshold
[ ] No active DDL operations in progress
[ ] Application connection pool supports failover
[ ] DNS TTL is low enough for endpoint switch (current TTL: ___s)
[ ] On-call team notified of failover window
[ ] Monitoring dashboards open and visible
[ ] Rollback plan reviewed

DECISION MATRIX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Condition                    | Planned | Emergency
Replication lag < 1s         | REQUIRED| BEST EFFORT
Long transactions drained    | REQUIRED| SKIP
Stakeholder notification     | REQUIRED| POST-HOC
Backup verified              | REQUIRED| REQUIRED
```

### Phase 3 — Execute Failover

```
FAILOVER EXECUTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Timestamp of failover initiation: ___

For planned failover:
  [ ] 1. Set primary to read-only mode
  [ ] 2. Wait for replication lag to reach 0
  [ ] 3. Promote standby to primary
  [ ] 4. Update DNS/endpoint to new primary
  [ ] 5. Verify new primary accepting writes

For emergency failover:
  [ ] 1. Promote standby immediately
  [ ] 2. Update DNS/endpoint to new primary
  [ ] 3. Verify new primary accepting writes
  [ ] 4. Assess data loss window (last lag value: ___ms)

[ ] Failover complete — timestamp: ___
[ ] Total failover duration: ___s
```

### Phase 4 — Post-Failover Validation

```
POST-FAILOVER CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] New primary accepting read and write queries
[ ] Application connections re-established:
    - Connection count: ___
    - Connection errors: ___
[ ] Query performance baseline met:
    - P50 latency: ___ms
    - P95 latency: ___ms
[ ] No data integrity issues detected
[ ] Old primary reconfigured as standby (if recoverable)
[ ] Replication re-established to new standby
[ ] Monitoring alerts updated for new topology
```

### Phase 5 — Recovery and Documentation

```
RECOVERY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Old primary recovered and added as replica
[ ] Replication verified on new replica
[ ] Backup schedule validated on new primary
[ ] Failover event documented in incident log
[ ] Runbook updated with any lessons learned
[ ] Stakeholders notified of completion
```

## Counter-Rationalizations

| Shortcut | Counter | Why |
|----------|---------|-----|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |

## Output Format

Produce a failover execution report with:
1. **Failover summary** (cluster, type, timeline, duration)
2. **Replication status** at time of failover (lag, data loss window)
3. **Application impact** (connection errors, downtime duration)
4. **Post-failover health** (performance metrics, replication status)
5. **Action items** for follow-up (topology hardening, runbook updates)