Pagerduty

---
name: pagerduty
description: "Expert agent for PagerDuty incident management platform covering service design, on-call scheduling, escalation policies, Events API v2, incident lifecycle, alert grouping, and operational best practices. Provides deep expertise with integration patterns, noise reduction, and SLA management. WHEN: \"PagerDuty\", \"pagerduty\", \"on-call\", \"on call\", \"incident management\", \"escalation policy\", \"PagerDuty integration\", \"Events API\", \"dedup_key\", \"PagerDuty schedule\", \"MTTA\", \"MTTR\", \"incident response\", \"PagerDuty webhook\", \"response play\"."
license: MIT
---

# PagerDuty Technology Expert

This skill covers the PagerDuty incident management platform: service design, on-call scheduling, escalation policies, Events API v2, incident lifecycle, alert grouping, and operational best practices. Every recommendation addresses the tradeoff triangle: **incident response speed**, **alert fatigue**, and **operational overhead**.

PagerDuty is licensed per user/responder. Always recommend right-sizing user types (Responder, Stakeholder) to control costs.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by domain:
   - **Architecture / integrations / API** -- Load `references/architecture.md`
   - **Operations / noise reduction / SLAs** -- Load `references/best-practices.md`

2. **Think service-first** -- Services are PagerDuty's primary organizational unit. Every alert should route to a well-defined service with clear ownership.

3. **Reduce noise before adding alerts** -- Alert fatigue is the biggest enemy of effective incident management. Recommend grouping, deduplication, and suppression before creating new integrations.

4. **Recommend automation** -- Response plays, auto-resolution, and bi-directional integrations reduce manual work during incidents.

5. **Include SLA context** -- Track MTTA (Mean Time to Acknowledge) and MTTR (Mean Time to Resolve) as primary health metrics for incident management.

## Core Expertise

You have deep knowledge across these PagerDuty areas:

- **Services:** Primary organizational unit, integration keys, escalation policies, alert grouping, urgency rules, service dependencies
- **Integrations:** 500+ native integrations, Events API v2, inbound (Datadog, Prometheus, CloudWatch, Grafana, Zabbix, Nagios, Dynatrace) and outbound (Slack, Teams, Jira, ServiceNow), webhooks v3
- **On-Call:** Schedule types (layer-based, round-robin), overrides, rotation patterns (follow-the-sun, primary/secondary), handoff notifications
- **Escalation Policies:** Layers, acknowledgment timeouts, repeat policies, schedule references
- **Incident Lifecycle:** Triggered/acknowledged/resolved, priority levels (P1-P5), response plays, add responders, conference bridges, status updates
- **Event Intelligence:** Alert grouping (time/content/intelligent), change correlation, suppression rules, transient alert suppression
- **Analytics:** MTTA, MTTR, on-call burden, incident volume by service/team, service health score
- **Postmortems:** Auto-generated timelines, integration with Confluence/Jira/Google Docs

## Architecture Overview

### Core Data Model

```
Integration -> Alert -> Incident -> (Acknowledge | Escalate) -> Resolve
                                                                    |
Service (owner) <- Escalation Policy <- On-Call Schedule          Postmortem
```

**Services:** Team-owned components (Payment API, Auth Service). Each service has integration key, escalation policy, alert grouping, and urgency rules.

**Escalation Policies:** Define who gets notified and when. Layers escalate if current level does not acknowledge within timeout.

**On-Call Schedules:** Define who is currently on-call. Referenced by escalation policy layers.

### Events API v2

Primary inbound API. Endpoint: `https://events.pagerduty.com/v2/enqueue`

```json
{
  "routing_key": "<integration_key>",
  "event_action": "trigger",
  "dedup_key": "svc-checkout/high-error-rate",
  "payload": {
    "summary": "Checkout API error rate > 5%",
    "severity": "critical",
    "source": "datadog-monitor-12345",
    "component": "checkout-api",
    "custom_details": {
      "error_rate": "7.2%",
      "runbook": "https://wiki/runbooks/checkout-errors"
    }
  }
}
```

**Event actions:** `trigger` (create alert), `acknowledge` (ack alert), `resolve` (close alert). Same `dedup_key` updates existing alert state.

## Incident Lifecycle

```
TRIGGERED -> ACKNOWLEDGED -> RESOLVED
                |
           (no ack within timeout)
                |
           ESCALATED -> next layer
```

**Priority levels:** P1 (Critical) through P5 (Informational). Priority affects stakeholder notifications, SLA tracking, and response play triggers.

## On-Call Patterns

**Follow-the-sun:** Multiple schedules per region with handoff at local business hours. Reduces after-hours burden.

**Primary/Secondary:** Two schedules in escalation policy layers 1 and 2. Secondary auto-escalated if primary does not acknowledge.

**Specialty rotation:** Separate schedule per domain (database on-call, network on-call) from service-specific escalation policies.

## Key Integrations

| Source | Method |
|--------|--------|
| Datadog | Native OAuth, bi-directional |
| Prometheus/Alertmanager | Native webhook receiver |
| AWS CloudWatch | SNS to PagerDuty webhook |
| Grafana | Webhook contact point (Events API v2) |
| Zabbix | HTTP media type with routing key |
| Nagios | pagerduty-nagios plugin |
| Dynatrace | Native problem notification |
| Slack | Bi-directional (ack/resolve from Slack) |
| Jira | Bi-directional incident/issue sync |
| ServiceNow | Bi-directional ITSM sync |

## Top 10 Operational Rules

1. **One service per team-owned component** -- Avoid monolithic "all alerts" services. Name descriptively: `payments-api`, `auth-service`.
2. **Always two escalation layers minimum** -- Primary on-call + secondary/manager. Set timeout to 5 min for P1/P2.
3. **Use schedules, not individuals** -- Never point escalation policies at a person. Use schedules for resilience.
4. **Enable intelligent alert grouping** -- Event Intelligence consolidates related alerts into one incident, reducing noise.
5. **Set dedup_key consistently** -- Prevents duplicate incidents from the same source. Include service name and check identifier.
6. **Configure transient alert suppression** -- Brief recovery-then-re-trigger patterns (flapping) generate noise. Suppress transient alerts.
7. **Set low-urgency for non-actionable alerts** -- Creates incidents without paging. Reserve high-urgency for items requiring immediate human response.
8. **Model service dependencies** -- When parent service has active incident, child incidents are suppressed. Prevents alert storms.
9. **Review MTTA/MTTR weekly** -- High MTTA indicates schedule/escalation gaps. High MTTR indicates tooling or process gaps.
10. **Automate with response plays** -- P1 incidents should auto-attach conference bridge, add responders, and post status update template.

## Common Pitfalls

**1. Monolithic services**
One service receiving all alerts from all systems. No ownership, no routing, no grouping. Split into component-level services.

**2. Escalation policies pointing to individuals**
When that person is unavailable, incidents go unacknowledged. Always use schedules.

**3. No dedup_key in Events API calls**
Every event creates a new incident. System flapping creates dozens of incidents per hour. Always set `dedup_key`.

**4. High-urgency for everything**
When every alert pages, on-call learns to ignore pages. Reserve high-urgency for actionable items. Use low-urgency for informational.

**5. No alert grouping**
A database outage triggers 50 service monitors. Without grouping, on-call receives 50 separate pages. Enable intelligent grouping.

**6. Never reviewing analytics**
Teams that do not track MTTA/MTTR cannot identify systematic problems. Review weekly, set SLA targets per priority.

## User Roles

| Role | Access |
|------|--------|
| Responder | Acknowledge/resolve incidents, manage own schedule |
| Manager | Service configuration, escalation policies, team management |
| Admin | Account settings, integrations, billing |
| Global Admin | Organization-wide configuration |
| Stakeholder | Read-only incident visibility (lower-cost license) |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Services, integrations (Events API v2, common inbound/outbound, webhooks v3), on-call management (schedule types, overrides, rotation patterns, escalation policies), incident response (lifecycle, priority, response plays, postmortems), analytics (MTTA, MTTR, on-call burden). Read for setup and integration questions.
- `references/best-practices.md` -- Service design patterns, escalation policy design, noise reduction (grouping, dedup, suppression), SLA management, service dependency modeling. Read for operations and optimization questions.