Prometheus

---
name: prometheus
description: >-
  Query Prometheus-compatible metrics endpoints (self-hosted, Grafana Cloud Prom, AMP, etc.)
  via the prometheus action and as a dashboard panel source.
---

# Prometheus

Direct integration with the Prometheus HTTP API. Works against any Prometheus-compatible endpoint: self-hosted Prometheus, Grafana Cloud, Amazon Managed Prometheus, Thanos, Cortex, Mimir.

## Credentials

| Env Var                   | Required | Notes                                                                       |
| ------------------------- | -------- | --------------------------------------------------------------------------- |
| `PROMETHEUS_URL`          | yes      | Base URL, no trailing slash, e.g. `https://prometheus.example.com`.         |
| `PROMETHEUS_USERNAME`     | no       | Basic auth username. Grafana Cloud uses the stack instance ID here.         |
| `PROMETHEUS_PASSWORD`     | no       | Basic auth password / API token. Must be paired with `PROMETHEUS_USERNAME`. |
| `PROMETHEUS_BEARER_TOKEN` | no       | Bearer token. Used only when no full `USERNAME` + `PASSWORD` pair is set.   |

Auth selection is deterministic: full basic-auth pair wins → bearer token → no `Authorization` header. Partial basic (username XOR password) is treated as no basic.

## Agent usage — `pnpm action prometheus`

| Mode           | Args                                                              | Purpose                                    |
| -------------- | ----------------------------------------------------------------- | ------------------------------------------ |
| `query`        | `--query <promql> [--time <RFC3339>]`                             | Instant query (default).                   |
| `query_range`  | `--query <promql> --start <RFC3339> --end <RFC3339> [--step 30s]` | Range query for time series.               |
| `labels`       |                                                                   | List all label names.                      |
| `label_values` | `--label <name>`                                                  | Values for one label (good for discovery). |
| `series`       | `--match '["up{job=\"api\"}"]'`                                   | Series matching one or more matchers.      |
| `metadata`     | `[--metric <name>]`                                               | Metric metadata (type, help text, unit).   |
| `alerts`       |                                                                   | Currently firing alerts.                   |

Step is auto-calculated when omitted (~250 points across the range, clamped to a 15s minimum). When the user asks "what metrics are available", start with `labels`, then `label_values --label=__name__`.

## Dashboard panels

Prometheus is a valid panel `source`. The `sql` field is still a string in the
dashboard config; put the serialized JSON descriptor in that string, not a
parsed object.

```json
"{\"promql\":\"rate(http_requests_total{job=\\\"api\\\"}[5m])\",\"mode\":\"range\",\"range\":\"1h\",\"step\":\"30s\"}"
```

Defaults: `mode=range`, `range=1h`, `step=auto`. Use `mode=instant` only for `metric` / `callout` chart types.

The dispatcher returns one row per (timestamp, series) with shape `{timestamp, series, value}`. Set panel `config` so charts render correctly:

```json
{
  "xKey": "timestamp",
  "yKey": "value"
}
```

When a query fans out into many series, the `series` column will hold a `metric_name{k1="v1",k2="v2"}` label per row — that's the natural grouping for a line chart with multiple series.

## Node Exporter dashboards

Use `list-sql-dashboards` to find an existing Node Exporter dashboard, then use
`update-dashboard` to create or repair one from the supported panel definitions:

```bash
```

`node-exporter-macos` is the comprehensive Darwin/Homebrew dashboard for macOS scrapes; it covers CPU, load, macOS memory fields, swap, APFS filesystems, disk IO, network devices, battery/power, scrape health, exporter process stats, Go runtime stats, and a metric-family coverage table. `node-exporter-full` is converted from Grafana dashboard 1860 revision 45 and lives at `seeds/dashboards/node-exporter-full.json`; it contains 124 Prometheus query panels, 16 section dividers, and native tabs for Overview, CPU & Memory, System, Storage, Network, and Exporter. The Full dashboard is Linux-focused because Grafana 1860 expects Linux collectors such as `node_memory_MemAvailable_bytes`, pressure stall information, `node_sockstat_*`, `node_netstat_*`, `node_timex_*`, `node_systemd_*`, and `node_hwmon_*`; Homebrew/macOS node_exporter omits many of those, so empty panels on macOS are expected. The templates use `job`, `instance`, and `range` filters; set `instance` to the Prometheus `instance` label from `node_uname_info`.

### Local setup via Homebrew (macOS)

```bash
brew install node_exporter prometheus
brew services start node_exporter   # metrics at http://localhost:9100/metrics
```

Edit `/opt/homebrew/etc/prometheus.yml`:

```yaml
global:
  scrape_interval: 1s

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]
```

```bash
brew services restart prometheus    # Prometheus UI at http://localhost:9090
```

Paste `http://localhost:9090` as the Prometheus URL in Data Sources, then use
`list-sql-dashboards` and `update-dashboard` to manage the desired Node Exporter
dashboard.

## Incident Investigation Pattern

**Query real metrics FIRST, analyze code second.** When investigating production
issues (spikes, outages, errors, performance degradation):

1. **Query actual metrics FIRST** — use Prometheus/Grafana, first-party
   Analytics Monitoring -> Errors, external Sentry, and Cloud Logging before
   looking at code. Real data tells you what happened; code only tells you what
   _could_ happen.
2. **Check upstream dependencies** — many incidents are caused by provider-side
   degradation (LLM APIs, external services), not local code. Query latency
   metrics for each upstream call.
3. **Trace the request flow** — identify which endpoints are involved, what
   external calls they make, and where connections can pile up.
4. **Look for cascade patterns** — upstream slowdown → connection pileup →
   autoscaling spike → retry flood → outage.
5. **Only analyze code/config after you have data** — deployment templates,
   autoscaling settings, and concurrency config are useful context but should
   not be the primary investigation method.

Common Prometheus queries for incident triage:

```promql
# Request rate by endpoint
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# P95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# LLM latency by model (if instrumented)
histogram_quantile(0.95, rate(llm_latency_bucket[5m])) by (model)
```

## Gotchas

- **Range queries return matrices**; instant queries return vectors. Both flatten to `{timestamp, series, value}` rows so charts work uniformly.
- **No query-result caching.** Metadata endpoints (`labels`, `label_values`, `metadata`) are cached for 10 minutes; `query` / `query_range` are never cached because they're time-sensitive.
- **Cardinality matters.** A wide-fanout PromQL produces many series and a busy chart. Reduce cardinality with explicit label matchers or aggregation (e.g. `sum by (job)(rate(...))`) before saving the panel.
- **HTTP 200 with `status: "error"`** is treated as a failure. The error message from the Prometheus response is surfaced verbatim.
- **No `db-query` for this.** Prometheus is its own backend. Do not try to read app DB tables to answer Prometheus questions.