Splunk Observability Dashboard Builder

---
name: splunk-observability-dashboard-builder
description: "Use when creating, planning, rendering, validating, or applying Splunk Observability Cloud dashboards
  from natural-language dashboard requests, JSON or YAML dashboard specs, SignalFlow chart definitions, or
  Observability dashboard-as-code workflows. Supports native classic Observability dashboard/chart APIs
  with render-first safety; treats modern dashboard sections, logs charts, service maps, and Dashboard
  Studio as advisory/secondary paths unless a verified API is available."
compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."
metadata:
  splunk_cloud_10_5: "not-applicable"
  compatibility_verified: "2026-08-20"
---

# Splunk Observability Dashboard Builder

## Prerequisites

| Tool or access | Purpose | Verify |
|---|---|---|
| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |
| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |
| Credential files for live modes | Keep secrets out of chat | Verify paths only |

## Workflow Overview

```text
┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘
```

## When to Activate

- Creating, planning, rendering, validating, or applying Splunk Observability Cloud dashboards from natural-language
  dashboard requests, JSON or YAML dashboard specs, SignalFlow chart definitions, or Observability dashboard-as-
  code.
- Preview and review the splunk observability dashboard builder workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

## Scope

Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.

## Examples

Inspect the supported setup modes before selecting one:

```bash
bash skills/splunk-observability-dashboard-builder/scripts/setup.sh --help
```

Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.

Inspect validation modes before running completion checks:

```bash
bash skills/splunk-observability-dashboard-builder/scripts/validate.sh --help
```

Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.

## Troubleshooting

| Issue | Cause | Resolution |
|---|---|---|
| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |
| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |
| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |
| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |

## Overview

Use this skill to turn a user's natural-language dashboard idea into a reviewed Splunk Observability Cloud dashboard specification, rendered API payloads, and optionally an applied native Observability dashboard.

The default path is **classic-api**: create custom dashboard groups, charts, and dashboards through the documented `/v2/dashboardgroup`, `/v2/chart`, and `/v2/dashboard` APIs. Modern dashboard features are documented as UI/advisory unless a public API is verified before use.

## Safety Rules

- Never ask for Splunk Observability API tokens, org access tokens, session tokens, passwords, or client secrets in conversation.
- Never pass tokens on the command line or as environment-variable prefixes.
- Require `--token-file` for live API operations.
- Reject direct token flags such as `--token`, `--access-token`, `--api-token`, `--o11y-token`, and `--sf-token`.
- Prefer `SPLUNK_O11Y_REALM` and `SPLUNK_O11Y_TOKEN_FILE` from the repo `credentials` file when present; these store only the realm and token-file path, not the token value.
- Render and validate before apply. Apply only when the user explicitly requests it.
- Create new custom dashboards by default. Use `--update-existing` only when the user explicitly asks and the spec includes existing dashboard/chart IDs.

## Primary Workflow

1. Interpret the request:
   - Identify audience, decision workflow, services or infrastructure scope, time range, dashboard group, filters, and desired visuals.
   - Ask only for missing non-secret values: realm, dashboard group name or ID, service/environment/cluster names, preferred dimensions, and target time range.
   - For vague requests, produce a starter dashboard and mark assumptions in the spec comments or final explanation.

2. Ground the dashboard in live metadata when possible:
   - Use `scripts/setup.sh --discover-metrics --realm <realm> --token-file <file> --query <term>` to discover metric names. Omit realm/token flags when `SPLUNK_O11Y_REALM` and `SPLUNK_O11Y_TOKEN_FILE` are configured in `credentials`. Simple bare terms such as `latency` are converted to `sf_metric:*latency*`.
   - Use metric and dimension names returned by the API instead of inventing names.
   - If live metadata is unavailable, render a reviewable draft and clearly mark metric names as assumptions.

3. Write or update a JSON or YAML spec:
   - Start from `templates/dashboard.example.json` for a dependency-free example, or `templates/dashboard.example.yaml` when PyYAML is installed.
   - Keep `mode: classic-api` for renderable native Observability dashboards.
   - Use `mode: modern-ui-advisory` or `mode: dashboard-studio-advisory` only to document UI/manual work.

4. Validate and render:

   ```bash
   bash skills/splunk-observability-dashboard-builder/scripts/setup.sh \
     --render \
     --spec skills/splunk-observability-dashboard-builder/templates/dashboard.example.json \
     --output-dir splunk-observability-dashboard-rendered
   ```

5. Review the rendered plan:
   - `metadata.json` summarizes coverage, assumptions, and warnings.
   - `charts/*.json` contains chart API payloads.
   - `dashboard.json` contains the dashboard API payload with chart placeholders.
   - `apply-plan.json` records the creation sequence without secrets.

6. Apply only when explicitly requested:

   ```bash
   bash skills/splunk-observability-dashboard-builder/scripts/setup.sh \
     --apply \
     --spec my-dashboard.json \
     --realm us0 \
     --token-file /tmp/splunk_o11y_api_token
   ```

   > By default, `--apply` creates new dashboard groups, charts, and
   > dashboards. Use `--update-existing` only with explicit existing object
   > IDs in the spec; the apply client fetches the current objects before PUT
   > so omitted writable fields are preserved where the API returns them.
   >
   > `--dry-run` is non-destructive and skips both the API calls and the
   > readable-token-file requirement, so CI preview-only jobs do not need
   > a real token path on disk. Live `--apply` retries 429/502/503/504
   > automatically with exponential backoff (cap: 4 attempts; honors
   > `Retry-After` when present).

7. Clean up validation smoke dashboards when needed:

   ```bash
   bash skills/splunk-observability-dashboard-builder/scripts/setup.sh \
     --cleanup \
     --apply-result splunk-observability-dashboard-rendered/apply-result.json
   ```

   Cleanup is intentionally guarded to rendered plans whose dashboard group and
   dashboard names start with `codex_live_validation`. Use the Observability UI
   or API directly for non-validation dashboards.

## Coverage Rules

- **Fully renderable through classic-api**: custom dashboard groups, custom dashboards, TimeSeriesChart, SingleValue, List, TableChart, Heatmap, Text charts, event overlays, dashboard filters/variables, chart time ranges, units, legends, thresholds, and detector links represented by chart properties.
- **Documented but not currently API-rendered by this skill**: pie/donut charts and event feed charts. Current product docs list them, but this renderer needs a verified classic `/v2/chart` schema before applying them. Use `modern-ui-advisory` or Text/link notes for now.
- **Metric-derived coverage**: Infrastructure Monitoring, Kubernetes/cloud infrastructure, Database Monitoring, APM service RED metrics, RUM Browser/Mobile metrics, Synthetic Monitoring metrics, Log Observer Connect metrics/log links, custom business metrics, AI Infrastructure Monitoring, and AI Agent/APM metrics when the required metrics are present.
- **Advisory/link-only coverage**: trace waterfalls, RUM sessions and session replay, synthetic waterfall detail, database explain plans, alerts/detector management, On-Call workflows, Observability Cloud for Mobile app workflows, product-native navigators, modern dashboard sections/subsections, service maps, and modern logs charts. Do not claim API rendering for these without verifying a public API first.
- **Dashboard Studio**: keep as a secondary path. It has Splunk platform version, realm, capability, trial, Unified Identity, and import limitations. Do not mix it into the native Observability apply path.

Read `references/coverage.md` for the full product coverage matrix and `references/classic-api.md` for API field guidance.

YAML specs require PyYAML in the Python interpreter used by `scripts/setup.sh`. Install repo dependencies with `python3 -m pip install -r requirements-agent.txt`, or use JSON specs.

## Spec Guidance

- Prefer one dashboard per operational question. Put broad estate overviews and incident drilldowns in separate dashboards.
- Use dashboard variables for common drilldowns such as `sf_environment`, `service.name`, `k8s.cluster.name`, `k8s.namespace.name`, `host.name`, `cloud.region`, and `deployment.environment`.
- Choose chart types intentionally:
  - TimeSeriesChart for trends, rates, latency, throughput, saturation.
  - SingleValue for current health, active alerts, SLO/error-budget snapshots, latest values.
  - List or TableChart for top-N entities and inventory/status summaries.
  - Heatmap for distributions and dense population comparisons.
  - Text for operator notes, runbook links, and assumptions.
- Use `modern-ui-advisory` for pie/donut, event feed, logs, service map, section/tab, or other modern-only visualizations until the public API schema is verified.
- Every non-text chart must include SignalFlow with at least one `publish()` output.
- Keep layouts within a 12-column grid. The validator rejects collisions and out-of-range chart positions.

## Update Existing Objects

For existing charts or dashboards, use `--update-existing` with fetch-modify-PUT semantics:

1. Add `dashboard.id` and each chart's `chart_id` to the spec.
2. Render and review the plan.
3. Apply with `--apply --update-existing`.

Do not create partial PUT payloads. The Observability chart API can null or remove writable properties omitted from updates.

## Scripts

- `scripts/setup.sh` - shell entrypoint for render, validate, discover, apply, and guarded cleanup.
- `scripts/render_dashboard.py` - validates specs and renders classic API payloads.
- `scripts/validate_dashboard.py` - static validation for specs or rendered payloads.
- `scripts/o11y_dashboard_api.py` - live API client using token files only.

## Useful Commands

Validate a draft spec:

```bash
bash skills/splunk-observability-dashboard-builder/scripts/setup.sh \
  --validate \
  --spec skills/splunk-observability-dashboard-builder/templates/dashboard.example.json
```

Render without applying:

```bash
bash skills/splunk-observability-dashboard-builder/scripts/setup.sh \
  --render \
  --spec skills/splunk-observability-dashboard-builder/templates/dashboard.example.json \
  --output-dir splunk-observability-dashboard-rendered
```

Discover metrics before writing SignalFlow:

   ```bash
   bash skills/splunk-observability-dashboard-builder/scripts/setup.sh \
     --discover-metrics \
     --query latency
   ```