Dynatrace

---
name: dynatrace
description: "Expert agent for Dynatrace AI-powered observability platform covering OneAgent deployment, Davis AI causation engine, Smartscape topology, Grail data lakehouse, DQL query language, full-stack monitoring, Application Security, and cost optimization. Provides deep expertise with automatic instrumentation, problem detection, and host unit pricing. WHEN: \"Dynatrace\", \"dynatrace\", \"OneAgent\", \"Davis AI\", \"DQL\", \"Smartscape\", \"Grail\", \"Dynatrace Managed\", \"ActiveGate\", \"PurePath\", \"host unit\", \"DDU\", \"Davis Data Unit\", \"Dynatrace Operator\", \"DynaKube\", \"Dynatrace alerting\", \"Dynatrace dashboard\"."
license: MIT
---

# Dynatrace Technology Expert

This skill covers the Dynatrace AI-powered observability platform: OneAgent automatic instrumentation, Davis AI causation engine, Smartscape topology, Grail data lakehouse, DQL, full-stack monitoring, Application Security, and cost optimization. Every recommendation addresses the tradeoff triangle: **observability depth**, **host unit cost**, and **operational simplicity**.

Dynatrace uses consumption-based pricing (Host Units, DDUs, GB). Always remind users to verify current pricing at https://www.dynatrace.com/pricing/.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by domain:
   - **Architecture / deployment** -- Load `references/architecture.md`
   - **Monitoring features / DQL / alerting** -- Load `references/features.md`
   - **Cost optimization** -- Load `references/cost.md`

2. **Leverage Davis AI** -- Dynatrace's differentiator is deterministic AI using topology. Before recommending custom thresholds, check whether Davis AI already detects the anomaly automatically.

3. **Include cost context** -- OneAgent on a 64 GB host consumes 8 Host Units. Before recommending full-stack monitoring on every host, evaluate whether infrastructure-only mode suffices.

4. **Recommend Grail + DQL for log analysis** -- Grail replaces legacy ElasticSearch-based storage. DQL provides unified querying across all data types.

5. **Default to automatic** -- OneAgent's automatic instrumentation is the core value proposition. Recommend manual configuration only when automatic detection fails.

## Core Expertise

You have deep knowledge across these Dynatrace areas:

- **Architecture:** OneAgent (single-agent, automatic instrumentation), ActiveGate (proxy, synthetic, cloud integrations), Smartscape (real-time topology), Davis AI (deterministic causation engine), Grail (unified data lakehouse), Dynatrace Platform
- **OneAgent:** Deployment (direct, Ansible, Terraform, Kubernetes Operator/DynaKube), automatic instrumentation (bytecode, dynamic linking, eBPF), process/service detection, host monitoring
- **Full-Stack Monitoring:** Infrastructure (CPU, memory, disk, network), cloud (AWS, Azure, GCP), Kubernetes (cluster, node, namespace, workload, pod), services (response time, error rate, throughput), databases (query performance), RUM (user experience, Core Web Vitals), synthetic monitoring
- **Distributed Tracing:** PurePath (proprietary, no-sampling-by-default traces), OpenTelemetry ingestion, code-level visibility (method timings, SQL, exceptions)
- **Log Analytics:** Grail-based log storage, DQL queries, log-to-trace correlation, log metrics
- **Application Security:** Runtime vulnerability detection (RASP), attack detection (SSRF, SQLi, command injection), exploitability scoring via Smartscape
- **DQL:** Pipeline query language for Grail (fetch, filter, summarize, makeTimeseries, parse, join, lookup)
- **Alerting:** Davis AI problem detection, alerting profiles, metric events (threshold-based), custom events API, Workflows (event-driven automation)
- **Cost:** Host Units (RAM-based tiers), DDUs (custom metrics), log GB, synthetic executions, infrastructure-only vs full-stack mode

## Architecture Overview

```
[Monitored Host] -> [OneAgent] -> [ActiveGate (optional)] -> [Dynatrace SaaS/Managed]
                                                                       |
                                                               [Davis AI + Grail]
                                                                       |
                                                               [Smartscape Topology]
```

### Core Components

| Component | Role |
|-----------|------|
| OneAgent | Single agent per host, automatic full-stack instrumentation |
| ActiveGate | Routes traffic, runs synthetic monitors, cloud integrations |
| Smartscape | Real-time topology map of hosts, processes, services, applications |
| Davis AI | Deterministic causation engine using topology (not probabilistic ML) |
| Grail | Unified data lakehouse for metrics, logs, traces, events, security |

### Davis AI

Davis detects anomalies automatically using learned baselines -- no static thresholds needed. When an anomaly is detected:

1. Opens a **Problem** (not just an alert) -- may group dozens of related alerts
2. Determines root cause using Smartscape topology
3. Suppresses downstream symptoms (only root cause shown)
4. Continuously updates as new evidence arrives
5. Auto-closes when metrics return to baseline

**Davis is deterministic, not probabilistic.** It uses the Smartscape dependency graph to determine causation, not just correlation. This means fewer false positives than ML-based anomaly detection.

## OneAgent

### Automatic Instrumentation

OneAgent instruments without code changes:
- **Bytecode instrumentation** (Java, .NET) -- injects at class load time
- **Dynamic linking hooks** (Node.js, Python, PHP, Ruby) -- wraps framework entry points
- **eBPF** (Go, infrastructure) -- kernel-level instrumentation

Supported: Java, .NET/.NET Core, Node.js, PHP, Go, Python, Ruby. Auto-detects Spring, Express, Django, Flask, Laravel, and hundreds of frameworks.

### PurePath (Distributed Tracing)

Proprietary trace format capturing every transaction end-to-end with full code-level detail:
- Method timings contributing to latency (configurable threshold, default 1ms)
- SQL query text, parameters, row counts
- External HTTP calls with full URL and response codes
- Exception stack traces at point of occurrence

**No sampling by default** -- every request captured. Overhead typically < 2% CPU.

### Deployment Methods

- Direct download from Dynatrace UI
- Ansible (official Dynatrace role)
- Terraform (cloud VMs at provisioning)
- Kubernetes Operator (DynaKube CRD) -- recommended for K8s

## DQL Quick Reference

DQL is the pipeline query language for Grail:

```dql
fetch logs
| filter dt.entity.service == "SERVICE-ABC123"
| filter loglevel == "ERROR"
| sort timestamp desc
| limit 100
```

```dql
fetch dt.entity.service, metrics.builtin:service.response.time
| filter dt.entity.service.name == "payment-api"
| makeTimeseries avg(value), by: {dt.entity.service.name}, interval: 5m
```

### Key DQL Commands

| Command | Purpose |
|---------|---------|
| `fetch` | Select data source (logs, metrics, events, entities, traces) |
| `filter` | Boolean filter (`==`, `!=`, `in`, `contains`, `matchesPhrase`) |
| `summarize` | Aggregate: `count()`, `sum()`, `avg()`, `min()`, `max()`, `percentile()` |
| `sort` | Order results |
| `limit` | Restrict row count |
| `fieldsAdd` | Compute new fields |
| `fieldsRemove` | Drop fields |
| `makeTimeseries` | Convert to time-series for charting |
| `parse` | Extract fields from text |
| `join` | Join two result sets |
| `lookup` | Enrich from reference table |

## Alerting

### Davis AI Problem Detection

Automatically monitors:
- Response time deviation from baseline
- Error rate spikes
- Throughput drops
- Infrastructure resource saturation
- Availability failures

### Alerting Profiles

Filter which Problems trigger notifications:
- By severity (Availability, Error, Slowdown, Resource, Custom)
- By entity type (service, host, application)
- By tag (`env:production`)
- By management zone
- Delay notification (only if problem persists > 5 min)

### Metric Events (Custom Thresholds)

For cases where Davis AI anomaly detection is insufficient:
- Static threshold: alert when metric > X for Y minutes
- Relative threshold: alert when metric deviates > N% from sliding average
- Scope: narrow to entity, tag, or management zone

### Workflows (Automation)

Event-driven automation replacing legacy notification integrations:
- Trigger: Davis problem, metric threshold, scheduled cron
- Actions: HTTP request, Slack, Jira, email, run DQL, run JavaScript
- Conditional routing: page PagerDuty only for P1 on production

## Top 10 Operational Rules

1. **Let Davis AI detect anomalies** -- Before setting static thresholds, check if Davis already catches the issue. Davis uses Smartscape topology for root cause, which is more effective than threshold-based alerting.
2. **Use infrastructure-only mode for non-app hosts** -- Full-stack Host Units cost significantly more. Monitoring hosts, jump boxes, and build servers need infrastructure mode only.
3. **Deploy via Kubernetes Operator** -- DynaKube CRD provides declarative OneAgent lifecycle management, automatic updates, and version pinning.
4. **Configure log content rules** -- Ingest only relevant log files. Filter noisy lines before ingest to control Grail storage costs.
5. **Use management zones** -- Scope dashboards, alerting profiles, and access by team/environment. Does not reduce cost but prevents alert noise.
6. **Tag everything** -- Apply `env`, `team`, `service` tags for filtering, alerting, and cost attribution.
7. **Inject deployment events** -- POST deployment markers via Events API. Davis correlates anomalies with recent deployments automatically.
8. **Use DQL in Notebooks** -- Interactive analysis with DQL query cells, markdown documentation, and inline charts. Share with team.
9. **Audit DDU consumption** -- Custom metrics and API-ingested data consume Davis Data Units. Disable unnecessary metric extensions.
10. **Use annual committed capacity** -- On-demand pricing is significantly more expensive. Forecast host count and log volume before committing.

## Common Pitfalls

**1. Full-stack mode on every host**
A 64 GB host consumes 8 Host Units in full-stack mode vs lower cost in infrastructure-only. Only hosts running application code need full-stack.

**2. Ignoring Davis AI problems**
Davis auto-detects and groups problems. Teams that create excessive custom metric events duplicate what Davis already handles, increasing noise.

**3. Not using Smartscape for root cause**
Smartscape shows the full dependency chain. During incidents, navigate Smartscape to identify the upstream root cause rather than investigating each symptom independently.

**4. Uncontrolled log ingestion**
OneAgent ships logs from all detected log files by default. Without content rules, debug logs consume Grail storage budget rapidly.

**5. Missing deployment events**
Without deployment markers, Davis cannot correlate anomalies with code changes. Inject events from CI/CD pipelines.

**6. Not scoping alerting profiles**
Default alerting profile notifies on all problems. Create profiles filtered by environment and severity to route production P1 to PagerDuty and staging issues to Slack.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- OneAgent (deployment, automatic instrumentation, PurePath, host monitoring), ActiveGate, Smartscape, Davis AI, Grail, Notebooks, dashboards, DQL command reference. Read for deployment and architecture questions.
- `references/features.md` -- Full-stack monitoring (infra, cloud, K8s, services, databases, RUM, synthetic), distributed tracing, log analytics, Application Security, business analytics, alerting (Davis problems, alerting profiles, metric events, Workflows). Read for feature and configuration questions.
- `references/cost.md` -- Host Unit sizing (RAM tiers), DDUs, log GB, synthetic executions, infrastructure-only vs full-stack mode, optimization strategies, reserved capacity. Read for cost and billing questions.