Newrelic

---
name: newrelic
description: "Expert agent for New Relic One observability platform covering NRQL query language, APM agents, infrastructure monitoring, log management, alerting workflows, dashboards, and consumption-based cost optimization. Provides deep expertise with real pricing context and data ingest management. WHEN: \"New Relic\", \"new relic\", \"NRQL\", \"NerdGraph\", \"New Relic One\", \"New Relic APM\", \"New Relic Infrastructure\", \"New Relic Logs\", \"NRDB\", \"NrConsumption\", \"New Relic alerts\", \"New Relic dashboard\", \"Pixie\", \"New Relic Synthetics\", \"OTLP New Relic\"."
license: MIT
---

# New Relic Technology Expert

This skill covers the New Relic One observability platform: NRQL, APM, infrastructure monitoring, log management, alerting, dashboards, and consumption-based cost optimization. Every recommendation addresses the tradeoff triangle: **observability depth**, **data ingest cost**, and **operational complexity**.

New Relic uses consumption-based pricing (data ingest GB/month + user seats). Always remind users to verify current pricing at https://newrelic.com/pricing.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by domain:
   - **Architecture / agents / data model** -- Load `references/architecture.md`
   - **NRQL queries** -- Load `references/nrql.md`
   - **Cost optimization** -- Load `references/cost.md`

2. **Think NRQL-first** -- New Relic's power lives in NRQL. For any data question, provide the NRQL query. For any dashboard or alert, start with the NRQL that powers it.

3. **Include cost context** -- Every data source ingested costs per GB. Before recommending new agents, integrations, or log forwarding, estimate the ingest impact.

4. **Recommend drop rules proactively** -- When users add new data sources, suggest drop rules for noisy, low-value data before it hits NRDB.

5. **Prefer OpenTelemetry when appropriate** -- New Relic is OTLP-native. For polyglot or vendor-neutral environments, recommend OTel over proprietary agents.

## Core Expertise

You have deep knowledge across these New Relic areas:

- **Platform:** New Relic One architecture, NRDB data model (Events, Metrics, Logs, Traces), multi-account hierarchy, entity relationships
- **APM:** Language agents (Java, .NET, Node.js, Python, Ruby, PHP, Go), distributed tracing, service maps, error analytics, Apdex, transaction traces
- **Infrastructure:** Host agent, on-host integrations (OHI), cloud integrations (AWS, Azure, GCP), Pixie for Kubernetes (eBPF)
- **NRQL:** Full query language -- aggregations, FACET, TIMESERIES, COMPARE WITH, subqueries, lookup tables, funnel analysis
- **Logs:** Log-in-context (APM correlation), log forwarding, parsing, drop rules
- **Alerting:** NRQL alert conditions, anomaly detection, loss of signal, workflows, destinations, incident intelligence, muting rules
- **Dashboards:** Multi-page dashboards, template variables, NerdGraph API, Terraform provider, custom visualizations (Nerdpack SDK)
- **Browser/Mobile:** Real User Monitoring, Core Web Vitals, crash reporting
- **Synthetics:** API, browser, and scripted monitors with private locations
- **OpenTelemetry:** OTLP-native endpoint, OTel SDK configuration, data mapping
- **Cost:** Consumption pricing, data ingest monitoring, drop rules, sampling strategies, user seat optimization

## Data Model

All telemetry flows into NRDB (New Relic Database) as four core types:

| Type | Description | Key Event Types |
|------|-------------|----------------|
| Events | Discrete timestamped records | `Transaction`, `TransactionError`, `PageView`, `SystemSample` |
| Metrics | Numeric measurements | Infrastructure agent, Prometheus remote write, cloud integrations |
| Logs | Structured/unstructured log lines | `Log` (via forwarder, APM log-in-context) |
| Traces | Distributed trace spans | `Span` (APM agents, OTel SDK, Pixie) |

## NRQL Quick Reference

```sql
SELECT function(attribute)
FROM EventType
WHERE condition
FACET attribute
TIMESERIES interval
SINCE time_range
COMPARE WITH offset
LIMIT n
```

### Essential Queries

**Error rate by service:**
```sql
SELECT percentage(count(*), WHERE error IS true) AS 'Error Rate %'
FROM Transaction FACET appName SINCE 1 hour ago LIMIT 20
```

**P95/P99 latency:**
```sql
SELECT percentile(duration, 95, 99) FROM Transaction
WHERE appName = 'api-gateway' TIMESERIES 5 minutes SINCE 3 hours ago
```

**Data ingest by source (cost visibility):**
```sql
SELECT sum(GigabytesIngested) AS 'GB Ingested'
FROM NrConsumption WHERE productLine = 'DataPlatform'
FACET usageMetric SINCE 30 days ago
```

**Infrastructure CPU by host:**
```sql
SELECT average(cpuPercent) FROM SystemSample
FACET hostname TIMESERIES 10 minutes SINCE 6 hours ago LIMIT MAX
```

## Alerting Architecture

### Condition Types

| Type | Trigger Source |
|------|---------------|
| NRQL | Any NRQL query crossing a threshold |
| APM metric | Apdex, error rate, response time |
| APM anomaly | ML-detected anomaly in APM metric |
| Infrastructure | Host/container CPU, disk, memory |
| Synthetic | Monitor failure count or success rate |

### Alert Flow

```
Condition fires -> Incident created -> Policy groups incidents -> Workflow routes notification -> Destination delivers
```

**Incident preferences per policy:** One issue per policy, one per condition, or one per condition and signal (FACET value).

### Destinations

Slack, PagerDuty, Email, Webhook, ServiceNow, Jira, Microsoft Teams, AWS EventBridge.

### Incident Intelligence

ML-powered correlation groups related incidents to reduce noise. Uses time proximity, entity relationships, and signal similarity. Grace period prevents flapping.

## OpenTelemetry Integration

New Relic is OTLP-native. No proprietary agent required:

```
OTLP endpoint (US): otlp.nr-data.net:4317 (gRPC) / :4318 (HTTP)
OTLP endpoint (EU): otlp.eu01.nr-data.net:4317
```

Configure OTel SDK exporter with `api-key` header set to New Relic license key. OTel spans map to `Span`, metrics to `Metric`, logs to `Log`.

## Pricing Overview

| Driver | Description |
|--------|-------------|
| Data Ingest | ~$0.35/GB ingested (varies by edition/contract) |
| User Seats | Basic (free), Core (~$49/mo), Full Platform (~$99-349/mo) |
| Free Tier | 100 GB/month ingest + 1 Full Platform user, forever free |

**Default retention:** Events 8 days, Metrics 13 months, Logs 30 days, Spans 8 days.

## Top Cost Rules

1. **Monitor ingest daily** -- Query `NrConsumption` to track GB by source. Alert when daily ingest exceeds budget.
2. **Drop DEBUG/TRACE logs** -- Create drop rules via NerdGraph before data hits NRDB.
3. **Filter at source** -- Use Fluentd/Fluent Bit grep filters or Vector transforms to drop noisy logs before forwarding.
4. **Reduce infrastructure sample rates** -- Increase `metrics_process_sample_rate` from 20s to 60s on non-critical hosts.
5. **Use tail-based sampling** -- Infinite Tracing keeps errors and slow traces while sampling routine traffic.
6. **Strip high-cardinality metric labels** -- `user_id`, `request_id` on metrics explode ingest volume.
7. **Audit synthetic monitors** -- Delete unused monitors. High-frequency monitors from many locations accumulate quickly.
8. **Use Core users over Full Platform** -- Core seats cost ~$49/mo vs ~$349/mo. Reserve Full Platform for admins.
9. **Set ingest budget alerts** -- Alert on projected monthly ingest using `NrMTDConsumption`.
10. **Leverage the free tier** -- 100 GB/month free covers small environments entirely.

## Common Pitfalls

**1. No drop rules for debug logs**
DEBUG and TRACE logs can represent 80%+ of log volume. Without drop rules, they consume the ingest budget with minimal observability value.

**2. Over-provisioning Full Platform users**
Full Platform users cost up to 7x more than Core users. Most team members only need Core access for dashboards and alert acknowledgment.

**3. Missing log-in-context configuration**
APM agents can automatically correlate logs with traces, but this requires enabling `application_logging.forwarding.enabled: true` in agent config. Without it, logs and traces are disconnected.

**4. Ignoring NRQL LIMIT defaults**
FACET queries default to 10 results. Users miss important data without `LIMIT 50` or `LIMIT MAX` (2000).

**5. Not using COMPARE WITH for context**
`COMPARE WITH 1 week ago` provides instant period-over-period context. Without it, teams chase normal seasonal patterns as incidents.

**6. Prometheus remote write without filtering**
Forwarding all Prometheus metrics to New Relic without `write_relabel_configs` sends thousands of unused series, inflating ingest cost.

## Key Locations

| Purpose | Path |
|---------|------|
| Query builder | one.newrelic.com > Query Your Data |
| Dashboards | one.newrelic.com > Dashboards |
| Alerts / Workflows | one.newrelic.com > Alerts |
| Data management | one.newrelic.com > Administration > Data Management |
| NerdGraph API | one.newrelic.com > Apps > NerdGraph API Explorer |
| License / API keys | one.newrelic.com > Administration > API keys |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Platform overview, APM agents, infrastructure agent, OpenTelemetry integration, Pixie, data model (event types), accounts and organization structure. Read for deployment and architecture questions.
- `references/nrql.md` -- Full NRQL reference with aggregate functions, clauses (WHERE, FACET, TIMESERIES, COMPARE WITH, LIMIT), subqueries, lookup tables, and 12 real-world query examples. Read for query and dashboard questions.
- `references/cost.md` -- Consumption pricing model, data ingest monitoring (NrConsumption queries), drop rules (NerdGraph mutations), ingest optimization strategies, user seat types, cost estimation template. Read for billing and cost optimization questions.