Elastic Security

---
name: elastic-security
description: "Expert agent for Elastic Security across all versions. Provides deep expertise in detection rules, EQL sequence detection, ES|QL piped analytics, Fleet/Elastic Agent management, Elastic Common Schema, ML anomaly detection, response actions, and Elasticsearch cluster operations. WHEN: \"Elastic Security\", \"Elastic SIEM\", \"EQL\", \"ES|QL\", \"Elastic Agent\", \"Fleet\", \"ECS\", \"Elastic detection rules\", \"Kibana security\", \"Elasticsearch cluster\"."
license: MIT
---

# Elastic Security

This skill covers Elastic Security across all supported versions (8.x through 9.x), with deep knowledge of:

- Detection rules engine (1,300+ prebuilt rules, custom rules)
- Event Query Language (EQL) for sequence-based detection
- ES|QL (Elasticsearch Query Language) for piped analytics (8.11+)
- Kibana Query Language (KQL) and Lucene query syntax
- Fleet and Elastic Agent for unified endpoint management
- Elastic Common Schema (ECS) for data normalization
- ML anomaly detection jobs for behavioral analytics
- Response actions (host isolation, process termination, file operations)
- Case management and investigation workflows
- Osquery integration for live host interrogation
- Elasticsearch cluster operations and scaling

## How to Approach Tasks

1. **Classify** the request:
   - **Detection engineering** -- Detection rules, EQL sequences, custom rules
   - **Investigation** -- ES|QL queries, KQL hunting, timeline analysis
   - **Architecture** -- Cluster sizing, Fleet deployment, agent management
   - **ML anomaly detection** -- Job configuration, anomaly interpretation
   - **Response** -- Response actions, case management, containment
   - **Cluster operations** -- Load `references/architecture.md`
   - **Optimization** -- Load `references/best-practices.md`

2. **Identify version** -- Elastic 8.x vs 9.x matters for ES|QL maturity, serverless availability, and feature access. Load the matching `references/versions/<v>.md` file.

3. **Check ECS compliance** -- Detection rules expect ECS-normalized data. Verify field mappings.

4. **Recommend** actionable guidance with query examples and Kibana configuration steps.

## Core Expertise

### Query Languages

Elastic Security supports multiple query languages for different use cases:

| Language | Use Case | Best For |
|---|---|---|
| **EQL** | Sequence detection, process trees | Ordered event sequences, parent-child relationships |
| **ES\|QL** | Piped analytics (8.11+) | Ad-hoc investigation, aggregations, SPL/KQL-like workflow |
| **KQL** | Quick filtering in Kibana | Dashboard filters, simple field searches |
| **Lucene** | Complex text search, regex | Full-text search, regular expressions, fuzzy matching |

### EQL (Event Query Language)

EQL is Elastic's language for ordered event detection -- detecting sequences of events:

```eql
// Detect credential dumping: process accessing LSASS memory
process where event.type == "start"
  and process.name == "rundll32.exe"
  and process.args : "comsvcs.dll*MiniDump*"
```

**Sequence detection (EQL's killer feature):**
```eql
// Detect: suspicious login followed by lateral movement within 10 minutes
sequence by user.name with maxspan=10m
  [authentication where event.outcome == "failure" and source.ip != null]
  [authentication where event.outcome == "success"]
  [process where event.type == "start" and process.name in ("psexec.exe", "wmic.exe", "wmiexec.py")]
```

**EQL sequence operators:**
- `sequence by field` -- Events must share the same field value
- `with maxspan=Nm` -- Events must occur within N time units
- `until [event]` -- Sequence is abandoned if this event occurs
- `[event] by field1 == field2` -- Join conditions between sequence events

```eql
// Detect lateral movement: authentication to multiple hosts
sequence by source.ip with maxspan=30m
  [authentication where event.outcome == "success"] by host.name
  [authentication where event.outcome == "success"] by host.name
  [authentication where event.outcome == "success"] by host.name
| filter length(unique(host.name)) >= 3
```

### ES|QL (Elasticsearch Query Language)

ES|QL is a piped query language (similar to SPL/KQL) introduced in 8.11:

```esql
FROM logs-*
| WHERE event.category == "authentication" AND event.outcome == "failure"
| STATS failure_count = COUNT(*), unique_users = COUNT_DISTINCT(user.name)
    BY source.ip
| WHERE failure_count > 20
| SORT failure_count DESC
| LIMIT 50
```

**ES|QL key commands:**

| Command | Purpose | Example |
|---|---|---|
| `FROM` | Source index/data stream | `FROM logs-endpoint*` |
| `WHERE` | Filter rows | `WHERE process.name == "cmd.exe"` |
| `STATS` | Aggregate | `STATS count = COUNT(*) BY host.name` |
| `EVAL` | Calculate fields | `EVAL duration = end - start` |
| `SORT` | Order results | `SORT count DESC` |
| `LIMIT` | Restrict rows | `LIMIT 100` |
| `KEEP` | Select columns | `KEEP @timestamp, user.name, source.ip` |
| `DROP` | Remove columns | `DROP message, tags` |
| `RENAME` | Rename columns | `RENAME src AS source_ip` |
| `DISSECT` / `GROK` | Parse strings | `DISSECT message "%{ip} %{method} %{path}"` |
| `ENRICH` | Lookup enrichment | `ENRICH geoip ON source.ip` |

### Detection Rules Engine

Elastic Security includes 1,300+ prebuilt detection rules with MITRE ATT&CK mapping:

**Rule types:**

| Type | Query Language | Use Case |
|---|---|---|
| **Custom query** | KQL or Lucene | Simple pattern matching |
| **EQL** | EQL | Sequence detection, process tree analysis |
| **ES\|QL** | ES\|QL (8.14+) | Piped analytics detections |
| **Threshold** | KQL | Count exceeds limit in time window |
| **Indicator match** | KQL + threat intel index | IOC matching against log data |
| **ML** | ML job | Anomaly detection based on behavioral models |
| **New terms** | KQL | Alert when a field value appears for the first time |

**Custom rule example:**
```json
{
  "name": "Suspicious PowerShell Encoded Command",
  "description": "Detects PowerShell with encoded commands",
  "risk_score": 73,
  "severity": "high",
  "type": "eql",
  "query": "process where event.type == \"start\" and process.name : \"powershell.exe\" and process.args : (\"-enc*\", \"-EncodedCommand*\", \"*[Convert]::FromBase64*\")",
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {"id": "TA0002", "name": "Execution"},
      "technique": [{"id": "T1059.001", "name": "PowerShell"}]
    }
  ],
  "tags": ["Windows", "PowerShell", "Execution"]
}
```

### Fleet and Elastic Agent

Fleet provides centralized management for Elastic Agents:

**Architecture:**
```
Fleet Server (runs on Elasticsearch cluster or dedicated host)
    |
    v
Agent Policies (define integrations and configurations)
    |
    v
Elastic Agents (deployed on endpoints)
    |
    ├── Endpoint Security integration (EDR capabilities)
    ├── System integration (OS-level logs and metrics)
    ├── Custom integrations (third-party data sources)
    └── Osquery integration (live host queries)
```

**Key concepts:**
- **Agent policies** -- Collections of integrations assigned to agents
- **Integrations** -- Data collection modules (400+ available)
- **Fleet Server** -- Coordination layer between Kibana and agents
- **Output configuration** -- Where agents send data (Elasticsearch, Logstash)

### Elastic Common Schema (ECS)

ECS defines a common field naming convention:

```json
{
  "@timestamp": "2026-04-08T12:00:00.000Z",
  "event.category": "process",
  "event.type": "start",
  "event.outcome": "success",
  "process.name": "powershell.exe",
  "process.args": ["-enc", "SQBFAFgA..."],
  "process.pid": 1234,
  "process.parent.name": "cmd.exe",
  "user.name": "jsmith",
  "host.name": "WORKSTATION-01",
  "source.ip": "10.0.0.50"
}
```

**ECS field categories:**
- `event.*` -- Event classification (category, type, outcome, action)
- `process.*` -- Process details (name, pid, args, parent)
- `user.*` -- User identity (name, domain, email)
- `host.*` -- Host details (name, os, ip)
- `source.*` / `destination.*` -- Network endpoints
- `file.*` -- File operations (name, path, hash)
- `network.*` -- Network metadata (transport, protocol, bytes)

### ML Anomaly Detection

Elastic's ML module detects anomalies in security data:

**Prebuilt security ML jobs:**
- Unusual process execution for user
- Rare domain name lookup
- Unusual network activity
- Abnormal Windows logon activity
- Unusual data transfer volume

**Custom ML job example:**
```json
{
  "description": "Detect unusual outbound data transfer",
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "function": "high_sum",
        "field_name": "destination.bytes",
        "over_field_name": "source.ip",
        "partition_field_name": "destination.geo.country_name"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  }
}
```

### Response Actions

Response actions enable direct containment from Elastic Security:

| Action | Description | Requirement |
|---|---|---|
| **Isolate host** | Network-isolate an endpoint | Elastic Defend integration |
| **Release host** | Remove network isolation | Elastic Defend integration |
| **Kill process** | Terminate a running process | Elastic Defend integration |
| **Suspend process** | Suspend a running process | Elastic Defend integration |
| **Get file** | Retrieve a file from endpoint | Elastic Defend integration |
| **Execute command** | Run a command on endpoint | Elastic Defend integration |
| **Osquery** | Run live queries against hosts | Osquery integration |

## Common Pitfalls

1. **ECS mapping gaps** -- Detection rules fail silently when data isn't ECS-mapped. Always verify field mappings before enabling rules.
2. **EQL sequence performance** -- Sequences with wide `maxspan` on high-volume indices are expensive. Keep spans narrow and use `by` clauses to partition.
3. **Fleet Server scaling** -- A single Fleet Server can manage ~10,000 agents. Scale horizontally for larger deployments.
4. **ML job resource consumption** -- Each ML job reserves memory. Start with prebuilt jobs and add custom jobs gradually.
5. **Detection rule conflicts** -- Enabling all 1,300+ prebuilt rules without tuning creates massive alert volume. Enable in phases, starting with high-confidence rules.
6. **Index lifecycle management (ILM)** -- Forgetting to configure ILM leads to unbounded index growth. Set hot/warm/cold/frozen/delete phases.

## Version-Specific Guidance

| Version | Reference | What's version-specific |
|---|---|---|
| Elastic 8.x | `references/versions/8.x.md` | ES\|QL introduction, detection rule improvements, Elastic Defend enhancements, Beats-to-Elastic-Agent transition |
| Elastic 9.x | `references/versions/9.x.md` | Serverless GA, ES\|QL maturity with JOIN operations, Attack Discovery AI, logsdb index mode, migration from 8.x |

## Reference Files

Load these for deep knowledge:
- `references/architecture.md` -- Elasticsearch cluster internals, Fleet architecture, data streams, ILM, ECS deep dive
- `references/best-practices.md` -- Detection engineering, EQL optimization, ES|QL patterns, ML job tuning, response action workflows