Logscale

---
name: logscale
description: "Expert agent for CrowdStrike Falcon LogScale (formerly Humio). Provides deep expertise in LQL query development, index-free streaming architecture, real-time search, Falcon EDR integration, bloom filters, data compression, community edition, and high-volume log analytics. WHEN: \"LogScale\", \"Falcon LogScale\", \"Humio\", \"LQL\", \"LogScale query\", \"streaming SIEM\", \"index-free\", \"CrowdStrike SIEM\", \"LogScale community\"."
license: MIT
---

# CrowdStrike Falcon LogScale

This skill covers CrowdStrike Falcon LogScale (formerly Humio), a streaming SIEM platform designed for high-volume, low-latency log analytics. It has deep knowledge of:

- LogScale Query Language (LQL) for real-time and historical search
- Index-free architecture (compressed raw data + bloom filters)
- Real-time streaming search with sub-second latency
- CrowdStrike Falcon integration (native EDR + SIEM convergence)
- Data compression and storage efficiency
- Parser development (custom log ingestion)
- Dashboards, alerts, and scheduled searches
- LogScale Collector (data shipping agent)
- Community Edition (free tier)
- Self-hosted and cloud deployment models

**Architecture note:** LogScale uses an index-free design that stores compressed raw data and uses bloom filters for fast search. This eliminates the storage overhead of traditional inverted indexes, enabling very high ingestion rates at lower storage costs.

## How to Approach Tasks

1. **Classify** the request:
   - **Investigation** -- LQL query development, live/historical search
   - **Detection engineering** -- Alert definitions, scheduled searches, Falcon integration
   - **Data onboarding** -- Parser development, ingest API, LogScale Collector
   - **Architecture** -- Cluster sizing, storage planning, deployment model
   - **CrowdStrike integration** -- Falcon sensor data, Falcon Fusion workflows

2. **Gather context** -- Deployment model (cloud vs self-hosted), CrowdStrike Falcon usage, daily ingestion volume, retention requirements

3. **Analyze** -- Apply LogScale-specific reasoning, especially around streaming search patterns

4. **Recommend** actionable guidance with LQL examples

## Core Expertise

### LQL (LogScale Query Language)

LQL is a piped query language optimized for streaming and historical search:

```lql
// Basic: Find failed SSH logins
#type=syslog
| "Failed password" OR "authentication failure"
| regex("Failed password for (?<user>\S+) from (?<src_ip>\S+)")
| groupBy([src_ip, user], function=count())
| sort(field=_count, order=desc)
| head(20)
```

**LQL key functions:**

| Category | Functions | Example |
|---|---|---|
| **Filtering** | String matching, regex, field comparison | `src_ip = "10.0.0.*"` |
| **Aggregation** | `count()`, `sum()`, `avg()`, `min()`, `max()`, `percentile()` | `groupBy(host, function=count())` |
| **Grouping** | `groupBy()`, `bucket()`, `timeChart()` | `timeChart(span=5m, function=count())` |
| **Transformation** | `regex()`, `replace()`, `lower()`, `upper()`, `format()` | `regex("src=(?<src>\S+)")` |
| **Statistical** | `stdDev()`, `variance()`, `top()`, `rare()` | `top(field=process_name, limit=10)` |
| **Join** | `join()`, `selfJoin()` | `join({#type=threat_intel}, field=ip)` |
| **Sorting** | `sort()`, `head()`, `tail()`, `reverse()` | `sort(field=bytes, order=desc)` |
| **Time** | `bucket()`, `timeChart()`, `now()` | `bucket(field=@timestamp, span=1h)` |

**Advanced LQL patterns:**

```lql
// Detect potential data exfiltration: unusual outbound bytes
#type=firewall action=allow direction=outbound
| bucket(field=@timestamp, span=1h, function=[sum(bytes_out) as total_bytes])
| total_bytes > 1000000000
| sort(field=total_bytes, order=desc)
```

```lql
// Rare process detection: processes seen on fewer than 3 hosts
#type=endpoint event_type=process_start
| groupBy([process_name], function=[count(as=exec_count), collectDistinct(host, as=unique_hosts)])
| length(field=unique_hosts, as=host_count)
| host_count < 3
| sort(field=exec_count, order=asc)
```

```lql
// Join with threat intelligence
#type=firewall
| join({#type=threat_intel | rename(field=indicator, as=ip)}, field=dest_ip, key=ip, include=[threat_type, confidence])
| confidence > 80
| groupBy([dest_ip, threat_type], function=count())
```

### Index-Free Architecture

LogScale's defining architectural choice:

```
Incoming events
    |
    v
Parsing (extract fields, assign tags)
    |
    v
Compression (zstd, ~10:1 ratio)
    |
    v
Segment files (compressed raw data)
    + Bloom filters (probabilistic field existence)
    + Tag metadata (fast pre-filtering)
    |
    v
Object storage or local disk
```

**How search works without an inverted index:**

1. **Tag filtering** -- Narrow to relevant segments using tags (sourcetype, host, etc.)
2. **Bloom filter check** -- Probabilistic test: "Does this segment POSSIBLY contain this search term?"
3. **Segment scan** -- Decompress and scan matching segments
4. **Field extraction** -- Extract fields from matching events (schema-on-read)

**Trade-offs vs. inverted index:**

| Aspect | Index-Free (LogScale) | Inverted Index (Splunk, Elastic) |
|---|---|---|
| Storage efficiency | Higher (no index overhead) | Lower (index ~30-100% of raw size) |
| Ingest throughput | Higher (no index build) | Lower (index building is CPU-intensive) |
| Point query speed | Slightly slower (segment scan) | Faster (direct index lookup) |
| Full-text search | Fast (bloom filter + scan) | Fastest (direct inverted index) |
| Aggregation | Good (scan + compute) | Depends on doc values / summaries |
| Streaming search | Native (scan new segments in real-time) | Requires special handling |

### Real-Time Streaming Search

LogScale excels at real-time queries:

- **Live queries** -- Attach to the data stream and see results in real-time
- **Sub-second latency** -- New events appear in search within milliseconds
- **Streaming dashboards** -- Dashboards update in real-time without polling
- **No indexing delay** -- Data is searchable immediately after ingestion (no index build lag)

### CrowdStrike Falcon Integration

Native integration between LogScale and Falcon EDR:

- **Falcon sensor telemetry** -- Direct ingestion of endpoint events from Falcon sensors
- **Falcon Fusion** -- Workflow automation triggered by LogScale alerts
- **Falcon Next-Gen SIEM** -- LogScale positioned as the SIEM for CrowdStrike customers
- **Unified investigation** -- Correlate Falcon EDR alerts with third-party log data in LogScale
- **Identity Protection** -- Integrate Falcon Identity Protection data for identity-based detection

### Parsers

Parsers define how LogScale extracts structure from raw logs:

```yaml
# Custom parser for JSON application logs
name: myapp-json
tests:
  - input: '{"timestamp":"2026-04-08T12:00:00Z","level":"ERROR","message":"Connection failed","src":"10.0.0.1"}'
    output:
      "@timestamp": "2026-04-08T12:00:00Z"
      level: "ERROR"
      message: "Connection failed"
      src: "10.0.0.1"
parseJson:
  field: "@rawstring"
  handleUnparsed: discard
setTimestamp:
  field: timestamp
  format: "yyyy-MM-dd'T'HH:mm:ss'Z'"
tag:
  - level
```

**Parser types:**
- **Built-in parsers** -- Pre-built for common sources (syslog, JSON, CEF, AWS, GCP, O365)
- **Custom parsers** -- User-defined for proprietary log formats
- **Regex-based** -- Extract fields using named capture groups
- **JSON/CSV** -- Structured format parsing
- **Grok patterns** -- LogScale supports Grok-style named patterns

### Data Ingestion

| Method | Use Case | Throughput |
|---|---|---|
| **LogScale Collector** | Endpoint log shipping (official agent) | High |
| **Elastic Beats** | If migrating from Elastic stack | High |
| **Falcon Sensor** | CrowdStrike endpoint telemetry | Native |
| **Ingest API** | Application/container logs (HTTP) | Very high |
| **Syslog** | Network devices, legacy systems | Medium |
| **Kafka** | Event streaming pipeline | Very high |
| **Cloud connectors** | AWS S3, Azure Event Hub, GCP Pub/Sub | High |

### Alerts and Scheduled Searches

```lql
// Alert: More than 100 failed logins in 5 minutes
#type=auth action=failure
| timeChart(span=5m, function=count())
| _count > 100
```

**Alert actions:**
- Email notification
- Webhook (HTTP POST to any endpoint)
- Slack / Teams integration
- PagerDuty / Opsgenie
- Falcon Fusion workflow trigger
- Custom script execution

### Community Edition

Free tier for individual use and small teams:

- **Daily ingestion:** 16 GB/day
- **Retention:** 7 days
- **Users:** 1 user
- **Repositories:** 1 repository
- **Features:** Full query language, dashboards, alerts
- **Ideal for:** Home lab, learning, small projects, log exploration

## Deployment Models

| Model | Management | Use Case |
|---|---|---|
| **Falcon LogScale (cloud)** | CrowdStrike-managed | Production, CrowdStrike customers |
| **Self-hosted (Kubernetes)** | Customer-managed | On-premises, air-gapped, data sovereignty |
| **Self-hosted (single node)** | Customer-managed | Small deployments, testing |
| **Community Edition** | Self-managed | Free, limited to 16 GB/day |

### Sizing Guidelines

| Daily Volume | Nodes | CPU (per node) | RAM (per node) | Storage |
|---|---|---|---|---|
| < 100 GB | 3 | 16 cores | 64 GB | NVMe SSD + S3/GCS |
| 100-500 GB | 6 | 32 cores | 128 GB | NVMe SSD + S3/GCS |
| 500 GB - 1 TB | 12 | 32 cores | 128 GB | NVMe SSD + S3/GCS |
| 1+ TB | 20+ | 64 cores | 256 GB | NVMe SSD + S3/GCS |

**Storage architecture:**
- Local NVMe SSD for real-time ingest buffer (1-7 days)
- Object storage (S3, GCS, Azure Blob) for long-term retention
- Total storage ≈ daily volume x compression ratio (0.1-0.15) x retention days

## Common Pitfalls

1. **Bloom filter false positives** -- Bloom filters are probabilistic. A small percentage of segments may be scanned unnecessarily. Not a problem in practice but explains why rare searches may scan more data than expected.
2. **Schema-on-read performance** -- Complex regex field extraction at search time is slower than pre-indexed fields. Use tags and parser-extracted fields for frequently-searched fields.
3. **Limited SOAR** -- LogScale's automation capabilities are basic compared to dedicated SOAR platforms. Use Falcon Fusion or an external SOAR for complex playbooks.
4. **Community Edition limitations** -- 16 GB/day and 7-day retention are insufficient for production security monitoring. Plan for licensed deployment.
5. **Parser maintenance** -- Custom parsers need updating when log source formats change. Version-control parsers and test with representative sample data.
6. **Falcon dependency for full value** -- LogScale works standalone, but the deepest value comes with CrowdStrike Falcon integration. Without Falcon, it's primarily a log analytics platform.