Mongodb

---
name: mongodb
description: "MongoDB technology expert covering ALL versions. Deep expertise in document modeling, aggregation pipeline, sharding, replica sets, indexing, and operational tuning. WHEN: \"MongoDB\", \"mongod\", \"mongos\", \"mongosh\", \"replica set\", \"sharding\", \"aggregation pipeline\", \"MQL\", \"BSON\", \"WiredTiger\", \"Atlas\", \"change streams\", \"mongodump\", \"mongostat\", \"mongotop\"."
license: MIT
---

# MongoDB

This skill covers MongoDB across all supported versions (6.0 through 8.0). It covers document modeling, the aggregation framework, sharding architecture, replica set internals, WiredTiger storage engine tuning, and production operations. For version-specific detail, see the matching file under `references/versions/`.

## When to Use This Skill vs. Version-Specific Guidance

**Use this agent when the question spans versions or is version-agnostic:**
- "How should I model a one-to-many relationship in MongoDB?"
- "Tune WiredTiger cache for a write-heavy workload"
- "Set up a 3-node replica set"
- "My aggregation pipeline is slow"
- "Choose a shard key for my collection"

**See the matching version reference when the question is version-specific:**
- "MongoDB 8.0 OIDC authentication" --> `references/versions/8.0.md`
- "MongoDB 7.0 compound wildcard indexes" --> `references/versions/7.0.md`
- "MongoDB 6.0 Queryable Encryption preview" --> `references/versions/6.0.md`

## How to Approach Tasks

When you receive a request:

1. **Classify** the request:
   - **Architecture/internals** -- Load `references/architecture.md`
   - **Performance diagnostics** -- Load `references/diagnostics.md`
   - **Configuration/operations** -- Load `references/best-practices.md`
   - **Version-specific feature** -- See the matching `references/versions/<v>.md` file
   - **Comparison with other databases** -- see the `overview` skill

2. **Determine version** -- Ask if unclear. Behavior differs significantly across versions (e.g., Queryable Encryption is preview in 6.0, GA in 7.0; compound wildcard indexes only in 7.0+).

3. **Analyze** -- Apply MongoDB-specific reasoning. Reference the document model, WiredTiger internals, the aggregation framework, and replication mechanics as relevant.

4. **Recommend** -- Provide actionable guidance with specific mongosh commands, configuration parameters, or schema changes.

5. **Verify** -- Suggest validation steps (explain(), db.currentOp(), db.serverStatus(), profiler output).

## Core Expertise

### Document Data Modeling

MongoDB stores data as BSON documents (Binary JSON). Schema design in MongoDB is driven by application access patterns, not by normalization rules. The fundamental decision is embedding vs. referencing.

**Embedding (denormalization):** Place related data inside the parent document.
```javascript
// Embed: order with line items (1:few, always read together)
{
  _id: ObjectId("..."),
  customer_id: ObjectId("..."),
  order_date: ISODate("2025-03-15"),
  status: "shipped",
  items: [
    { sku: "WIDGET-A", qty: 3, price: 9.99 },
    { sku: "GADGET-B", qty: 1, price: 24.50 }
  ],
  shipping_address: {
    street: "123 Main St",
    city: "Springfield",
    state: "IL",
    zip: "62701"
  }
}
```

**When to embed:**
- 1:1 relationships (address inside user)
- 1:few relationships where the "few" side is bounded (order items, tags)
- Data that is always read together (the whole document is the unit of work)
- Data that is updated together atomically (single-document ACID is free)

**Referencing (normalization):** Store a reference (ObjectId) and look up separately.
```javascript
// Reference: user document points to orders in a separate collection
// users collection
{ _id: ObjectId("u1"), name: "Alice", email: "alice@example.com" }

// orders collection
{ _id: ObjectId("o1"), user_id: ObjectId("u1"), total: 59.47, items: [...] }
```

**When to reference:**
- 1:many where "many" is unbounded (user -> thousands of orders)
- Many:many relationships (students <-> courses)
- Data that is frequently updated independently (avoid rewriting large documents)
- Data that is read independently from the parent (orders fetched without loading user)
- When documents would exceed the 16MB BSON limit

**Schema Design Anti-Patterns:**

| Anti-Pattern | Problem | Fix |
|---|---|---|
| Unbounded arrays | Document grows past 16MB; write amplification on every append | Reference pattern or bucket pattern |
| Massive documents | Entire document is rewritten on any field update | Split into multiple collections |
| Over-normalization | Too many lookups ($lookup is expensive, not a JOIN) | Embed frequently co-accessed data |
| Collection-per-date | Millions of collections; metadata overhead | Single collection with date field + TTL index |
| Storing large BLOBs | 16MB doc limit; memory pressure | Use GridFS or external storage (S3) |
| Schema-less chaos | No validation; inconsistent field names/types | Use JSON Schema validation |

**Advanced Patterns:**

- **Bucket pattern:** Group time-series data into fixed-size buckets (e.g., 1 document per sensor per hour with an array of readings). Reduces document count and index size.
- **Computed pattern:** Pre-compute aggregated values on write (running totals, counts) to avoid expensive reads.
- **Outlier pattern:** Flag documents that exceed normal array sizes and handle them differently.
- **Attribute pattern:** Store variable key-value pairs as an array of {k, v} objects for indexable polymorphic data.
- **Polymorphic pattern:** Store different entity types in the same collection with a discriminator field.
- **Extended reference pattern:** Copy frequently accessed fields from the referenced document to avoid lookups.
- **Subset pattern:** Embed only the most recent N items; archive the rest in a separate collection.

### Aggregation Pipeline

The aggregation pipeline is MongoDB's data processing framework. Documents pass through a sequence of stages, each transforming the data.

**Key stages:**

| Stage | Purpose | Example |
|---|---|---|
| `$match` | Filter documents (like WHERE) | `{ $match: { status: "active" } }` |
| `$project` | Reshape documents (include/exclude/compute fields) | `{ $project: { name: 1, total: { $multiply: ["$qty", "$price"] } } }` |
| `$group` | Aggregate by key (like GROUP BY) | `{ $group: { _id: "$category", total: { $sum: "$amount" } } }` |
| `$sort` | Order documents | `{ $sort: { created_at: -1 } }` |
| `$limit` / `$skip` | Pagination | `{ $limit: 20 }` |
| `$unwind` | Deconstruct array field into one doc per element | `{ $unwind: "$items" }` |
| `$lookup` | Left outer join to another collection | See below |
| `$addFields` / `$set` | Add or overwrite fields | `{ $addFields: { fullName: { $concat: ["$first", " ", "$last"] } } }` |
| `$replaceRoot` / `$replaceWith` | Replace document root | `{ $replaceRoot: { newRoot: "$metadata" } }` |
| `$facet` | Run multiple pipelines in parallel on same input | Multi-dimensional aggregations |
| `$bucket` / `$bucketAuto` | Group into value ranges | Histogram distributions |
| `$merge` | Write results to a collection (upsert-capable) | Materialized views, incremental pipelines |
| `$out` | Write results to a collection (replace) | One-shot output |
| `$unionWith` | Combine results from another collection | Multi-collection aggregations |
| `$graphLookup` | Recursive graph traversal | Org charts, category trees |
| `$densify` | Fill gaps in time-series or numeric sequences | Missing data points |
| `$fill` | Fill null/missing values | Forward fill, linear interpolation |
| `$setWindowFields` | Window functions (running totals, ranks, moving averages) | Analytics |
| `$search` / `$searchMeta` | Atlas Search (full-text, fuzzy, autocomplete) | Text search on Atlas |
| `$vectorSearch` | Atlas Vector Search for similarity | Semantic search, RAG |
| `$documents` | Create documents from expressions inline | Testing, seed data |
| `$changeStream` | Open a change stream as a pipeline stage | Event-driven pipelines |

**$lookup examples:**

```javascript
// Basic lookup (left outer join)
{
  $lookup: {
    from: "inventory",
    localField: "item",
    foreignField: "sku",
    as: "inventory_docs"
  }
}

// Correlated subquery lookup (3.6+)
{
  $lookup: {
    from: "warehouses",
    let: { order_item: "$item", order_qty: "$quantity" },
    pipeline: [
      { $match: { $expr: { $and: [
        { $eq: ["$stock_item", "$$order_item"] },
        { $gte: ["$instock", "$$order_qty"] }
      ]}}},
      { $project: { _id: 0, warehouse: 1, instock: 1 } }
    ],
    as: "matching_warehouses"
  }
}
```

**Pipeline optimization rules:**
1. Place `$match` as early as possible to reduce documents flowing through later stages
2. `$match` before `$project` allows index utilization
3. `$sort` + `$limit` together enable a top-N optimization (sorted limit)
4. Consecutive `$match` stages merge automatically
5. `$project` + `$match` swap is done by the optimizer when safe
6. Use `{ allowDiskUse: true }` for pipelines exceeding the 100MB per-stage memory limit
7. Avoid `$unwind` on large arrays when `$filter` or array operators suffice

### Sharding Architecture

Sharding distributes data across multiple mongod instances (shards) for horizontal scaling.

**Components:**
- **Shard:** Each shard is a replica set holding a subset of the data
- **mongos:** Query router; directs operations to the correct shard(s)
- **Config servers:** 3-member replica set storing metadata (chunk ranges, shard topology)

**Shard key strategies:**

| Strategy | How It Works | Best For | Risk |
|---|---|---|---|
| **Hashed sharding** | Hash of shard key value distributes evenly | Write-heavy workloads needing even distribution | Range queries require scatter-gather |
| **Ranged sharding** | Contiguous ranges of shard key per chunk | Range queries, time-series with careful key | Monotonic keys create hot spots |
| **Zone-based sharding** | Assign shard key ranges to specific shards (zones) | Data residency, tiered storage, geographic locality | Requires careful zone range management |

**Shard key selection criteria:**
1. **High cardinality** -- Many distinct values (avoid booleans, enums with few values)
2. **Even distribution** -- Values are spread uniformly (avoid monotonically increasing keys alone)
3. **Query isolation** -- Most queries include the shard key (targeted queries vs. scatter-gather)
4. **Write distribution** -- Writes spread across shards (hashed keys help here)
5. **Immutability** -- Shard key values should rarely change (updates to shard key fields require delete + re-insert prior to 4.2, document migration from 4.2+)

**Common shard key patterns:**
```javascript
// Hashed: good write distribution, scatter-gather for range queries
sh.shardCollection("mydb.events", { _id: "hashed" })

// Compound: targeted range queries + distribution
sh.shardCollection("mydb.logs", { tenant_id: 1, timestamp: 1 })

// Hashed prefix + range: best of both worlds
sh.shardCollection("mydb.telemetry", { device_id: "hashed", ts: 1 })  // 7.0+ only
```

### Replica Sets

A replica set is a group of mongod instances maintaining the same data set for high availability.

**Topology:**
- **Primary:** Receives all write operations. Exactly one primary at any time.
- **Secondary:** Replicates from the primary. Can serve reads with appropriate read preference. Participates in elections.
- **Arbiter:** Votes in elections but holds no data. Use only to break ties with even-numbered data-bearing members.

**Key concepts:**
- **Oplog:** Capped collection (local.oplog.rs) that records all write operations. Secondaries tail the oplog to replicate.
- **Write concern (w):** How many replica set members must acknowledge a write.
  - `w: 1` -- Primary only (fast, risk of data loss on primary failure)
  - `w: "majority"` -- Majority of voting members (durable, recommended)
  - `w: 0` -- Fire-and-forget (no acknowledgment)
- **Read preference:** Where reads are directed.
  - `primary` -- All reads from primary (default, strongest consistency)
  - `primaryPreferred` -- Primary, fallback to secondary
  - `secondary` -- Only secondaries (eventual consistency)
  - `secondaryPreferred` -- Secondary, fallback to primary
  - `nearest` -- Lowest latency member
- **Read concern:** The consistency/isolation level for read operations.
  - `"local"` -- Returns the most recent data from the queried member (may be rolled back)
  - `"majority"` -- Returns data confirmed by a majority (durable reads)
  - `"linearizable"` -- Read reflects all majority-committed writes before the read (strongest, single-doc only)
  - `"snapshot"` -- Point-in-time consistent reads (for multi-document transactions)
  - `"available"` -- Returns data with no guarantee; fastest on secondaries

**Elections and failover:**
- An election occurs when the primary becomes unreachable (heartbeat timeout: 10 seconds default)
- Members with higher `priority` are preferred as primary
- Members with `priority: 0` can never become primary (dedicated secondaries)
- `electionTimeoutMillis` defaults to 10000ms (10s); lower values mean faster failover but more false elections
- A majority of voting members (N/2 + 1) must participate in an election

### WiredTiger Storage Engine

WiredTiger is the default storage engine since MongoDB 3.2. It provides document-level concurrency, compression, and checkpoint-based durability.

**Cache management:**
- Default cache size: 50% of (RAM - 1GB), minimum 256MB
- Internal cache stores data in an uncompressed, in-memory format different from on-disk
- Eviction starts when cache is 80% full (eviction target); becomes aggressive at 95%
- Dirty data eviction triggers at 5% dirty (target), aggressive at 20% dirty

**Compression options:**
| Level | Algorithm | Ratio | CPU Cost | Use |
|---|---|---|---|---|
| Collection data | snappy (default) | ~2:1 | Low | General purpose |
| Collection data | zlib | ~3-5:1 | Medium | Storage-constrained |
| Collection data | zstd | ~3-5:1 | Low-Medium | Best balance (4.2+) |
| Index prefix | Prefix compression (default) | ~30% savings | Minimal | Always enabled |
| Journal | snappy (default) | ~2:1 | Low | General purpose |

**Checkpoints:**
- WiredTiger writes a new checkpoint every 60 seconds (default) or when the journal reaches 2GB
- Checkpoint = consistent snapshot of all data written to disk
- Between checkpoints, the journal provides durability (write-ahead log)
- Recovery replays journal entries since the last checkpoint

### Index Types

MongoDB supports a rich variety of index types:

| Index Type | Description | When to Use |
|---|---|---|
| **Single field** | Index on one field | Equality/range queries on a single field |
| **Compound** | Index on multiple fields | Queries filtering/sorting on multiple fields; follows ESR rule |
| **Multikey** | Automatic index on array fields | Querying inside arrays |
| **Text** | Full-text search index | Text search (`$text` operator) |
| **Geospatial (2dsphere)** | Spherical geometry | Location queries (`$near`, `$geoWithin`) |
| **Geospatial (2d)** | Flat geometry | Legacy 2D coordinate queries |
| **Wildcard** | Dynamic index on arbitrary fields | Querying polymorphic or schema-flexible documents |
| **Compound wildcard** (7.0+) | Wildcard combined with fixed fields | Polymorphic data with known filter prefixes |
| **TTL** | Automatic document expiration | Session data, logs, temporary records |
| **Unique** | Enforces uniqueness | Preventing duplicates on a field |
| **Partial** | Indexes only documents matching a filter | Sparse data, indexing only active records |
| **Hidden** | Index exists but is invisible to query planner | Testing index removal without dropping |
| **Hashed** | Hash of field value | Hashed shard keys, equality-only lookups |
| **Clustered** (5.3+) | Collection stored in index order | Primary key ordered storage, time-series-like access |

**ESR rule for compound indexes:** The optimal field order in a compound index is:
1. **E**quality fields first (exact match filters)
2. **S**ort fields next (supports the sort without in-memory sort)
3. **R**ange fields last (range predicates like `$gt`, `$lt`, `$in`)

```javascript
// Query: db.orders.find({ status: "shipped", total: { $gte: 100 } }).sort({ date: -1 })
// Optimal index: { status: 1, date: -1, total: 1 }
//                  ^ Equality   ^ Sort    ^ Range
```

**Index intersection:** MongoDB can combine results from multiple single-field indexes, but a well-designed compound index almost always outperforms intersection. Design compound indexes to cover your queries.

### Query Optimization with explain()

The `explain()` method reveals the query plan and execution statistics:

```javascript
// Execution statistics (most useful)
db.orders.find({ status: "active" }).sort({ date: -1 }).explain("executionStats")

// All plans considered (for plan competition analysis)
db.orders.find({ status: "active" }).explain("allPlansExecution")
```

**Key explain() fields to examine:**

| Field | What to Look For | Concern If |
|---|---|---|
| `winningPlan.stage` | IXSCAN, COLLSCAN, FETCH, SORT | COLLSCAN on large collections |
| `executionStats.totalDocsExamined` | Documents scanned | Much higher than `nReturned` |
| `executionStats.totalKeysExamined` | Index entries scanned | Much higher than `nReturned` |
| `executionStats.nReturned` | Documents returned | - |
| `executionStats.executionTimeMillis` | Total time | > 100ms for simple queries |
| `winningPlan.inputStage.indexName` | Which index was used | None (COLLSCAN) |
| `rejectedPlans` | Alternative plans the planner discarded | Useful for index design |

**Efficiency ratio:** `nReturned / totalDocsExamined` should approach 1.0. A ratio of 0.01 means scanning 100 documents for every 1 returned -- a sign of missing or suboptimal indexes.

### Security Model

**Authentication mechanisms:**
- SCRAM-SHA-256 (default since 4.0) -- Username/password
- x.509 certificates -- Mutual TLS for members and clients
- LDAP proxy authentication -- Enterprise only
- Kerberos -- Enterprise only
- OIDC (8.0+) -- OpenID Connect for cloud-native auth
- AWS IAM -- For Atlas deployments

**Authorization (RBAC):**
- MongoDB uses role-based access control with built-in and user-defined roles
- Built-in roles: `read`, `readWrite`, `dbAdmin`, `userAdmin`, `clusterAdmin`, `backup`, `restore`, `root`
- Principle of least privilege: Create custom roles with specific action+resource combinations

```javascript
// Create a custom role
db.adminCommand({
  createRole: "appReadWrite",
  privileges: [
    { resource: { db: "myapp", collection: "" }, actions: ["find", "insert", "update", "remove"] }
  ],
  roles: []
})

// Create a user with the custom role
db.adminCommand({
  createUser: "app_user",
  pwd: "securePassword",
  roles: [{ role: "appReadWrite", db: "myapp" }]
})
```

**Encryption:**
- **In-transit:** TLS/SSL for all client-server and intra-cluster communication
- **At-rest:** WiredTiger encrypted storage engine (Enterprise) or OS-level encryption (dm-crypt, LUKS, BitLocker)
- **Field-level encryption (FLE):** Client-side encryption of specific fields before sending to server (4.2+)
- **Queryable Encryption:** Encrypted fields that can be queried without decryption on the server (6.0 preview, 7.0 GA)

### Backup Strategies

| Method | Consistency | Speed | Granularity | Use Case |
|---|---|---|---|---|
| **mongodump/mongorestore** | Point-in-time with --oplog | Slow for large DBs | Database/collection level | Small-medium DBs, selective restore |
| **Filesystem snapshots** | Consistent with journaling | Fast | Full instance | Large DBs, quick recovery |
| **Atlas Continuous Backup** | Continuous with oplog | Managed | Point-in-time (1-sec granularity) | Atlas deployments |
| **Ops Manager / Cloud Manager** | Continuous | Managed | Point-in-time | Enterprise self-managed |

```bash
# mongodump with oplog for point-in-time consistency
mongodump --host rs0/primary:27017 --oplog --out /backup/$(date +%Y%m%d)

# mongorestore with oplog replay
mongorestore --host rs0/primary:27017 --oplogReplay /backup/20250315

# mongodump specific collection with compression
mongodump --db myapp --collection orders --gzip --archive=/backup/orders.gz
```

### Connection Management

MongoDB drivers maintain connection pools to avoid per-operation TCP handshake overhead:

- **Default pool size:** Varies by driver (typically 100 connections per MongoClient)
- **maxPoolSize:** Maximum connections per server in the connection string
- **minPoolSize:** Minimum connections to keep open (warm pool)
- **maxIdleTimeMS:** How long a connection can sit idle before being closed
- **waitQueueTimeoutMS:** How long an operation waits for a connection from the pool

```
// Connection string with pool settings
mongodb://user:pass@host1:27017,host2:27017,host3:27017/mydb?replicaSet=rs0&maxPoolSize=200&minPoolSize=10&maxIdleTimeMS=30000&w=majority&readPreference=secondaryPreferred
```

**Connection management pitfalls:**
- Creating a new MongoClient per request (connection churn, exhaust limits)
- Setting maxPoolSize too high (each connection uses ~1MB on the server; 5000 connections = 5GB overhead)
- Not setting `maxIdleTimeMS` behind load balancers that close idle connections

### Monitoring: currentOp, serverStatus, Profiler

See `references/monitoring.md#monitoring-currentop-serverstatus-profiler` — db.currentOp(), db.serverStatus() metrics, and database profiler usage for diagnosing in-flight and slow operations.

## Common Pitfalls

1. **Using MongoDB as a relational database** -- Excessive use of $lookup to simulate JOINs. If most queries require multi-collection joins, reconsider the data model or use an RDBMS.

2. **Unbounded array growth** -- Pushing to arrays without limit. Arrays that grow indefinitely cause document migrations (moves to larger storage slots), write amplification, and can hit the 16MB limit.

3. **Missing indexes on common queries** -- A COLLSCAN on a million-document collection is orders of magnitude slower than an IXSCAN. Run `explain()` on every query used in production.

4. **Wrong shard key** -- A monotonically increasing shard key (like ObjectId or timestamp) creates a "hot shard" where all new inserts go to the same shard. Use hashed sharding or a compound key with good distribution.

5. **Ignoring write concern** -- Default `w: 1` acknowledges only the primary. If the primary crashes before replicating to a secondary, the write is lost. Use `w: "majority"` for durable writes.

6. **Not sizing the oplog** -- A small oplog on a write-heavy system means secondaries that fall behind cannot catch up (they need to perform an initial sync). Size the oplog to hold at least 24-72 hours of writes.

7. **WiredTiger cache thrashing** -- If the working set exceeds the WiredTiger cache, eviction becomes aggressive and latency spikes. Monitor cache fill percentage and dirty page ratio.

8. **Running without authentication** -- MongoDB ships with authentication disabled by default. Always enable `--auth` or `security.authorization: enabled` in production.

9. **Large document updates** -- Updating a single field in a 10MB document rewrites the entire document. Design schemas so frequently updated fields are in smaller documents.

10. **Using the default ObjectId as shard key** -- ObjectId is monotonically increasing (timestamp prefix), creating a hot-shard problem. Use `{ _id: "hashed" }` or a better shard key.

## Version-specific guidance

| Version | Status | Key Feature | Reference |
|---|---|---|---|
| **MongoDB 8.0** | Current (Aug 2024) | Queryable Encryption enhancements, OIDC auth, improved query execution | `references/versions/8.0.md` |
| **MongoDB 7.0** | Supported | Compound wildcard indexes, Queryable Encryption GA, ShardingReady | `references/versions/7.0.md` |
| **MongoDB 6.0** | End of Life (projected) | Queryable Encryption preview, cluster-to-cluster sync, time-series improvements | `references/versions/6.0.md` |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- WiredTiger internals, replication protocol, sharding internals, query planner, locking model, transaction implementation. Read for "how does MongoDB work internally" questions.
- `references/diagnostics.md` -- serverStatus, currentOp, profiler, replication monitoring, sharding diagnostics, index analysis, memory analysis. Read when troubleshooting performance or operational issues.
- `references/best-practices.md` -- Production deployment checklist, replica set sizing, shard key selection, security hardening, backup strategies, monitoring setup, capacity planning. Read for configuration and operational guidance.

## Diagnostic Scripts

Ready-made mongosh scripts in `scripts/` (MongoDB 6.0+), numbered by investigation order. All read-only.

- `scripts/01-current-ops.js` -- Long-running operations with COLLSCAN detection
- `scripts/02-profiler-slow-queries.js` -- Slowest ops from system.profile (docsExamined vs nreturned)
- `scripts/03-collection-stats.js` -- Collection/index sizes, unbounded-growth detection
- `scripts/04-index-usage.js` -- Dead indexes via $indexStats (with unique/TTL/restart caveats)
- `scripts/05-replset-status.js` -- Member states, replication lag, oplog window