Containerd

---
name: containerd
description: "Expert coverage of containerd across all supported versions: CRI implementation, snapshotter architecture, content store, namespaces, NRI plugin framework, Sandbox API, nerdctl CLI, and Kubernetes integration. Use for \"containerd\", \"nerdctl\", \"CRI\", \"snapshotter\", \"NRI\", \"Sandbox API\", \"containerd config\", \"ctr\", \"crictl\", \"containerd 2\"."
license: MIT
---

# containerd

This skill covers containerd across all supported versions (1.7 through 2.2.x), including:

- CRI (Container Runtime Interface) implementation for Kubernetes
- Snapshotter architecture (overlayfs, stargz, nydus, btrfs, zfs)
- Content-addressable store and metadata management
- containerd namespaces (resource isolation between clients)
- NRI (Node Resource Interface) plugin framework
- Sandbox API for Pod lifecycle management
- Transfer Service for content movement
- nerdctl (Docker-compatible CLI for containerd)
- containerd 2.x breaking changes and migration

## How to Approach Tasks

1. **Classify** the request:
   - **CRI/Kubernetes** -- Load `references/architecture.md` for CRI plugin, runtime configuration, snapshotter setup
   - **Standalone usage** -- Guide using nerdctl or ctr CLI
   - **Plugin development** -- NRI plugin framework, Sandbox API
   - **Migration** -- containerd 1.7 to 2.x upgrade paths
   - **Performance** -- Snapshotter selection, lazy pulling, content store optimization

2. **Identify version** -- Determine containerd version. Key boundaries: v1.7 (NRI alpha, Transfer Service), v2.0 (major breaking changes, config v3), v2.1 (Schema 1 removal), v2.2 (mount manager, extended NRI). If unclear, ask.

3. **Load context** -- Read `references/architecture.md` for deep architectural knowledge.

4. **Analyze** -- Apply containerd-specific reasoning. containerd is a low-level runtime, not a user-facing tool like Docker.

5. **Recommend** -- Provide actionable guidance with config.toml snippets, CLI examples, and Kubernetes manifests.

## Core Architecture

```
Clients (dockerd, kubelet, nerdctl)
         | gRPC (CRI or containerd API)
  containerd daemon
    |-- Content Store (OCI layers, compressed blobs, content-addressable)
    |-- Metadata Store (BoltDB: images, containers, leases, snapshots)
    |-- Snapshotter (overlayfs, stargz, nydus, btrfs, native)
    |-- Runtime Service
    |     |-- containerd-shim-runc-v2 --> runc
    |     |-- containerd-shim-kata-v2 --> Kata Containers
    |     |-- containerd-shim-runsc-v1 --> gVisor
    |-- Transfer Service (image import/export)
    |-- NRI (Node Resource Interface plugins)
    +-- CRI Plugin (Kubernetes kubelet integration)
```

### Namespaces

containerd namespaces isolate resources between clients sharing the same containerd instance. These are **containerd-level namespaces**, not Linux kernel namespaces.

| Namespace | Client | Purpose |
|---|---|---|
| `moby` | Docker Engine | Docker's containers and images |
| `k8s.io` | Kubernetes kubelet | Kubernetes pods and images |
| `default` | ctr CLI | Default namespace for ctr commands |

```bash
ctr --namespace moby containers ls      # Docker's containers
ctr --namespace k8s.io containers ls    # Kubernetes containers
ctr --namespace k8s.io images ls        # Kubernetes images
```

This allows Docker and Kubernetes to share a single containerd instance without interference.

### Content Store

The content store is a content-addressable store of immutable blobs (image layers, manifests, configs):

```bash
# List content
ctr content ls

# Fetch image to content store (no unpack)
ctr images fetch docker.io/library/nginx:latest

# Content stored at:
# /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
```

Content is immutable and shared across all namespaces. Garbage collection removes unreferenced content via leases.

## CRI Implementation

containerd's built-in CRI plugin is the standard Container Runtime Interface for Kubernetes. The kubelet communicates with containerd over a gRPC socket.

### Configuration (config.toml v3)

```toml
# /etc/containerd/config.toml (containerd 2.x)
version = 3

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true    # required for cgroup v2 and systemd-based distros

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
      runtime_type = "io.containerd.runsc.v1"

  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"
```

### Registry Mirror Configuration

```
/etc/containerd/certs.d/
  docker.io/
    hosts.toml:
      server = "https://registry-1.docker.io"
      [host."https://mirror.internal.example.com"]
        capabilities = ["pull", "resolve"]
  registry.internal:5000/
    hosts.toml:
      server = "http://registry.internal:5000"
      [host."http://registry.internal:5000"]
        capabilities = ["pull", "push", "resolve"]
        skip_verify = true
```

### crictl (CRI Debugging CLI)

```bash
# crictl talks directly to the CRI endpoint
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock

crictl pods                        # list pods
crictl ps                          # list containers
crictl images                      # list images
crictl logs <container-id>         # container logs
crictl inspect <container-id>      # container details
crictl stats                       # resource usage
crictl pull docker.io/library/nginx:latest
```

## Snapshotter Architecture

Snapshotters manage the layered filesystem for containers. They unpack image layers and present a union view to the container.

| Snapshotter | Mechanism | Use Case | Lazy Pull |
|---|---|---|---|
| overlayfs | Kernel OverlayFS | Default, general purpose | No |
| native | Bind mounts (copy) | Older kernels without OverlayFS | No |
| btrfs | Btrfs subvolumes | Btrfs filesystems | No |
| zfs | ZFS datasets | ZFS filesystems | No |
| stargz | eStargz format, lazy pulling | Large images, reduce startup time | Yes |
| nydus | RAFS v6, lazy pulling | Large images, better compression | Yes |

### Lazy Pulling (stargz / nydus)

Traditional image pull downloads and unpacks all layers before starting a container. Lazy pulling mounts the remote image and downloads layer content on-demand:

```bash
# Use stargz snapshotter
ctr run --snapshotter stargz docker.io/library/nginx:latest mynginx /bin/sh

# Convert image to eStargz format
ctr-remote image optimize docker.io/library/nginx:latest \
  registry.example.com/nginx:esgz
```

**Benefits**: Container starts in seconds even with multi-GB images. Only accessed files are downloaded.
**Trade-off**: First file access may have latency; requires registry to stay available.

## NRI (Node Resource Interface)

NRI is a framework for domain-specific plugins that react to container lifecycle events. Enabled by default in containerd 2.0.

### Plugin Capabilities

- **CPU affinity/pinning**: Pin containers to specific CPU cores based on annotations
- **NUMA topology**: Allocate memory from the correct NUMA node
- **GPU/accelerator allocation**: Assign GPU devices to containers
- **Device injection**: Add custom devices to container specs
- **OCI hook injection**: Modify container OCI spec at creation time
- **Resource balancing**: Adjust resource limits based on node utilization

### How NRI Works

```
Container lifecycle event (CreateContainer, StartContainer, etc.)
  |
  containerd --> NRI broker --> NRI plugin (Unix socket)
  |                              |
  |                              +-- returns ContainerAdjustment
  |                                  (modified OCI spec, resources, devices)
  +-- applies adjustments to container
```

containerd 2.2 passes extended container status to NRI plugins, enabling richer lifecycle decisions.

### NRI Plugin Configuration

```toml
# /etc/containerd/config.toml
[plugins."io.containerd.nri.v1.nri"]
  disable = false
  socket_path = "/var/run/nri/nri.sock"
  plugin_path = "/opt/nri/plugins"
```

## Sandbox API

The Sandbox API provides explicit lifecycle management for Pod sandboxes, separating sandbox creation from container creation:

```
Traditional: CreatePodSandbox creates both the sandbox (pause container) and network namespace
Sandbox API: Sandbox lifecycle is explicit -- Create, Start, Update, Stop, Delete
```

This decoupling is critical for VM-based runtimes (Kata Containers, Firecracker) where the sandbox IS the VM:
- Sandbox creation starts the VM
- Container creation runs processes inside the VM
- Sandbox can be updated (resource adjustment) independently

containerd 2.x added an Update endpoint for modifying sandbox spec, runtime, extensions, and labels on running sandboxes.

## Transfer Service

A unified API for moving content between stores, registries, and local archives:

```bash
# Transfer image between registries
ctr transfer docker.io/library/nginx:latest localhost:5000/nginx:latest

# Transfer handles:
# - Registry authentication
# - Parallel layer downloads
# - Content verification (digest matching)
# - Decompression
# - Snapshotter integration
```

## nerdctl (Docker-compatible CLI)

nerdctl is the recommended user-facing CLI for containerd, providing Docker-compatible commands:

```bash
# Container lifecycle
nerdctl run -d --name nginx nginx:latest
nerdctl ps
nerdctl exec -it nginx bash
nerdctl logs nginx
nerdctl stop nginx && nerdctl rm nginx

# Image management
nerdctl build -t myapp:latest .
nerdctl push registry.example.com/myapp:latest
nerdctl pull registry.example.com/myapp:latest

# Compose
nerdctl compose up -d
nerdctl compose down

# Rootless
nerdctl run --rootless -d nginx:latest

# Snapshotter selection
nerdctl run --snapshotter stargz -d nginx:latest

# Image encryption (OCI)
nerdctl image encrypt --recipient jwe:public.pem myapp:latest myapp:encrypted

# Namespace selection
nerdctl --namespace k8s.io ps    # view Kubernetes containers
```

nerdctl is the recommended CLI for:
- Development and CI/CD with containerd
- Debugging Kubernetes node containers
- Edge computing (K3s, which bundles containerd)

## containerd 2.x Breaking Changes

| Area | Change | Migration |
|---|---|---|
| Config | `version = 3` required | Update `/etc/containerd/config.toml` header |
| Shims | shim v1 removed | Use `containerd-shim-runc-v2` |
| Storage | AUFS snapshotter removed | Switch to overlayfs |
| Images | Schema 1 disabled (2.0), removed (2.1) | Re-push as OCI/Docker schema 2 |
| API | Legacy v1 API surface removed | Update gRPC clients |
| cgroup | cgroup v2 default, v1 being phased | Migrate to cgroup v2 |
| NRI | Enabled by default | Audit NRI plugins |

## Common Pitfalls

1. **Config version mismatch**: containerd 2.x requires `version = 3` in config.toml. Using old config silently falls back to defaults.
2. **SystemdCgroup not set**: On cgroup v2 systems with systemd, failing to set `SystemdCgroup = true` causes container creation failures.
3. **Namespace confusion**: containerd namespaces are NOT Linux namespaces. Docker images are in "moby", K8s in "k8s.io". Querying the wrong namespace shows empty results.
4. **ctr vs nerdctl vs crictl**: `ctr` is the low-level debug CLI (not user-friendly). `nerdctl` is Docker-compatible. `crictl` is for Kubernetes CRI debugging. Use the right tool.
5. **Registry config path**: containerd 2.x uses `/etc/containerd/certs.d/` for registry mirrors, not inline config in config.toml.
6. **Schema 1 images**: Extremely old images will fail to pull on containerd 2.1+. Re-push in modern format.

## Reference Files

Read these for implementation depth:

- `references/architecture.md` -- CRI internals, snapshotter details, NRI, namespaces, nerdctl, containerd 2.x changes. Read for architecture and Kubernetes integration questions.