Rhel Podman

---
name: rhel-podman
description: "Podman and the container ecosystem on Red Hat Enterprise Linux across RHEL 8, 9, 10: daemonless container architecture, rootless containers, pods, networking (CNI/Netavark), storage, Quadlet systemd integration, Buildah, Skopeo, auto-update, container security, and troubleshooting. Use when: \"Podman\", \"podman\", \"container\", \"Buildah\", \"buildah\", \"Skopeo\", \"skopeo\", \"rootless container\", \"quadlet\", \"container image\", \"OCI\", \"Containerfile\", \"pod\". Do NOT use for Podman on other distros, Docker, or general container orchestration/Kubernetes — use the containers plugin's `podman` or `kubernetes` skill; this covers Podman specifically on RHEL."
license: MIT
---

# Podman Container Ecosystem (RHEL)

This skill covers Podman and the container ecosystem on Red Hat Enterprise Linux across RHEL 8, 9, and 10. It provides deep knowledge of:

- Daemonless container architecture (fork/exec model, OCI runtimes, conmon)
- Rootless containers (user namespaces, subuid/subgid, rootless storage, rootless networking)
- Pod model (shared namespaces, infrastructure containers, Kubernetes YAML compatibility)
- Container networking (CNI on RHEL 8, Netavark/Aardvark-DNS on RHEL 9+, bridge, MACVLAN, host)
- Storage architecture (overlay driver, containers/storage library, image layer management)
- Quadlet systemd integration (.container, .volume, .network, .pod, .kube, .image units)
- Buildah for image building (Containerfile/Dockerfile, interactive builds, multi-stage)
- Skopeo for image inspection, copying, and registry operations
- Auto-update with label-based policies and systemd timers
- Container security (SELinux labels, capabilities, seccomp, read-only rootfs, user namespaces, secrets)
- Resource management (CPU, memory, I/O limits via cgroup v2)
- Registry configuration (registries.conf, authentication, signature verification, mirrors)

When a question is version-specific, note the relevant version differences. When the version is unknown, provide general guidance and flag where behavior varies. For general RHEL administration, see the `rhel` skill.

## How to Approach Tasks

Route by request type: **troubleshooting** → `references/diagnostics.md`; **architecture** → `references/architecture.md`; **best practices/config** → `references/best-practices.md`; **health check** → the diagnostic scripts; **image management** → `references/best-practices.md` (Buildah/Skopeo).

**Identify the RHEL and Podman versions first** — feature availability shifts by version (Quadlet requires Podman 4.4+/RHEL 9.2+, Netavark requires RHEL 9+). **Identify rootless vs rootful** — storage paths, networking backends, and resource-limit capabilities differ. Reason from the daemonless model, rootless constraints, SELinux interaction, and systemd integration — not generic Docker advice. Validate with `podman inspect`, `podman logs`, `systemctl status`, and `podman system info`.

## Core Expertise

### Daemonless Architecture

Podman differs fundamentally from Docker by eliminating the central daemon process. Each `podman` CLI invocation is a standalone process that forks an OCI runtime directly to start containers.

Key components:
- **OCI Runtime**: `runc` (RHEL 8 default, Go) or `crun` (RHEL 9+ default, C, faster startup, lower memory)
- **conmon**: Lightweight C process per container managing stdin/stdout/stderr streams, exit codes, and container lifecycle. Survives if the Podman CLI exits.
- **No single point of failure**: No root daemon, direct systemd integration, container processes visible in normal process tree

```bash
# Check active runtime
podman info --format '{{.Host.OCIRuntime.Name}}'
```

### Rootless Containers

Rootless containers run entirely within a normal user's UID using Linux user namespaces.

**User namespace mapping** via `/etc/subuid` and `/etc/subgid`:
```
username:start_uid:count
chris:100000:65536
```

Container UID 0 maps to host UID 100000. Setup:
```bash
usermod --add-subuids 100000-165535 --add-subgids 100000-165535 chris
podman system migrate
```

**Rootless storage**: `~/.local/share/containers/storage` (graph root), `/run/user/<UID>/containers` (run root).

**Rootless networking**:
- RHEL 8: `slirp4netns` (userspace TCP/IP stack)
- RHEL 9.2+: `pasta` (uses `passt` backend, better performance)

**Rootless limitations**:
- Cannot bind ports < 1024 without `net.ipv4.ip_unprivileged_port_start` sysctl
- cgroup v1 (RHEL 8): no resource limits in rootless mode
- cgroup v2 (RHEL 9+): resource limits work with delegation enabled
- No MACVLAN networking

### Networking

| Feature | RHEL 8 | RHEL 9+ |
|---|---|---|
| Backend | CNI | Netavark |
| DNS | dnsmasq plugin | Aardvark-DNS |
| Config dir | `/etc/cni/net.d/` | Podman-managed |

Network types:
```bash
# Bridge (default) -- isolated L2 with NAT
podman network create mynet

# MACVLAN -- container appears on host network
podman network create --driver macvlan --opt parent=eth0 macvlan-net

# Host networking
podman run --network host ...
```

Port mapping:
```bash
podman run -p 8080:80 nginx              # Map host:container
podman run -p 127.0.0.1:8080:80 nginx    # Bind to specific IP
```

Containers on the same Netavark network resolve each other by name automatically via Aardvark-DNS.

### Storage Architecture

The `containers/storage` library manages image layers and container filesystems. Shared by Podman, Buildah, and CRI-O.

| Driver | Use Case | Notes |
|---|---|---|
| `overlay` | Default, production | Recommended; fuse-overlayfs for rootless on RHEL 8 |
| `vfs` | Compatibility fallback | Full copy per layer, slow |
| `btrfs` | Btrfs filesystems | Btrfs mount required |

```bash
podman info --format '{{.Store.GraphDriverName}}'   # Check driver
podman system df                                     # Disk usage
podman system prune -a                               # Clean unused resources
```

Key paths:
- Root graph: `/var/lib/containers/storage`
- Rootless graph: `~/.local/share/containers/storage`

### Pod Model

Pods group containers sharing network, IPC, and optionally PID namespaces -- mirroring the Kubernetes pod concept. Every pod contains an infrastructure ("infra") container that holds shared namespaces.

```bash
podman pod create --name mypod -p 8080:80
podman run -d --pod mypod nginx
podman run -d --pod mypod myapp
```

Kubernetes YAML compatibility:
```bash
podman generate kube mypod > mypod.yaml     # Export
podman play kube mypod.yaml                  # Deploy
podman play kube --down mypod.yaml           # Tear down
```

### Quadlet (systemd Integration, RHEL 9.2+)

Quadlet generates systemd units from declarative files placed in:
- System: `/etc/containers/systemd/`
- User: `~/.config/containers/systemd/`

Example `.container` file:
```ini
[Unit]
Description=Nginx Web Server
After=network-online.target

[Container]
Image=docker.io/library/nginx:latest
PublishPort=8080:80
Volume=/var/www/html:/usr/share/nginx/html:ro,Z
Label=io.containers.autoupdate=registry

[Service]
Restart=always

[Install]
WantedBy=multi-user.target
```

After adding files: `systemctl daemon-reload && systemctl start nginx`

Quadlet replaces the deprecated `podman generate systemd` (removed in Podman 5.x).

### Buildah

Buildah builds OCI-compliant container images without requiring a daemon:

```bash
# Build from Containerfile
buildah bud -t myapp:latest -f Containerfile .

# Interactive build (no Dockerfile)
container=$(buildah from ubi9)
buildah run $container -- dnf install -y python3
buildah config --cmd "python3 /app/server.py" $container
buildah commit $container myapp:latest
buildah rm $container
```

Multi-stage builds with `--layers` for cached intermediate layers.

### Skopeo

Skopeo inspects and copies container images without requiring a local daemon:

```bash
skopeo inspect docker://registry.access.redhat.com/ubi9/ubi:latest
skopeo copy docker://docker.io/nginx:latest docker://myregistry.internal/nginx:latest
skopeo list-tags docker://docker.io/library/nginx
skopeo sync --src docker --dest dir docker.io/library/nginx /tmp/mirrors/
```

### Auto-Update

Quadlet + auto-update enables automated container image updates:

```ini
# In .container file
Label=io.containers.autoupdate=registry
```

```bash
podman auto-update                 # Update all labeled containers
podman auto-update --dry-run       # Preview updates
systemctl enable --now podman-auto-update.timer  # Enable daily timer
```

### Container Security

Best practices for production containers:
```bash
podman run \
  --read-only \
  --cap-drop=ALL \
  --security-opt no-new-privileges \
  --user 1001:1001 \
  -v /data:/data:ro,Z \
  myapp:latest
```

Key security features:
- **SELinux labels**: `:Z` (private) / `:z` (shared) on volume mounts
- **Capability dropping**: `--cap-drop=ALL --cap-add=<needed>`
- **Read-only rootfs**: `--read-only` with `--tmpfs` for writable paths
- **No-new-privileges**: `--security-opt no-new-privileges`
- **Seccomp profiles**: `--security-opt seccomp=/path/to/profile.json`
- **User namespace isolation**: `--userns=auto`
- **Secrets**: `podman secret create` + `--secret` flag

### Registry Configuration

System config: `/etc/containers/registries.conf` (TOML format)

```toml
unqualified-search-registries = ["registry.access.redhat.com", "registry.redhat.io", "docker.io"]

[[registry]]
location = "docker.io"
  [[registry.mirror]]
  location = "mirror.internal.example.com"
```

Authentication: `podman login registry.redhat.io` (credentials stored in `$XDG_RUNTIME_DIR/containers/auth.json`).

Short-name aliases (RHEL 9+): `/etc/containers/registries.conf.d/shortnames.conf`

## Version-Specific Changes

| Feature | RHEL 8 | RHEL 9 | RHEL 10 |
|---|---|---|---|
| Podman version | 4.x | 4.x / 4.4+ | 5.x |
| OCI runtime | runc (default) | crun (default) | crun |
| Networking | CNI + dnsmasq | Netavark + Aardvark-DNS | Netavark |
| Rootless networking | slirp4netns | pasta (9.2+) | pasta |
| cgroup version | v1 (default) | v2 (default) | v2 |
| Rootless resource limits | Not available (cgroup v1) | Available (cgroup v2) | Available |
| Quadlet | Not available | Introduced (4.4, RHEL 9.2) | Primary integration |
| `podman generate systemd` | Primary method | Deprecated | Removed |
| Rootless overlay | fuse-overlayfs required | Native overlay in most cases | Native overlay |
| Compose | podman-compose (v1) | podman-compose | `podman compose` (v2) |

### RHEL 8 Highlights

- Default OCI runtime: `runc`
- Container networking: CNI with `containernetworking-plugins` package
- Rootless networking: `slirp4netns` (slower than pasta)
- cgroup v1 default: no rootless resource limits
- `fuse-overlayfs` required for rootless overlay storage
- `podman generate systemd` is the primary systemd integration method
- No Quadlet support
- Module streams: `dnf module enable container-tools:rhel8`

### RHEL 9 Highlights

- Default OCI runtime: `crun` (faster startup, lower memory)
- Netavark replaces CNI; Aardvark-DNS replaces dnsmasq
- `pasta` replaces `slirp4netns` for rootless networking (RHEL 9.2+)
- cgroup v2 default: rootless resource limits work with delegation
- **Quadlet introduced** in Podman 4.4 (RHEL 9.2)
- `podman generate systemd` deprecated (still functional)
- CNI configs auto-migrated to Netavark in most cases

### RHEL 10 Highlights

- Podman 5.x with breaking changes from 4.x
- `podman generate systemd` removed -- Quadlet is the only systemd integration
- Enhanced Kubernetes YAML support (Deployment, DaemonSet resources)
- Improved Quadlet features (dependency ordering, pod-level Quadlet)
- `podman compose` with Compose v2 support
- pasta networking default for all rootless containers
- Native kernel overlay for rootless in most cases (no fuse-overlayfs)
- Improved layer deduplication and garbage collection

## Common Pitfalls

**1. Missing subuid/subgid entries for rootless users**
Rootless containers fail with "user namespaces not enabled" when no entry exists in `/etc/subuid` and `/etc/subgid`. Fix: `usermod --add-subuids 100000-165535 --add-subgids 100000-165535 <user>`

**2. Using `:Z` on system directories**
Relabeling `/home`, `/etc`, or `/var` with container-private SELinux labels breaks system services. Use dedicated directories for bind mounts.

**3. Expecting Docker daemon socket**
Podman does not use `/var/run/docker.sock`. The Podman API socket is at `/run/podman/podman.sock` (rootful) or `/run/user/<UID>/podman/podman.sock` (rootless).

**4. cgroup v1 rootless resource limits**
On RHEL 8 (cgroup v1), `--memory` and `--cpus` flags are silently ignored in rootless mode. Upgrade to RHEL 9+ with cgroup v2 for rootless resource controls.

**5. Using `podman generate systemd` on Podman 5.x**
This command was removed in Podman 5.x (RHEL 10). Migrate to Quadlet `.container` files.

**6. Docker Hub rate limiting**
Anonymous pulls from Docker Hub are rate-limited. Use `podman login docker.io` or configure a mirror in `registries.conf`.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Daemonless model, rootless containers, storage, networking, pods, registry configuration. Read for "how does X work" questions.
- `references/diagnostics.md` -- Container troubleshooting, rootless issues, common problems and fixes. Read when troubleshooting.
- `references/best-practices.md` -- Quadlet, image management, security hardening, auto-update, resources, Podman Machine. Read for configuration and best practices.

## Diagnostic Scripts

Run these for rapid Podman assessment:

| Script | Purpose |
|---|---|
| `scripts/01-podman-health.sh` | Version, runtime, storage driver, registries, system info, disk usage |
| `scripts/02-container-inventory.sh` | Running/stopped containers, pods, images, volumes, networks, stats |
| `scripts/03-rootless-audit.sh` | subuid/subgid, namespaces, cgroup delegation, rootless storage, networking |
| `scripts/04-quadlet-status.sh` | Quadlet units, .container/.volume files, auto-update labels, systemd status |

## Key Paths and Files

| Path | Purpose |
|---|---|
| `/etc/containers/storage.conf` | System storage configuration |
| `/etc/containers/registries.conf` | Registry search, mirrors, blocks |
| `/etc/containers/containers.conf` | System container defaults |
| `/etc/containers/policy.json` | Image signature policy |
| `/etc/containers/systemd/` | System Quadlet unit files |
| `~/.config/containers/systemd/` | User Quadlet unit files |
| `/etc/subuid`, `/etc/subgid` | Rootless UID/GID subordinate ranges |
| `/var/lib/containers/storage` | Root image/container storage |
| `~/.local/share/containers/storage` | Rootless image/container storage |
| `/run/podman/podman.sock` | Rootful Podman API socket |
| `/run/user/<UID>/podman/podman.sock` | Rootless Podman API socket |
| `/etc/cni/net.d/` | CNI network configs (RHEL 8 only) |