Proxmox

---
name: proxmox
description: "Proxmox VE expertise across supported versions (8.x, 9.x): QEMU/KVM virtual machines, LXC system containers, Corosync clustering, HA Manager, Ceph integrated storage, ZFS, LVM-thin, vzdump/PBS backup, SDN (VXLAN/EVPN), Linux bridge and OVS networking, Cloud-Init, OVMF/UEFI, VirtIO drivers, and REST API automation via pvesh. Use for \"Proxmox\", \"PVE\", \"Proxmox VE\", \"qm\", \"pct\", \"pvesh\", \"pvecm\", \"Ceph Proxmox\", \"vzdump\", \"PBS\", \"Proxmox Backup Server\", \"pveam\", \"Corosync\", \"HA Manager\", \"LXC\", \"ZFS Proxmox\", \"SDN Proxmox\". Do NOT use for standalone Linux container runtimes (Docker/Podman/Kubernetes) — that's the `containers` plugin; LXC here refers only to Proxmox's built-in system containers."
license: MIT
---

# Proxmox VE

This skill covers Proxmox VE across all supported versions (8.x and 9.x). It covers:

- QEMU/KVM virtual machines (qm CLI, VirtIO, OVMF/UEFI, Cloud-Init, QEMU Guest Agent)
- LXC system containers (pct CLI, unprivileged containers, templates, bind mounts)
- Corosync clustering (quorum, QDevice, multi-ring, fencing)
- HA Manager (resource scheduling, fencing methods, IPMI/watchdog)
- Ceph integrated storage (RBD, CephFS, OSD management, pool configuration)
- ZFS storage (pools, datasets, snapshots, send/receive, compression)
- LVM and LVM-thin provisioning
- Backup (vzdump snapshot/stop/suspend modes, PBS incremental backup, retention)
- Software Defined Networking (VNets, Zones, VXLAN, EVPN, BGP)
- Linux bridge and Open vSwitch networking (VLANs, bonding, LACP)
- Built-in firewall (datacenter/node/VM levels, security groups)
- REST API (pvesh CLI, API tokens, Terraform integration)
- Web UI administration (port 8006, HTTPS)
- VM templates and linked clones
- PCIe passthrough and VFIO

## How to Approach Tasks

When you receive a request:

1. **Classify** the request type:
   - **Troubleshooting** -- Load `references/diagnostics.md`
   - **Optimization** -- Load `references/best-practices.md`
   - **Architecture** -- Load `references/architecture.md`
   - **Administration** -- Follow the admin guidance below
   - **Scripting** -- Apply qm/pct/pvesh/bash expertise directly

2. **Identify version** -- Determine which PVE version the user is running. If unclear, ask. Key differences exist between 8.x and 9.x (firewall backend, Ceph version, kernel).

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply Proxmox-specific reasoning, not generic Linux advice.

5. **Recommend** -- Provide actionable guidance with specific commands (qm, pct, pvesh, pvecm).

6. **Verify** -- Suggest validation steps using the web UI, CLI checks, or journal logs.

## Core Expertise

### QEMU/KVM Virtual Machines

VMs are defined in `/etc/pve/qemu-server/<vmid>.conf`. Each VM runs as a QEMU process with KVM acceleration. The `qm` command is the primary CLI.

```bash
# VM lifecycle
qm list                                # List all VMs
qm start 100                           # Start VM
qm shutdown 100                        # Graceful ACPI shutdown
qm stop 100                            # Force power off
qm status 100                          # Check status
qm config 100                          # Show configuration

# Create a VM
qm create 200 --name myvm --memory 2048 --cores 2 \
  --net0 virtio,bridge=vmbr0 --scsi0 local-lvm:32

# Disk operations
qm disk resize 100 scsi0 +10G         # Grow disk
qm disk move 100 scsi0 ceph-storage   # Move disk to different storage
qm disk import 100 disk.qcow2 local-lvm  # Import disk image

# Snapshots
qm snapshot 100 pre-update            # Create snapshot
qm listsnapshot 100                    # List snapshots
qm rollback 100 pre-update            # Rollback
qm delsnapshot 100 pre-update         # Delete snapshot

# Migration
qm migrate 100 node2 --online         # Live migration (shared storage)
qm migrate 100 node2 --online --with-local-disks  # Live migration with disk copy
```

**VirtIO drivers** are essential for performance. Linux guests have built-in support. Windows guests require the virtio-win ISO. Key devices: `virtio-scsi` (disk), `virtio` (network), `virtio-balloon` (memory), `virtio-rng` (entropy).

**QEMU Guest Agent:** Install `qemu-guest-agent` in the guest for graceful shutdown, IP reporting, and filesystem freeze before snapshots.

### LXC System Containers

LXC containers share the host kernel but have isolated namespaces. They are lighter than VMs, start in seconds, and are ideal for Linux workloads.

```bash
# Container lifecycle
pct list                               # List all containers
pct start 101                          # Start container
pct shutdown 101                       # Graceful shutdown
pct stop 101                           # Force stop
pct config 101                         # Show configuration

# Create unprivileged container
pct create 201 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname myct --memory 512 --swap 512 \
  --rootfs local-lvm:8 --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1

# Container access
pct enter 101                          # Enter shell (nsenter)
pct exec 101 -- systemctl status       # Run command inside

# Templates
pveam update                           # Update template index
pveam available --section system       # List available templates
pveam download local debian-12-standard_12.7-1_amd64.tar.zst
```

**Unprivileged containers** (default): UIDs mapped to high-range host UIDs. Container root cannot access host resources. Always prefer unprivileged.

**Privileged containers:** Use only when direct hardware access is required. Security risk.

### Clustering

Proxmox clustering uses Corosync for inter-node communication and the pmxcfs cluster filesystem for configuration synchronization.

```bash
# Cluster operations
pvecm create mycluster                # Create cluster (first node)
pvecm add <ip-of-first-node>          # Join additional node
pvecm status                          # Cluster and quorum status
pvecm nodes                           # List nodes with IDs
pvecm expected 1                      # Emergency: set expected votes (recovery only)
```

**Quorum:** Requires majority of votes. 3 nodes = quorum at 2. For 2-node clusters, add a QDevice:
```bash
pvecm qdevice setup <external-host-ip>
```

### HA Manager

HA automatically restarts VMs/containers on another node if their current node fails. Requires shared storage (Ceph/NFS) and working fencing.

**Fencing is critical.** Without fencing, HA cannot safely restart resources because the failed node's state is unknown (it might still be running VMs, causing dual-write corruption).

| Fencing Method | Description | Reliability |
|---|---|---|
| IPMI/iDRAC/iLO | Out-of-band power management | Most reliable |
| Hardware watchdog | Watchdog timer triggers reboot | Good |
| Softdog | Software watchdog (fallback) | Acceptable |

### Storage

**Local storage types:**
| Type | Snapshots | Thin Provisioning | Use Case |
|---|---|---|---|
| Directory (qcow2) | Yes | Yes | Simple, default |
| LVM | No (LVM snap only) | No | Basic block storage |
| LVM-thin | Yes | Yes | Good balance |
| ZFS | Yes (instant) | Yes | Best for single-node |

**Shared storage types:**
| Type | Live Migration | Use Case |
|---|---|---|
| Ceph RBD | Yes | Best for HA clusters |
| CephFS | N/A (files) | Templates, backups, ISOs |
| NFS | Yes | Simple shared storage |
| iSCSI | Yes | SAN environments |
| GlusterFS | Yes | Alternative to Ceph |

### Backup

**vzdump** creates backups in `.vma` (VM) or `.tar.zst` (LXC) format:
```bash
vzdump 100 --storage pbs --mode snapshot   # Backup to PBS
vzdump --all --storage local               # Backup everything
```

**Proxmox Backup Server (PBS):** Separate product providing incremental, deduplicated, encrypted backups. Client-side encryption with AES-256-GCM. Verification and retention policies.

Backup modes:
- **snapshot** (recommended): uses LVM/ZFS/Ceph snapshots, minimal downtime
- **suspend**: suspends VM during backup
- **stop**: stops VM for maximum consistency

### Networking

**Linux bridges** (`vmbr0`, etc.) are the default. Physical NICs enslaved to bridges; VMs/containers attach virtual interfaces.

```bash
# VLAN-aware bridge (in /etc/network/interfaces)
# auto vmbr0
# iface vmbr0 inet static
#   address 192.168.1.10/24
#   gateway 192.168.1.1
#   bridge-ports eno1
#   bridge-stp off
#   bridge-fd 0
#   bridge-vlan-aware yes
#   bridge-vids 2-4094
```

**SDN (Software Defined Networking, PVE 8.x+):**
- VNets: virtual networks spanning the cluster
- Zones: Simple, VLAN, QinQ, VXLAN, EVPN
- EVPN/BGP for multi-site overlays
- Apply SDN config: `pvesh set /cluster/sdn`

**Bonding modes:** active-backup (mode 1), 802.3ad LACP (mode 4), balance-alb (mode 6).

### pvesh REST API

`pvesh` exposes the full Proxmox REST API from CLI:
```bash
pvesh get /cluster/status                    # Cluster health
pvesh get /cluster/resources --type vm       # All VMs
pvesh get /nodes/<node>/status               # Node CPU/RAM/uptime
pvesh get /nodes/<node>/qemu/<vmid>/config   # VM config
pvesh create /nodes/<node>/qemu/<vmid>/status/start  # Start VM
```

All web UI operations are API calls. API tokens can be created for automation.

## Common Pitfalls

**1. No fencing configured before enabling HA**
HA without fencing can cause split-brain and data corruption. Always configure and test IPMI/iDRAC fencing before adding HA resources.

**2. Overcommitting memory without KSM monitoring**
Proxmox enables KSM (Kernel Same-page Merging) by default. For VMs, do not overcommit memory in production. LXC containers are safer to overcommit.

**3. Using privileged containers in production**
Privileged containers give container root host-level device access. Use unprivileged containers exclusively unless a specific application requires privileged.

**4. Running Ceph on spinning disks without separate journal**
Ceph OSD performance on HDDs is poor without NVMe journal/WAL devices. Use NVMe for OSD WAL/DB, HDDs only for bulk capacity.

**5. Not separating Ceph and cluster networks**
Ceph replication traffic can saturate the cluster network. Use dedicated NICs/VLANs for Ceph public and cluster networks.

**6. Using pvecm expected in production without understanding**
`pvecm expected` temporarily overrides quorum requirements. Misuse can cause split-brain. Only use for controlled single-node recovery.

**7. Ignoring ZFS ARC memory usage**
ZFS ARC (Adaptive Replacement Cache) uses available RAM aggressively. Set `zfs_arc_max` to leave sufficient memory for VMs. Check with `cat /proc/spl/kvm/arcstats`.

**8. Not backing up /etc/pve/ before cluster operations**
The cluster filesystem `/etc/pve/` contains all VM/container/storage/network configuration. Back up before any cluster modification.

**9. Forgetting QEMU Guest Agent for snapshot consistency**
Without the guest agent, vzdump snapshot mode cannot freeze the filesystem. Install `qemu-guest-agent` in all VMs.

**10. Thin provisioning without disk space monitoring**
LVM-thin, qcow2, and Ceph all allow overcommit. Monitor actual usage; set alerts at 80% to avoid pool exhaustion.

## Version Notes

### Proxmox VE 8.x (Debian 12 "Bookworm")
- Linux kernel 6.2-6.8+
- OpenZFS 2.2 with dRAID support
- Ceph Quincy/Reef/Squid
- SDN moved to stable (8.1+)
- Notification framework (8.2+)
- Enhanced VMware OVA import (8.3+)

### Proxmox VE 9.x (Debian 13 "Trixie")
- Linux kernel 6.12+
- Full nftables firewall backend (replaces iptables)
- OpenZFS 2.3
- Ceph Squid (19.x) default
- QEMU 9.x with VirtIO improvements
- LXC 6.x improvements

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- KVM+LXC internals, storage backends, networking models, clustering, backup. Read for "how does X work" questions.
- `references/best-practices.md` -- ZFS vs LVM-thin selection, Ceph design, network architecture, HA fencing, resource overcommit. Read for design and operations questions.
- `references/diagnostics.md` -- Web UI monitoring, qm/pct diagnostics, cluster health, Ceph status, log locations. Read when troubleshooting.

## Script Library

- `scripts/01-pve-health.sh` -- Cluster status, node health, storage, Ceph, VM/CT counts
- `scripts/02-pve-inventory.sh` -- Full inventory report (VMs, containers, storage, nodes)