Kvm

---
name: kvm
description: "KVM/QEMU with libvirt expertise across Linux distributions (RHEL, Ubuntu, Debian, Fedora, openSUSE): kernel hypervisor, QEMU device emulation, libvirt (libvirtd/virtqemud), virsh, virt-install, qemu-img, virt-manager, storage pools, virtual networking (bridges, NAT, macvtap, OVS, SR-IOV), VirtIO, CPU pinning, NUMA/hugepages, live migration, snapshots, and guest image tools (guestfish, virt-customize). Use for \"KVM\", \"QEMU\", \"libvirt\", \"virsh\", \"virt-install\", \"qemu-img\", \"virt-manager\", \"libvirtd\", \"virtqemud\", \"virt-top\", \"virt-customize\", \"guestfish\", \"virtiofs\", \"VFIO\", \"IOMMU passthrough\", \"/dev/kvm\", \"qemu-system\". Do NOT use for Linux container runtimes (Docker/Podman/containerd) — that's the `containers` plugin; do NOT use for general RHEL administration unrelated to virtualization — that's the `os` plugin's `rhel` skill."
license: MIT
---

# KVM/QEMU with libvirt

This skill covers KVM/QEMU with libvirt across Linux distributions (RHEL, Ubuntu, Debian, Fedora, openSUSE). It covers:

- KVM kernel hypervisor (kvm, kvm_intel/kvm_amd modules, /dev/kvm, EPT/NPT)
- QEMU device emulation (machine types, VirtIO devices, OVMF/UEFI, QMP/HMP)
- libvirt management layer (libvirtd, virtqemud modular daemons, connection URIs)
- virsh CLI (domain lifecycle, snapshots, migration, networks, storage pools)
- virt-install provisioning (ISO install, cloud images, cloud-init, PXE)
- qemu-img disk management (create, convert, resize, snapshot, backing files)
- virt-manager GUI and cockpit-machines web UI
- Storage pools (dir, LVM, iSCSI, Ceph RBD, NFS, GlusterFS)
- Virtual networking (NAT/virbr0, bridged, macvtap, OVS, SR-IOV, isolated)
- VirtIO paravirtualization (virtio-blk, virtio-scsi, virtio-net, virtio-balloon)
- Performance tuning (CPU pinning, NUMA, hugepages, I/O threads, cache modes)
- Live migration (shared storage, copy-storage, post-copy)
- Snapshot management (internal, external, disk-only, with-memory)
- Guest image tools (guestfish, virt-customize, virt-sysprep, virt-df)
- VFIO/IOMMU device passthrough (GPU, NIC, NVMe)
- virtiofs host-guest filesystem sharing

## How to Approach Tasks

When you receive a request:

1. **Classify** the request type:
   - **Troubleshooting** -- Load `references/diagnostics.md`
   - **Optimization** -- Load `references/best-practices.md`
   - **Architecture** -- Load `references/architecture.md`
   - **Administration** -- Follow the admin guidance below
   - **Scripting** -- Apply virsh/bash/libvirt expertise directly

2. **Identify the distribution** -- KVM/libvirt behavior varies between distros. RHEL 9+ uses modular daemons (virtqemud). Ubuntu/Debian may use libvirtd. Package names differ.

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply KVM/libvirt-specific reasoning, not generic hypervisor advice.

5. **Recommend** -- Provide actionable guidance with specific commands (virsh, virt-install, qemu-img).

6. **Verify** -- Suggest validation steps using virsh, virt-top, or journalctl.

## Core Expertise

### KVM Architecture

KVM is a Linux kernel module that turns the kernel into a Type-1 hypervisor. Two modules: `kvm` (core) + `kvm_intel` or `kvm_amd` (vendor extensions).

```bash
# Verify KVM support
lsmod | grep kvm
grep -E 'vmx|svm' /proc/cpuinfo
# or
kvm-ok   # (cpu-checker package)

# Check EPT/NPT (hardware page tables)
cat /sys/module/kvm_intel/parameters/ept   # 1 = enabled
cat /sys/module/kvm_amd/parameters/npt     # 1 = enabled
```

QEMU runs as a userspace process per VM. KVM handles CPU/memory execution; QEMU handles device emulation. The `/dev/kvm` character device is the kernel interface.

### libvirt Management

libvirt provides a uniform API for managing KVM/QEMU (and other hypervisors). Two daemon models:

| Daemon | Model | Distro Default |
|---|---|---|
| `libvirtd` | Monolithic (all-in-one) | Ubuntu, Debian, older RHEL |
| `virtqemud` | Modular (per-driver) | RHEL 9+, Fedora 36+ |

```bash
# Start the appropriate daemon
systemctl enable --now libvirtd          # monolithic
systemctl enable --now virtqemud.socket  # modular

# Connection URIs
virsh -c qemu:///system list --all       # local system VMs (default)
virsh -c qemu+ssh://user@host/system list --all  # remote via SSH
```

### virsh Domain Lifecycle

```bash
# List and status
virsh list --all                          # all VMs (running + stopped)
virsh dominfo myvm                        # ID, UUID, state, memory, vCPUs

# Power operations
virsh start myvm
virsh shutdown myvm                       # graceful ACPI
virsh destroy myvm                        # force off (like pulling power)
virsh reboot myvm
virsh suspend myvm                        # pause vCPUs
virsh resume myvm

# Define and manage
virsh define myvm.xml                     # register VM from XML
virsh undefine myvm                       # remove definition (keeps disk)
virsh undefine myvm --remove-all-storage  # remove definition + disks
virsh edit myvm                           # edit XML in $EDITOR
virsh dumpxml myvm                        # export current XML
```

### VM Provisioning (virt-install)

```bash
# Linux VM from ISO
virt-install \
  --name ubuntu-24 \
  --ram 4096 --vcpus 4 \
  --disk size=40,format=qcow2,bus=virtio,cache=none \
  --cdrom /srv/isos/ubuntu-24.04-server.iso \
  --os-variant ubuntu24.04 \
  --network network=default,model=virtio \
  --graphics spice --video virtio \
  --channel unix,target_type=virtio,name=org.qemu.guest_agent.0 \
  --boot uefi \
  --noautoconsole

# Cloud image with cloud-init
qemu-img resize ubuntu-cloud.qcow2 30G
virt-install \
  --name mycloud --ram 2048 --vcpus 2 \
  --disk ubuntu-cloud.qcow2,bus=virtio,cache=none \
  --disk seed.iso,device=cdrom \
  --os-variant ubuntu24.04 \
  --network network=default,model=virtio \
  --import --noautoconsole

# Windows VM with VirtIO drivers
virt-install \
  --name win11 --ram 8192 --vcpus 4 \
  --disk size=80,format=qcow2,bus=virtio,cache=none \
  --cdrom /srv/isos/Win11.iso \
  --disk /srv/isos/virtio-win.iso,device=cdrom \
  --os-variant win11 \
  --network network=default,model=virtio \
  --boot uefi --tpm backend.type=emulator,backend.version=2.0 \
  --machine q35 --noautoconsole
```

### Disk Management (qemu-img)

```bash
# Create images
qemu-img create -f qcow2 myvm.qcow2 50G
qemu-img create -f qcow2 -b base.qcow2 -F qcow2 clone.qcow2  # linked clone

# Convert between formats
qemu-img convert -f vmdk -O qcow2 myvm.vmdk myvm.qcow2         # VMware to KVM
qemu-img convert -f qcow2 -O raw myvm.qcow2 myvm.raw           # qcow2 to raw

# Resize
qemu-img resize myvm.qcow2 +20G

# Inspect
qemu-img info myvm.qcow2
qemu-img check myvm.qcow2                # corruption check
```

### Snapshots

```bash
# Internal snapshot (qcow2, VM can be running)
virsh snapshot-create-as myvm snap1 "before update" --disk-only --atomic

# List and manage
virsh snapshot-list myvm --tree
virsh snapshot-revert myvm snap1 --running
virsh snapshot-delete myvm snap1

# External snapshot (creates new overlay file)
virsh snapshot-create-as myvm snap1 "pre-patch" \
  --diskspec vda,snapshot=external --disk-only
```

### Live Migration

```bash
# Shared storage migration (NFS/Ceph)
virsh migrate --live myvm qemu+ssh://dest-host/system

# Migration with disk copy (no shared storage)
virsh migrate --live --copy-storage-all myvm qemu+ssh://dest-host/system

# Persistent migration (undefine on source after success)
virsh migrate --live --persistent --undefinesource \
  myvm qemu+ssh://dest-host/system

# Monitor migration progress
virsh domjobinfo myvm
virsh domjobabort myvm              # cancel in-progress migration
```

### Networking

| Type | Use Case | Host-Guest | Performance |
|---|---|---|---|
| NAT (virbr0) | Default, outbound internet | Yes | Good |
| Bridged (br0) | VMs as LAN peers | Yes | Best |
| macvtap | Direct on physical NIC | No host-VM | Very good |
| OVS | SDN, VLANs, tunnels | Yes | Best |
| SR-IOV | Hardware VF passthrough | No | Near-native |
| Isolated | VM-to-VM only | No external | Good |

```bash
# Manage default NAT network
virsh net-list --all
virsh net-start default
virsh net-autostart default

# Create bridged network on host
nmcli con add type bridge ifname br0
nmcli con add type bridge-slave ifname eth0 master br0
```

### Storage Pools

```bash
# Define and start a directory pool
virsh pool-define-as mypool dir --target /srv/vmdisks
virsh pool-build mypool
virsh pool-start mypool
virsh pool-autostart mypool

# Create a volume
virsh vol-create-as mypool myvm.qcow2 50G --format qcow2

# List pools and volumes
virsh pool-list --all
virsh vol-list mypool
```

### Guest Image Tools

```bash
# Offline customization (no VM boot needed)
virt-customize -a myvm.qcow2 \
  --install "nginx,curl" \
  --root-password password:changeme \
  --ssh-inject root:file:/root/.ssh/id_ed25519.pub \
  --hostname newhost --selinux-relabel

# Sysprep before cloning
virt-sysprep -a myvm.qcow2

# Inspect guest filesystem
virt-filesystems -a myvm.qcow2 -l
virt-df -a myvm.qcow2
virt-cat -a myvm.qcow2 /etc/hostname
```

## Common Pitfalls

**1. Not enabling IOMMU for device passthrough**
Add `intel_iommu=on` or `amd_iommu=on` to kernel command line for VFIO/passthrough. Without IOMMU, device assignment fails or causes instability.

**2. Using emulated devices instead of VirtIO**
Emulated e1000 or IDE devices have 5-10x higher CPU overhead than VirtIO. Always use VirtIO for disk and network. Windows guests need the virtio-win driver ISO.

**3. Using cache=writeback without UPS**
`cache=writeback` risks data loss on power failure. Use `cache=none` with `io=native` for production (safest with battery-backed storage).

**4. Forgetting to install QEMU guest agent**
Without `qemu-guest-agent`, the host cannot: gracefully shutdown the VM, freeze filesystems for snapshots, or query guest IP addresses.

**5. Over-allocating vCPUs beyond NUMA node**
VMs with more vCPUs than a single NUMA node's core count will span NUMA nodes, causing 30-40% memory latency penalty. Check topology with `virsh capabilities`.

**6. Not using hugepages for large VMs**
VMs with 8+ GB RAM benefit significantly from 1 GB hugepages. Reduces TLB misses and improves memory performance.

**7. External snapshots without cleanup**
External snapshots create overlay files. Without `virsh blockcommit` or `blockpull` to merge them, the chain grows indefinitely and wastes space.

**8. Running libvirtd instead of modular daemons on RHEL 9+**
RHEL 9+ defaults to modular daemons (virtqemud). Running both libvirtd and virtqemud causes conflicts.

**9. Bridged networking without disabling netfilter on bridge**
Linux netfilter (iptables/nftables) processes bridge traffic by default, causing unexpected packet drops. Disable with:
```bash
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
```

**10. Not testing migration compatibility before production**
CPU feature mismatches between hosts cause migration failures. Use `cpu mode='host-model'` for migration safety, or ensure identical CPU generations.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- KVM kernel module, QEMU emulation, libvirt daemons, storage pools, networking models, XML domain definition. Read for "how does X work" questions.
- `references/best-practices.md` -- VirtIO configuration, hugepages, CPU pinning, NUMA tuning, cache modes, I/O threads, security. Read for design and operations questions.
- `references/diagnostics.md` -- virt-top monitoring, domstats, KVM debugfs, QEMU logs, troubleshooting common failures. Read when troubleshooting.

## Script Library

- `scripts/01-virsh-health.sh` -- Node info, pool/network/domain status, resource usage
- `scripts/02-virsh-inventory.sh` -- All VMs with state, CPU, memory, disk, network details