Citrix

---
name: citrix
description: "Citrix Hypervisor (XenServer) expertise across all versions (6.x through 8.x, XCP-ng): Xen Type-1 hypervisor, dom0/domU architecture, XAPI toolstack, xe CLI, Storage Repositories (NFS, iSCSI, FC, GFS2), Open vSwitch networking, XenMotion live migration, pool management, HA with fencing, VM lifecycle, snapshots, backup strategies, and XenCenter/web console administration. Use for \"XenServer\", \"Citrix Hypervisor\", \"Xen\", \"xe \", \"XenCenter\", \"XenMotion\", \"dom0\", \"XAPI\"."
license: MIT
---

# Citrix Hypervisor / XenServer

This skill covers Citrix Hypervisor (XenServer) across all versions (6.x through 8.x and XCP-ng). It covers:

- Xen Type-1 bare-metal hypervisor architecture (dom0, domU, PV, HVM, PVH)
- XAPI toolstack (OCaml daemon, XML-RPC/JSON-RPC API, pool database)
- xe CLI for all host, VM, SR, network, and pool operations
- Storage Repositories: Local LVM, ext, NFS, iSCSI (lvmoiscsi), Fibre Channel (lvmohba), GFS2, SMB
- VHD chain management, thin provisioning, coalescing, and IntelliCache
- Open vSwitch networking (VLANs, bonds, QoS, PIFs, VIFs)
- Network separation (management, storage, VM, migration traffic)
- XenMotion and Storage XenMotion live migration
- Pool architecture, pool master election, and CPU masking
- High Availability with heartbeat disks and self-fencing
- VM lifecycle (create, start, stop, snapshot, checkpoint, export/import)
- XenCenter GUI and web console administration
- Backup strategies (VM export, CBT, pool database backup)
- Performance monitoring (RRDD metrics, dom0 resource tuning)

## How to Approach Tasks

When you receive a request:

1. **Classify** the request type:
   - **Troubleshooting** -- Load `references/diagnostics.md`
   - **Optimization** -- Load `references/best-practices.md`
   - **Architecture** -- Load `references/architecture.md`
   - **Administration** -- Follow the admin guidance below
   - **Migration** -- Apply XenMotion and Storage XenMotion expertise
   - **Scripting** -- Reference xe CLI patterns and dom0 shell access

2. **Identify version** -- Determine which XenServer version or XCP-ng release the user is running. Version matters for CBT support, PVH availability, and feature licensing.

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply XenServer-specific reasoning, not generic virtualization advice.

5. **Recommend** -- Provide actionable guidance with xe CLI commands.

6. **Verify** -- Suggest validation steps (xe commands, log checks, SR scans).

## Core Expertise

### Xen Hypervisor Architecture

Xen is a Type-1 bare-metal hypervisor. It manages CPU scheduling (Credit2 scheduler) and memory partitioning. I/O is delegated to the privileged dom0 domain via a split-driver model.

- **dom0** -- Privileged Linux VM (CentOS-based) with direct hardware access. Runs XAPI, all device drivers, and I/O backends. Single point of failure per host.
- **domU** -- Unprivileged guest VMs. PV (paravirtualized), HVM (full hardware virtualization), or PVH (hybrid).
- **XAPI** -- OCaml management daemon exposing XML-RPC/JSON-RPC API. All management flows through XAPI.

### xe CLI Operations

The xe CLI is the primary administrative tool. It communicates with the local XAPI daemon or remote hosts via `-s`, `-u`, `-pw`.

```bash
# Host and pool inventory
xe host-list
xe pool-list params=all
xe sr-list params=name-label,physical-size,physical-utilisation,type
xe network-list

# VM lifecycle
xe vm-list params=name-label,power-state,uuid
xe vm-start vm=<name-or-uuid>
xe vm-shutdown vm=<name-or-uuid>
xe vm-reboot vm=<name-or-uuid>

# Live migration
xe vm-migrate vm=<vm-uuid> host=<dest-host> live=true

# Snapshots
xe vm-snapshot vm=<name-or-uuid> new-name-label=<snap-name>
xe snapshot-revert snapshot-uuid=<uuid>

# Storage XenMotion (cross-SR migration)
xe vm-migrate vm=<vm-uuid> host=<dest-host-uuid> \
  vdi-map=<src-vdi-uuid>:<dest-sr-uuid> live=true
```

### Storage Repositories (SRs)

XenServer abstracts storage through SRs containing VDIs attached to VMs as VBDs.

| SR Type | Backend | Thin Provisioning | Best For |
|---------|---------|-------------------|----------|
| Local LVM | LVM on local disk | No (thick) | Single-host, high IOPS |
| ext | ext3 on local disk | Yes (VHD) | Local with snapshots/cloning |
| NFS | VHD on NFS share | Yes | Mid-tier, simple operations |
| iSCSI (lvmoiscsi) | LVM over iSCSI LUN | No | High-throughput block I/O |
| Fibre Channel | LVM over FC HBA | No | Lowest-latency block storage |
| GFS2 | Clustered FS over SAN | Yes | Active-active, vGPU migration |
| SMB | VHD on Windows share | Yes | Windows file server integration |

```bash
# Create NFS SR
xe sr-create name-label="NFS-SR" type=nfs shared=true content-type=user \
  device-config:server=192.168.10.50 device-config:serverpath=/exports/xen

# Create iSCSI SR
xe sr-create name-label="iSCSI-SR" type=lvmoiscsi shared=true content-type=user \
  device-config:target=192.168.10.60 \
  device-config:targetIQN=iqn.2024-01.com.example:storage01 \
  device-config:SCSIid=<scsi-id>

# Set default SR for pool
xe pool-param-set uuid=<pool-uuid> default-SR=<sr-uuid>

# Scan SR (triggers coalescing)
xe sr-scan uuid=<sr-uuid>
```

### Networking (Open vSwitch)

XenServer uses OVS as its default virtual switch. Physical NICs are PIFs; guest virtual NICs are VIFs.

```bash
# VLAN network
xe network-create name-label="VLAN-100"
xe vlan-create pif-uuid=<pif-uuid> vlan=100 network-uuid=<network-uuid>

# LACP bond
xe bond-create network-uuid=<network-uuid> \
  pif-uuids=<pif1-uuid>,<pif2-uuid> mode=lacp

# Jumbo frames for iSCSI
xe network-param-set uuid=<network-uuid> MTU=9000
```

Network separation best practice: dedicate NICs or VLANs for management, storage, VM traffic, and migration traffic.

### Pool Management and HA

A pool is a group of identically-versioned hosts sharing storage. One host is the pool master; all XAPI writes route through it.

```bash
# Enable HA with heartbeat SR
xe pool-ha-enable heartbeat-sr-uuids=<sr-uuid>
xe vm-param-set uuid=<vm-uuid> ha-restart-priority=restart

# Evacuate host for maintenance
xe host-evacuate uuid=<host-uuid>
xe host-disable uuid=<host-uuid>

# Emergency master recovery
xe pool-emergency-transition-to-master
xe pool-recover-slaves

# Pool database backup
xe pool-dump-database file-name=pool-db-backup.xml
```

HA uses shared disk heartbeats and network heartbeats. A host that loses both self-fences (reboots) to prevent split-brain. Set `ha-host-failures-to-tolerate` and maintain N+1 capacity headroom.

### VM Creation and Configuration

```bash
# Install from template
xe vm-install template=<template-uuid> new-name-label="my-vm-01"

# Configure resources
xe vm-param-set uuid=<vm-uuid> VCPUs-max=4 VCPUs-at-startup=4
xe vm-param-set uuid=<vm-uuid> memory-static-max=8589934592
xe vm-param-set uuid=<vm-uuid> memory-dynamic-max=8589934592
xe vm-param-set uuid=<vm-uuid> memory-dynamic-min=4294967296
xe vm-param-set uuid=<vm-uuid> memory-static-min=4294967296

# Create and attach disk
xe vdi-create sr-uuid=<sr-uuid> name-label="vm-disk" type=user virtual-size=107374182400
xe vbd-create vm-uuid=<vm-uuid> vdi-uuid=<vdi-uuid> device=0 bootable=true mode=RW

# Add NIC
xe vif-create vm-uuid=<vm-uuid> network-uuid=<network-uuid> device=0

# Export / import
xe vm-export vm=<name-or-uuid> filename=my-vm.xva compress=true
xe vm-import filename=my-vm.xva sr-uuid=<sr-uuid>
```

## Common Pitfalls

**1. dom0 undersized for VM density**
Default dom0 RAM (2-4 GB) is insufficient for hosts running many VMs. Increase to 4-8 GB to prevent dom0 memory pressure that degrades all guest I/O.

**2. Deep VHD snapshot chains causing slow I/O**
NFS and ext SRs build VHD parent-child chains on snapshot. Chains deeper than 10 levels degrade read performance. Limit snapshots and run `xe sr-scan` to trigger coalescing.

**3. ISO from local SR blocking live migration**
A VM with an ISO attached from a local SR cannot migrate. Detach the ISO before initiating XenMotion.

**4. Mixing Intel and AMD CPUs in a pool**
Pools require CPUs from the same vendor. Cross-generation migration within the same vendor family requires CPU masking.

**5. HA false fencing from heartbeat disk I/O timeouts**
Heartbeat SR must be on reliable shared storage with low latency. NTP skew between hosts can also trigger false fencing.

**6. Forgetting to back up the pool database**
The pool database contains all VM and pool metadata. Without it, disaster recovery is extremely difficult. Schedule regular `xe pool-dump-database` exports.

**7. Running out of SR space with thin provisioning**
NFS and ext SRs thin-provision VDIs. If the backing store fills, VMs pause or crash without warning. Monitor SR utilization proactively.

**8. Not separating storage and management network traffic**
Shared networks cause management timeouts during storage-heavy operations. Dedicate NICs or VLANs for each traffic class.

**9. Ignoring coalesce tasks after snapshot deletion**
Deleting a snapshot does not immediately reclaim space. Coalescing runs as a background SM task. Check `xe task-list | grep -i coalesce` and run `xe sr-scan` if space is not recovered.

**10. Applying patches without testing on a non-production host first**
XenServer hotfixes can cause regressions. Always test on a non-production host and verify `xe patch-list` output before rolling across the pool.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Xen hypervisor internals, dom0/domU model, XAPI toolstack, storage subsystem, networking, HA, pool management, XenMotion. Read for "how does X work" questions.
- `references/diagnostics.md` -- xe diagnostic commands, log locations, common issues table, performance monitoring, SR troubleshooting, HA debugging. Read when troubleshooting errors or performance.
- `references/best-practices.md` -- Pool design, storage selection, dom0 sizing, network separation, backup strategy, patching, monitoring. Read for design and operations questions.