While Xen is a wonderful virtualization platform, there are a number of lesser known limitations of Xen which aren't well documented. You learn these limitations from first-hand experience.

Xen modes of operation

There are 3 modes of operation for Xen:

• 32bit
• 32bit+pae
• 64bit

The hypervisor mode must match the PV mode. As dom0 is a PV, that means it must match the mode of the hypervisor. This goes for all PV domains.

This means you can't run a pure 32bit PV under a 64bit hypervisor. Nor can you run a 32bit+pae PV under anything but a 32bit+pae hypervisor It must match, all the way through.

The Xen developers are working to fix this, eventually.

The same is not true for HVM operation: you can run 32bit HVM domains under a 64bit hypervisor/dom0.

The easiest way to find out what modes are available to you is to run "xm info | grep xen_caps". That will tell you exactly what guests you can run with your current setup.

Xen does not page

The Xen hypervisor does not page/swap to disk. In fact, the Xen hypervisor isn't directly aware of disk storage at all. All IO goes through the dom0 kernel which communicates with PCI devices.

Xen only manages available RAM.

By default, the Xen Balloon driver allows PV domains to be allocated some amount of RAM (up to maxmem) or reduced to some miminum amount of RAM (minmem), on the fly.

HVM domains allocate maxmem on start, and cannot be resized dynamically (you must restart the domain).

The Xen Balloon driver is shunned all over the xen-devel list historically. It has gotten better over time, though it still has some interesting behaviors.

With the current 3.0.4, for example, if you are running a PV domain with less than maxmem memory assign and save that domain to migrate it, when to restore the domain, it will allocate maxmem memory to it.

Every version of Xen tweaks the behavior of memory allocation just a little more. The full history of said behavior is still well beyond my understanding at this time.

Xen shared pages are limited

When a domU is started, there are a number of "shared pages" between the dom0 and the domU for them to communicate using a system of grants and page flipping between them.

Sadly, this grant space is limited. So limited in fact, that other Xen limits were introduced:

Xen 3.0.3 limits domUs to 3 network interfaces

This is due in part to the above shared page pool limitations.

People were using many many network interfaces, each incurring additional stress on the limited shared resources for inter-domain communication.

Apparently, part of the "fix" was to impose an artificial restriction of 3 network interfaces for all domUs in Xen 3.0.3.

Xen has a potential DoS condition if netloop isn't used

This one is particularly disturbing, and hard to explain or gauge how limiting it really is.

When a domU sends a packet to dom0, the ethernet frame is put into a shared page and access is granted for dom0 to use it.

While dom0 is using that page for the shared ethernet frame, there is a danger that a busy network might drain all available shared pages and Xen may panic.

As long as dom0 is immediately copying off frames to another network interface to be shipped off, there is no problem.

If, however, packets are destined to be processed by dom0 userspace, that skb sits in kernel space until the userspace daemon processes that packet's contents. This causes a strain and potential exhaustion of shared dom0/domU pages for these packets to sit around until they are handled.

Ouch.

This is where netloop comes in. Netloop is a Xen driver that provides a vif0.0/veth0 pair locally to the dom0 explicitly to be used to buffer those ethernet frames. By adding vif0.0 to a bridge along with the vif of a domU guest, any packets destined to be handled by dom0 userspace can take its sweet time and no problems will befall the system.

If you have any dom0 servicing domUs with userspace daemons, and you're not using a netloop to copy the frames, you may want to rethink this immediately. This includes routed/bridged/natted configurations, anything where a packet is handled by a dom0 userspace daemon coming from a domU.

Xen schedulers

There are 3 schedulers in Xen:

• BVT
• SEDF
• CREDIT

Both BVT and SEDF are "complex and buggy", and will go away in future releases.

CREDIT

• Is the simplest of the bunch to use.
• Handles SMP much more efficiently than both of the previous schedulers.
• Doesn't have the real-time behavior of SEDF (time-sensitive guests can be impacted, such as VoIP or any RTP streaming applications)
• Is the default scheduler in 3.0.3 and newer
• Is the only one that will survive going forward

Xen HVM gotchas

HVM domains require an Intel VT or an AMD V (SVM) capable processor. You can check your cpuinfo flags for "vmx" or "svm" to see if your processor has support for this feature.

The qemu bios used by xen is not patched for lba48, and you are limited to 160G disks.

You can use the commercial XenSource PV drivers (from XenExpress) to avoid the qemu-dm hardware emulation overhead.

HVM domains currently do not suspend/restore/migrate, much less live migrate. The announcement for 3.0.4 suggests that this is a feature slated for 3.0.5.

SMP support for HVM guests in 3.0.4 is better, as is support for other non-windows and non-linux guests, but I've yet to get SMP HVM guests working myself.

Xen volume size limits

There were numerous reports of 2TB limits with Xen vbd volumes in as late as Xen 3.0.3, even with 64bit. No, I do not know if 3.0.4 addressed them.

Xen logical volume resizing

You can't resize LVM2 logical volumes on the fly and have the domU see them to allow them to resize their filesystems without rebooting.

This means downtime whenever I need to grow a domU's filesystem. I get to lvextend it, reboot the domU, then xfs_growfs the filesystem. In that order.

Frequency Scaling kills Xen

Just turn off any frequency scaling in your dom0 (like AMD powernowd, or cpufreq settings), it drives Xen crazy.

Xen's ACPI support

Xen has minimal ACPI support. Don't think you're going to get S3 or S5 sleep suspend/resume working with Xen on your laptop. If you do, LET ME KNOW.

Xen Xserver video drivers

The nVidia video driver needs the following patch to work with Xen.

There have been a couple of reports of symbol errors when loading this. No, I haven't ried it myself, this patch was from someone else via IRC (nick long forgotten):

patch-nv-1.0-9625-xenrt.txt

Xen PVs run ring1, not ring0

This means you can't run VMWare, QEMU/kqemu, or Linux kvm under a Xen PV (this includes dom0, which is a glorified PV).

In theory, you should be able to run VMWare or QEMU/kqemu under an HVM domU.

Xen supported kernels

Xen 3.0.3 ships with patches for Linux 2.6.16.29. Xen 3.0.4 ships with patches for Linux 2.6.16.33.

If you have a newer kernel running Xen, it's probably a distribution patched version.

This means, if you want a driver from 2.6.18 or 2.6.19, you either need to backport said driver to 2.6.16.x, or you need to bravely forge ahead and risk help from the xen-devel team.

Not that you're entirely unsupported, just that your distribution is bravely adopting a newer kernel with untested/unsupported patches.

In conclusion

Those are most of the biggies that people seem to clamor about the most. If you have any others, please drop me a line.