Unlike AMD's V (svm) support, Intel's VT (vmx) mode requires BIOS support.

More specifically, your motherboard vendor (or system vendor) must allow enabling vmx mode in their BIOS. Without BIOS support, you cannot use vmx mode.

Vendors apparently can disable vmx support in their systems entirely by setting the lock bit in the Feature Control MSR. Some vendors like HP have taken to disabling VT support in laptops, claiming that they disable it because they don't test it before shipping...

If your system BIOS supports enabling VT, doing so does NOT immediately make VT mode available. In fact you must hard power cycle the CPU for this change to take effect.

While documented fairly frequently (based on my google results), this apparently continues to bite new Xen HVM users.

Even systems without BIOSes sometimes need fixes as well.

Some early Macs with VT support needed modifications for DFI support for VT mode, I suffered through this with my early Mac Mini core duo.

Oh dear. I've really messed things up this time. I am entirely off base, and have confused a large number of people (including myself, apparently).

Any reference you've seen from me regarding VMI being a device interface is entirely wrong.

Any reference you've seen from me about Rusty maintaining VMI is entirely wrong.

This is a recent dialog with aliguori, someone directly involved in kvm/xen development, enough to tell me that I'm entirely off base:

*aliguori* paravirt_ops is a low-level paravirtualization interface.
it doesn't make any hypercalls but allows for "modules" to hook that
paravirtualization interface and then translate to the underlying
hypervisor's paravirtualization interface
*aliguori* there is a paravirt_ops implementation for VMI, Xen, and KVM
at the moment
*aliguori* you can think of paravirt_ops as paravirtualization
infrastructure, and then xen/vmi/kvm's paravirt_ops implementation as
drivers for specific hypervisors
*aliguori* and btw, there is no such thing as VMI device drivers
*aliguori* VMI is strictly a CPU paravirtualization interface
<aliguori> Zachary Amsden is doing the VMI paravirt_ops implementation,
Jeremy Fitzhardinge is doing the Xen paravirt_ops implementation, and 
Rusty is doing the lhype implementation (and I guess Ingo is sort of 
doing the KVM implementation)

Argh. So, mea culpa. I really messed that one up now, didn't I.

Anything I said about virtual devices is apparently entirely off base. Now I get to ensure that future posts are accurate on this matter.

IOMMUs and the future of hardware virtualization

There is one last thing to think about: isolation capable IOMMUs. Soon next generation Intel VT-d and AMD SR-IOV capable CPUs should be out with isolation capable IOMMUs. This means that you will see huge speed improvements from IO virtualization, and the potential to both assign PCI devices to hardware virtualized operating systems and have new "virtual aware" devices from hardware vendors that can be shared by multiple guests at a hardware level.

According to jnalley's post on the Xen developer IRC channel, "SR-IOV allows a PCI-e device to present virtual functions to the root complex. This would allow a guest OS (domU) to access the device directly."

Intel VT-d and AMD IOV should be out sometime Real Soon Now

For more information on SR-IOV, visit the specifications for SR (and MR) IOV.

I hope this helps clears things up.

Again, my apologies for those who were misled by my misunderstanding.

Yesterday, someone stumbled into the #kvm channel and mentioned that VirtualBox has gone OpenSource.

After some frantic questions and listening to the #vbox channel, it became apparent that there are some benefits and limitations of VirtualBox worth noting.

VirtualBox can use Intel/VT or AMD-V/SVM if available, but does not require it. Much like VMWare, which take the same hybrid software/hardware approach to virtualization. For 32bit guests, this can be much faster than pure VT/SVM.

VirtualBox (herein referred to as VBox) is similar to VMWare workstation or VMWare server, in that it has a ring0 kernel driver for a linux host.

This ring0 requirement means that it is not compatible with a Xen paravirtualized domU (and that includes dom0).

VBox leverages QEMU heavily for software emulation of real-mode and other critical code sections, as well as for hardware emulation.

QEMU has a closed source kernel module, kqemu, and a somewhat alpha quality opensource equivalent, qvm86, that do the software code-scanning method of virtualization. They do not require or recognize VT/SVM.

VBox's primary competitor is the kvm project, which provides QEMU based VT/SVM guests. The downside of kvm, of course, is the requirement for VT/SVM support from your CPU. VirtualBox has no such limitation.

VBox only supports 32bit host kernels and 32bit guest images. There is no 64bit support for either running under a 64bit Linux host kernel, or running a 64bit guest OS. The website does mention that 64bit support is under active development, however.

VBox has yet another virtual bus of virtual devices, akin to Xen's paravirtualized XenBus devices (or Virtual Iron's NexBus). While hardware devices are available (PCNet32, etc) using QEMU hardware emulation, VBox also has some excellent video/network/disk drivers that eliminate the hardware chipset emulation overhead.

VMWare tried to make VMI a standard for paravirtualized bus devices. The Linux kernel developer community initially balked, but VMI support lives on in Rusty's paravirt-ops patches. Recently, Ingo has been making great strides with paravirtualized kvm support.

One oddity is that VBox uses .VDI files for its disk images. Not QEMU's QCOW format, not VMWare's VMDK format, and not RAW disk image format.

And for the n00bs that keep popping in and asking about 3d support. No, VBox doesn't proxy 3d. No, QEMU doesn't proxy 3d. Yes, you can use a 3d card with a Xen paravirtualized domain (NOT with an HVM domain).

The only virtualization platform that supports 3d for Windows guests, that I am aware of, is VMWare 5.0 and later which have a somewhat crashy "beta" DirectX 3d support. (Simply add "mks.enable3d = TRUE" to your .vmx file by hand, for more info try googling for "mks.enable3d").

Parallels has promised 3d guests for 4th quarter of this year. If they deliver it, I will be pleasantly suprised.

If you really need 3d gaming for Windows games on a non-Windows platform, consider Transgaming's Cedega product line. Yes, it is Wine. Yes, there is a 50% overhead for the emulation. No, you're not going to do much better without running windows bare iron.

Where does this leave me? In limbo, mostly. I have a 32bit farm of Xen hosts moving toward a 64bit Xen hosting platform at the moment. Xen appears to be crawling while other tech like kvm and virtualbox keep popping up to challenge it. Xen's "maturity" is only really a year at best with its HVM support (quite a lead in tech terms), I can see l-hype/kvm and virtualbox quickly overshadowing Xen in the near future.

Eventually, VMI/paravirt-ops is going to level the playing field with standardized guest device drivers, regardless of hosting platform. Until then, we continue to craft guests based on the virtualization platform under which they will be run.

While Xen is a wonderful virtualization platform, there are a number of lesser known limitations of Xen which aren't well documented. You learn these limitations from first-hand experience.

Xen modes of operation

There are 3 modes of operation for Xen:

• 32bit
• 32bit+pae
• 64bit

The hypervisor mode must match the PV mode. As dom0 is a PV, that means it must match the mode of the hypervisor. This goes for all PV domains.

This means you can't run a pure 32bit PV under a 64bit hypervisor. Nor can you run a 32bit+pae PV under anything but a 32bit+pae hypervisor It must match, all the way through.

The Xen developers are working to fix this, eventually.

The same is not true for HVM operation: you can run 32bit HVM domains under a 64bit hypervisor/dom0.

The easiest way to find out what modes are available to you is to run "xm info | grep xen_caps". That will tell you exactly what guests you can run with your current setup.

Xen does not page

The Xen hypervisor does not page/swap to disk. In fact, the Xen hypervisor isn't directly aware of disk storage at all. All IO goes through the dom0 kernel which communicates with PCI devices.

Xen only manages available RAM.

By default, the Xen Balloon driver allows PV domains to be allocated some amount of RAM (up to maxmem) or reduced to some miminum amount of RAM (minmem), on the fly.

HVM domains allocate maxmem on start, and cannot be resized dynamically (you must restart the domain).

The Xen Balloon driver is shunned all over the xen-devel list historically. It has gotten better over time, though it still has some interesting behaviors.

With the current 3.0.4, for example, if you are running a PV domain with less than maxmem memory assign and save that domain to migrate it, when to restore the domain, it will allocate maxmem memory to it.

Every version of Xen tweaks the behavior of memory allocation just a little more. The full history of said behavior is still well beyond my understanding at this time.

Xen shared pages are limited

When a domU is started, there are a number of "shared pages" between the dom0 and the domU for them to communicate using a system of grants and page flipping between them.

Sadly, this grant space is limited. So limited in fact, that other Xen limits were introduced:

Xen 3.0.3 limits domUs to 3 network interfaces

This is due in part to the above shared page pool limitations.

People were using many many network interfaces, each incurring additional stress on the limited shared resources for inter-domain communication.

Apparently, part of the "fix" was to impose an artificial restriction of 3 network interfaces for all domUs in Xen 3.0.3.

Xen has a potential DoS condition if netloop isn't used

This one is particularly disturbing, and hard to explain or gauge how limiting it really is.

When a domU sends a packet to dom0, the ethernet frame is put into a shared page and access is granted for dom0 to use it.

While dom0 is using that page for the shared ethernet frame, there is a danger that a busy network might drain all available shared pages and Xen may panic.

As long as dom0 is immediately copying off frames to another network interface to be shipped off, there is no problem.

If, however, packets are destined to be processed by dom0 userspace, that skb sits in kernel space until the userspace daemon processes that packet's contents. This causes a strain and potential exhaustion of shared dom0/domU pages for these packets to sit around until they are handled.

Ouch.

This is where netloop comes in. Netloop is a Xen driver that provides a vif0.0/veth0 pair locally to the dom0 explicitly to be used to buffer those ethernet frames. By adding vif0.0 to a bridge along with the vif of a domU guest, any packets destined to be handled by dom0 userspace can take its sweet time and no problems will befall the system.

If you have any dom0 servicing domUs with userspace daemons, and you're not using a netloop to copy the frames, you may want to rethink this immediately. This includes routed/bridged/natted configurations, anything where a packet is handled by a dom0 userspace daemon coming from a domU.

Xen schedulers

There are 3 schedulers in Xen:

• BVT
• SEDF
• CREDIT

Both BVT and SEDF are "complex and buggy", and will go away in future releases.

CREDIT

• Is the simplest of the bunch to use.
• Handles SMP much more efficiently than both of the previous schedulers.
• Doesn't have the real-time behavior of SEDF (time-sensitive guests can be impacted, such as VoIP or any RTP streaming applications)
• Is the default scheduler in 3.0.3 and newer
• Is the only one that will survive going forward

Xen HVM gotchas

HVM domains require an Intel VT or an AMD V (SVM) capable processor. You can check your cpuinfo flags for "vmx" or "svm" to see if your processor has support for this feature.

The qemu bios used by xen is not patched for lba48, and you are limited to 160G disks.

You can use the commercial XenSource PV drivers (from XenExpress) to avoid the qemu-dm hardware emulation overhead.

HVM domains currently do not suspend/restore/migrate, much less live migrate. The announcement for 3.0.4 suggests that this is a feature slated for 3.0.5.

SMP support for HVM guests in 3.0.4 is better, as is support for other non-windows and non-linux guests, but I've yet to get SMP HVM guests working myself.

Xen volume size limits

There were numerous reports of 2TB limits with Xen vbd volumes in as late as Xen 3.0.3, even with 64bit. No, I do not know if 3.0.4 addressed them.

Xen logical volume resizing

You can't resize LVM2 logical volumes on the fly and have the domU see them to allow them to resize their filesystems without rebooting.

This means downtime whenever I need to grow a domU's filesystem. I get to lvextend it, reboot the domU, then xfs_growfs the filesystem. In that order.

Frequency Scaling kills Xen

Just turn off any frequency scaling in your dom0 (like AMD powernowd, or cpufreq settings), it drives Xen crazy.

Xen's ACPI support

Xen has minimal ACPI support. Don't think you're going to get S3 or S5 sleep suspend/resume working with Xen on your laptop. If you do, LET ME KNOW.

Xen Xserver video drivers

The nVidia video driver needs the following patch to work with Xen.

There have been a couple of reports of symbol errors when loading this. No, I haven't ried it myself, this patch was from someone else via IRC (nick long forgotten):

patch-nv-1.0-9625-xenrt.txt

Xen PVs run ring1, not ring0

This means you can't run VMWare, QEMU/kqemu, or Linux kvm under a Xen PV (this includes dom0, which is a glorified PV).

In theory, you should be able to run VMWare or QEMU/kqemu under an HVM domU.

Xen supported kernels

Xen 3.0.3 ships with patches for Linux 2.6.16.29. Xen 3.0.4 ships with patches for Linux 2.6.16.33.

If you have a newer kernel running Xen, it's probably a distribution patched version.

This means, if you want a driver from 2.6.18 or 2.6.19, you either need to backport said driver to 2.6.16.x, or you need to bravely forge ahead and risk help from the xen-devel team.

Not that you're entirely unsupported, just that your distribution is bravely adopting a newer kernel with untested/unsupported patches.

In conclusion

Those are most of the biggies that people seem to clamor about the most. If you have any others, please drop me a line.