Search:
Where I Work
NKS
Subscribe
Add to Google
RSS 0.91
RSS 1.0
RSS 2.0
ATOM 1.0
RSS 2.0 and ATOM
Network
View Ian's profile on LinkedIn
Archives
2007 April (1)
2007 February (1)
2007 January (4)
2006 December (2)
2006 November (2)
2006 September (5)
2006 August (4)
2006 July (1)
2006 June (3)
2006 May (2)
2006 March (4)
2006 February (4)
2006 January (1)
2005 December (8)
2005 November (26)
2005 October (10)
2005 September (17)
2005 August (87)
2005 July (48)
2005 June (34)
2005 May (24)
2005 April (243)
2004 April (1)
2004 February (3)
2003 August (2)
2003 June (2)
2003 May (8)
2003 January (1)
2002 September (1)
2002 July (4)
2002 June (2)
2002 May (5)
2002 April (15)
2002 March (15)
Projects
CornFS
DENSO NAV
Rage Powered
Tampa Bay
TampaBad
SLUG
ob-buttons
Creative Commons OpenSource Linux Individual-i GeoURL Linux Speakeasy Speed Test
Twitter

follow icblenke at http://twitter.com
Google
Ian's shared items in Google Reader (subscribe)

CONSUMER AND GOVERNMENTAL AFFAIRS BUREAU EXTENDS EXPIRING CERTIFICATIONS FOR CERTAIN PROVIDERS OF VIDEO RELAY SERVICE AND IP RELAY SERVICE

CONSUMER AND GOVERNMENTAL AFFAIRS BUREAU EXTENDS EXPIRING CERTIFICATIONS FOR CERTAIN PROVIDERS OF VIDEO RELAY SERVICE AND IP RELAY SERVICE

Structure and Practices of the Video Relay Service Program

The YouTube Video You Don’t See

Example Show

Shop with confidence across the web

Helicopter view of your driving directions on Google Maps

Google CIO and others talk DevOps and "Disaster Porn" at Surge

Burning Man 2011 - Yes we were there.

September 08, 2011

Getting Started on the Google API

CACertMan app to address DigiNotar & other bad CA’s

Tangled

Custom Class Loading in Dalvik

Jingle Adventures contd…

TWO REPORTS OF ADVISORY COMMITTEES ON DISABILITIES ISSUES RELEASED

Join the White House Disability Group Monthly Call on July 27

Multiple APK Support in Android Market

Debugging Android JNI with CheckJNI

Android 3.2 Platform and Updated SDK tools

Geektalk

Believe in yourself

Forever alone involuntary flashmob

PS3 root key released - sign and run anything

lunar eclipse shadow on earth

hotpot NFC tags in portland

Oh, little bobby tables

Don't have a front-facing camera?

Tango.me

Looxcie

Mobile phone product testing: Models

Visual 6502

Extruding Light

Foam Printer

How Can the LHC withstand 1 Petabyte of Data a Second?

Linus Torvalds is now officially a US Citizen

Backin up quartet

Oh, hell yes.

Portland bike lanes get mario symbols

Skype RC4 claimed reverse-engineered

Best ever cease and desist

wkhtmltopdf - just awesome

Measurement Lab - Google IO BigQuery session is live querying 60 billion rows instantly

All you need is a little egotism, and $6

Examply punycode link

Convert IDN punycode to/from native characters

Sparkfun free day tomorrow: 1/7

websockets

C thulu ftagn recursion

Need a recursive DNS server? Use 8.8.8.8 and 8.8.4.4

Google Public DNS

JIQL - Java JDBC wrapper for Google DataStore

OpenNebula

Trillions

ZFS L2ARC ZIL on SSD

Swimming in OpenCL

Unicorn == Mongrel delayed_job

Remus - Transparent HA for Xen

Go

What DNS is not

Crossbow Virtual Wire Demo Tool

Banner ads on flies

PoolParty

Eucalyptus MySQL SOLR RabbitMQ Varnish == Nebula.nasa.gov

Nebula.nasa.org

Ubuntu Enterprise Cloud (UEC)

Evernote

Apple drops ZFS due to legal concerns

Peering disputes between Cogent and Hurricane Electric

Equinix to acquire Switch and Data for $689 million

We Are All Connected

Project kxen renamed project HXEN

Pomegranate Phone

Lessconf Jacksonville - followed the next day by Barcamp

Stick-figure guide to advanced AES crypto

Why you should pay attention to Google Wave

rails-primer - how to easily host rails projects on appengine

AppEngine-JRuby on google code

Ruby on Google AppEngine: appengine-jruby video

Dataliberation.org - The Data Liberation Front - a group concerned with moving data in and out of google

Detecting Spammers with SNARE: Spatio-temporal Network-level Automatic Reputation Engine

Proxmox VE - OpenVZ KVM Cluster appliance management

Sun/Oracle kill of SXCE: Sysadmins everywhere cry in horror.

Essentials of Metaheuristics

making water drinkable through nano-filtration

Pigin 2.6.1 adds Xmpp voice and video support

Opera Unite

Setting up a Layer-3 tunnel VPN using ssh 4.3 and -w option tun devices

shadowserver.org - botnet hunting resources

OpenBSC - a Siemens BS-11 microBTS or a ip.access nanoBTS == your own GSM tower

Voxbone's 883 country code

Apple keyboard firmware hack

Karesansui Project - a Xen management harness from Japan

eunicycle

Pygowave Server - Run your own Google Wave server

Happy Sysadmin Day!

Bokode

Bass cannon

Xen clocksource0 time went backwards

Internet vs World Population stats

BBC article on sat-3 cut

sat-3 cut

iPeak - RAIN

Asankya - RAIN

Apple pulls Google Voice app from iPhone - AT&T's fault

HadoopDB

live-android boot ISO - very neat

How to update your GeoIP information in addition to SWIPping

EATR

Google Wave hackathon on 20th/21st, if you happen to be in Mountainview

Did I mention OTOY here before?

NeatX - NX for Ganeti

STuPiD - STUN/TURN using PHP in Dispair

Aviary.com

Browser based Server-side 3D gaming from OTOY

Cisco's replacement for the WRT54GL is the WRT160NL

Spinn3r.com - Index the blogosphere

Team ARIN

Parts of galaxy Messier 87 are missing

DRAEGER ALCOTEST 7110 MKIII-C Evaluation of Breathalizer Source Code

Cyclops

Google's AJAX playground

How Michael Osinski Helped Build the Bomb That Blew Up Wallstreet

Bruce Perens - A Cyber-Attach on an American City

How Google and Facebook are using R

adito - the new gpl fork of the old sslexplorer project

A date idea: forklift sunset

Psytechnics - VVoiP QoE

r1soft cdp

IP Address geolocation for free

Shapeways - $50 "3-D poem rings" until the end of the month

GrandCentral to become Google Voice

Wolframalpha is coming

Hosted Xen Project

VirtualGL X11 transport

TurboVNC VirtualGL == FAST network GL

Ben Rockwood's presentation at the OpenSolaris Storage Summit: ZFS in the trenches

The Crisis of Credit Visualized on Vimeo

10gen - a java based app hosting infrastructure

Engineyard Vertebra - another cloud infrastructure management harness

Eucalyptus - an opensource EC2 compatible hosting infrastructure

asciicasts.com

railsbrain.com <-- ajaxified rdoc

AP IMPACT: SWAT Teams Deployed in 911 fraud

Lessons learned by people who have quit Google

Makwana indicted for Fanny Mae malware

"physicalized" servers

Zentific svn repo: alpha available

Holographic Space-Time ?

DACS - Distribution and Configuration System - version 2.0

Video of Cisco IOS attack talk at Chaos Computer Conference

Cosmic radio background noise 6 times higher than expected

We get a leap second tonight

Grow your own bioluminescent algae

Johnson and Ruby/Javascript

Two turntables and a git repo

Quartz Composer and Cruise Control status

Truthy and stupid.rb

The nature of truth

Get2Human

Sunay Tripathi's Solaris Networking Blog

Merry Christmas from XKCD

Merry Christmas from Chiron Beta Prime

Prius Emergency Generator

German folk tune Jazz improv

Memcached speed improvements

FSF sues Cisco

Asterisk Vishing Alert

Google's Native Client... the next ActiveX?

Waterballs

YAGNI development assistant

HA-xVM demo video posted

Kemari 1.0 released - HA Xen

The Decline and Fall of Agile

Zone Alarm 2009 Free Tomorrow

kenai.com - xVM Server Project site

58% Spam Drop from one colo shutdown

Xenomips - a Xen friendly domU version of Dynamips - Emulate a Cisco 7200

Debian and Android dual-boot on the G1

Sipper (SIPr) - a SIP testing framework in ruby

DBslayer - a SQL abstraction layer using JSON

Clojure - JVM based LISP dialect with immutable persistent data structures that are inherently thread safe

Fingerworks keyboard in a MacBookPro

NfSen - Netflow Sensor

The Phoenix BIOS hypervisor is Xen

Do you live in a Constitution-Free zone?

Puppet presentation at NYCOSUG this month

Kemari - Xen lock-step HA

XenSmartIO - Infiniband IO for Xen

Starting with b100, OpenSolaris has virtual consoles

OpenSolaris testfarm build server interface now available

Firefox M9 Fenric - Maemo alpha

SystemZ - aka Sirius - a port of OpenSolaris to IBM System Z mainframe OS running in z/VM mode

40.8% efficient solar cell

FREDNET

World sunlight map

Solaris and ZFS on a Dell 2950, tweaking notes

Logstalgia

Early Access Windows PV drivers for xVM

Economics: The Theory of Interstellar Trade

COMSTAR Admin Guide PDF file

The Financial Crisis: What Happened and What's Next?

3.5" DIY SSD drive

Microsoft usurping ODF

Cisco to run Windows 2008 on their appliance virtually for services

Packetfence: an OpenSource Network Access Control system

Public.resource.org

persist.js - an alternative to gears

Chinese building "impossible" EM drive

Supertinykeyboard

COMSTAR SMTF - solaris FC, SAS, and iSCSI targets

Flexiscale - yet another control panel?

RightScale - cloud control panels?

GoGrid, a servepath company.

OSCON in 37 minutes

Criticial ESXi remote vulnerability in openwsman

Parasitic power

Microsoft FUD on VMWare: vmwarecostswaytoomuch.com

nmap builds zenmap topology maps

Thu, 04 Jan 2007

While Xen is a wonderful virtualization platform, there are a number of lesser known limitations of Xen which aren't well documented. You learn these limitations from first-hand experience.

Xen modes of operation

There are 3 modes of operation for Xen:

  • 32bit
  • 32bit+pae
  • 64bit

The hypervisor mode must match the PV mode. As dom0 is a PV, that means it must match the mode of the hypervisor. This goes for all PV domains.

This means you can't run a pure 32bit PV under a 64bit hypervisor. Nor can you run a 32bit+pae PV under anything but a 32bit+pae hypervisor It must match, all the way through.

The Xen developers are working to fix this, eventually.

The same is not true for HVM operation: you can run 32bit HVM domains under a 64bit hypervisor/dom0.

The easiest way to find out what modes are available to you is to run "xm info | grep xen_caps". That will tell you exactly what guests you can run with your current setup.

Xen does not page

The Xen hypervisor does not page/swap to disk. In fact, the Xen hypervisor isn't directly aware of disk storage at all. All IO goes through the dom0 kernel which communicates with PCI devices.

Xen only manages available RAM.

By default, the Xen Balloon driver allows PV domains to be allocated some amount of RAM (up to maxmem) or reduced to some miminum amount of RAM (minmem), on the fly.

HVM domains allocate maxmem on start, and cannot be resized dynamically (you must restart the domain).

The Xen Balloon driver is shunned all over the xen-devel list historically. It has gotten better over time, though it still has some interesting behaviors.

With the current 3.0.4, for example, if you are running a PV domain with less than maxmem memory assign and save that domain to migrate it, when to restore the domain, it will allocate maxmem memory to it.

Every version of Xen tweaks the behavior of memory allocation just a little more. The full history of said behavior is still well beyond my understanding at this time.

Xen shared pages are limited

When a domU is started, there are a number of "shared pages" between the dom0 and the domU for them to communicate using a system of grants and page flipping between them.

Sadly, this grant space is limited. So limited in fact, that other Xen limits were introduced:

Xen 3.0.3 limits domUs to 3 network interfaces

This is due in part to the above shared page pool limitations.

People were using many many network interfaces, each incurring additional stress on the limited shared resources for inter-domain communication.

Apparently, part of the "fix" was to impose an artificial restriction of 3 network interfaces for all domUs in Xen 3.0.3.

Xen has a potential DoS condition if netloop isn't used

This one is particularly disturbing, and hard to explain or gauge how limiting it really is.

When a domU sends a packet to dom0, the ethernet frame is put into a shared page and access is granted for dom0 to use it.

While dom0 is using that page for the shared ethernet frame, there is a danger that a busy network might drain all available shared pages and Xen may panic.

As long as dom0 is immediately copying off frames to another network interface to be shipped off, there is no problem.

If, however, packets are destined to be processed by dom0 userspace, that skb sits in kernel space until the userspace daemon processes that packet's contents. This causes a strain and potential exhaustion of shared dom0/domU pages for these packets to sit around until they are handled.

Ouch.

This is where netloop comes in. Netloop is a Xen driver that provides a vif0.0/veth0 pair locally to the dom0 explicitly to be used to buffer those ethernet frames. By adding vif0.0 to a bridge along with the vif of a domU guest, any packets destined to be handled by dom0 userspace can take its sweet time and no problems will befall the system.

If you have any dom0 servicing domUs with userspace daemons, and you're not using a netloop to copy the frames, you may want to rethink this immediately. This includes routed/bridged/natted configurations, anything where a packet is handled by a dom0 userspace daemon coming from a domU.

Xen schedulers

There are 3 schedulers in Xen:

  • BVT
  • SEDF
  • CREDIT

Both BVT and SEDF are "complex and buggy", and will go away in future releases.

CREDIT

  • Is the simplest of the bunch to use.
  • Handles SMP much more efficiently than both of the previous schedulers.
  • Doesn't have the real-time behavior of SEDF (time-sensitive guests can be impacted, such as VoIP or any RTP streaming applications)
  • Is the default scheduler in 3.0.3 and newer
  • Is the only one that will survive going forward

Xen HVM gotchas

HVM domains require an Intel VT or an AMD V (SVM) capable processor. You can check your cpuinfo flags for "vmx" or "svm" to see if your processor has support for this feature.

The qemu bios used by xen is not patched for lba48, and you are limited to 160G disks.

You can use the commercial XenSource PV drivers (from XenExpress) to avoid the qemu-dm hardware emulation overhead.

HVM domains currently do not suspend/restore/migrate, much less live migrate. The announcement for 3.0.4 suggests that this is a feature slated for 3.0.5.

SMP support for HVM guests in 3.0.4 is better, as is support for other non-windows and non-linux guests, but I've yet to get SMP HVM guests working myself.

Xen volume size limits

There were numerous reports of 2TB limits with Xen vbd volumes in as late as Xen 3.0.3, even with 64bit. No, I do not know if 3.0.4 addressed them.

Xen logical volume resizing

You can't resize LVM2 logical volumes on the fly and have the domU see them to allow them to resize their filesystems without rebooting.

This means downtime whenever I need to grow a domU's filesystem. I get to lvextend it, reboot the domU, then xfs_growfs the filesystem. In that order.

Frequency Scaling kills Xen

Just turn off any frequency scaling in your dom0 (like AMD powernowd, or cpufreq settings), it drives Xen crazy.

Xen's ACPI support

Xen has minimal ACPI support. Don't think you're going to get S3 or S5 sleep suspend/resume working with Xen on your laptop. If you do, LET ME KNOW.

Xen Xserver video drivers

The nVidia video driver needs the following patch to work with Xen.

There have been a couple of reports of symbol errors when loading this. No, I haven't ried it myself, this patch was from someone else via IRC (nick long forgotten):

patch-nv-1.0-9625-xenrt.txt

Xen PVs run ring1, not ring0

This means you can't run VMWare, QEMU/kqemu, or Linux kvm under a Xen PV (this includes dom0, which is a glorified PV).

In theory, you should be able to run VMWare or QEMU/kqemu under an HVM domU.

Xen supported kernels

Xen 3.0.3 ships with patches for Linux 2.6.16.29. Xen 3.0.4 ships with patches for Linux 2.6.16.33.

If you have a newer kernel running Xen, it's probably a distribution patched version.

This means, if you want a driver from 2.6.18 or 2.6.19, you either need to backport said driver to 2.6.16.x, or you need to bravely forge ahead and risk help from the xen-devel team.

Not that you're entirely unsupported, just that your distribution is bravely adopting a newer kernel with untested/unsupported patches.

In conclusion

Those are most of the biggies that people seem to clamor about the most. If you have any others, please drop me a line.

Google
 
Web ian.blenke.com