[sf-lug] A Xen kind of experience

[Michael Paoli]

A Xen kind of experience

Anyway, one person's impression and experience thus far with Xen.

What: Xen - one particular flavor of virtual machine (VM)
http://www.xen.org/
http://en.wikipedia.org/wiki/Xen
http://www.google.com/search?hl=en&q=%22virtual+machine%22&btnG=Google+Search
License: GPL http://www.gnu.org/licenses/#GPL

So, ... the scenario: (as of several weeks ago) to set up
Debian GNU/Linux 4.0 r3 "Etch" i386
http://www.debian.org/
as a VM (for BALUG http://www.balug.org/) under Xen on
a CentOS release 5 (Final) host (the spiffy new SF-LUG box).
http://www.centos.org/
http://www.sf-lug.org/

Experience/background of implementer:
Linux/Unix systems administration experience - extensive
Debian systems administration experience - fairly extensive
Red Hat systems administration experience - substantial
CentOS systems administration experience - a fair amount
(CentOS is also functionally quite highly similar to Red Hat, due to its
origins and the distribution's objectives)
VM - conceptual - fairish
VM prior hands on experience - VMware (http://www.vmware.com/) only
(just set up one VMware installation prior - a BALUG VM on an
untangle.com host).

I'll draw *some* comparisons to VMware ... mostly just because that's
the only other VM (well, non-mainframe software VM, anyway) that I've
had hands on experience with ... but I'll try not to go too heavy on the
comparisons.

Very initial impressions: Xen (vs.) VMware ...
licenses/free/freedom
Xen *much* better license - free as in freedom
VMware ... free ... as in beer, ... not freedom (comparatively sucky
license, stuff tied up in closed binaries, etc. ... but nevertheless
quite functionally useable ... at least for certain purposes).
Access/control - notwithstanding the above ... VMware ... very initial
control a bit more of a pain to set up (binaries, only supported on
certain operating system distributions/versions/releases/architectures)
... but once in place, a pretty nice GUI (pretty polished, easy to use)
... but no readily noticeable CLI for handling such, or for massive
scalability, etc. (maybe it's there, but such wasn't readily apparent,
... or may not be within the free as in beer scope).  Xen - much nicer
in not having highly restrictive control requirements ... got ssh or X
and can run a superuser (root) shell on the host?  - if so one's well
on the way to start setting up and controlling Xen.  Privilege
isolation?  Actually better with VMware ... don't have to be superuser
to set up and manage a VM with VMware, whereas one needs to be
superuser to do so with Xen (though sudo might be quite useful for
limiting such use, that's likely only practical for "routine
operations" after setting up a Xen VM ... much more mucking about as
superuser is needed with Xen to configure and set up a VM).  "Full"
virtualization vs. paravirtualization?
http://en.wikipedia.org/wiki/Paravirtualization
VMware seems to me to come out ahead there.  Although newer Xen also
supports full virtualization, it seems a bit rocky yet, in my
experience and in the environment I used it in (perhaps due to some
other needed support/dependencies for it to work ... but in VMware it
"just worked", in Xen, attempting fully virtualization, I got as far on
doing install of the operating system in the VM as recognizing and
getting ready to partition the virtual disk - then the VM locked up
with an error, and would continue no further - so much for my first
attempts at full virtualization under Xen ... and unfortunately the
installation attempt took a fair while to get to that point where it
locked up, so I wasn't inclined to make many repeated attempts).
Anyway, enough comparisons for now.

So, ... I start looking around at man(1) pages for and related to Xen
(which was already installed, and even had a couple VMs in place -
which may or may not have been functioning/operational) on the CentOS
release 5 (Final) host.  A word or two on terminology - in the context
of VM, "host" is commonly used to refer to the system which hosts the
VM, and "guest" as the (virtual) system within the VM.  Particular
terminology used often varies with the flavor of one's VM ...  perhpas
a bit more on that later.  So anyway, I find and read - and take notes
from relevant man pages - most notably xm(1) and xmdomain.cfg(5).
Well, initially, that all seemed fairly well and good, ... also
contained appropriate references, ... initially most notably, to where
the configuration files are - at least by default - and the syntax for
configuration.  So, I look over the installed VMs as examples.  That's
where stuff starts to get "interesting" ... I'd say more *problematic*.
 Much of the configuration (keywords and syntax) used in the existing
VMs wasn't covered at all in xmdomain.cfg(5) ... like probably about
50% of what was configured in the existing VMs wasn't documented in the
man page of how to configure them!  Not a great start.  So ... off to
reading and checking more references, web searches, etc.  The
impressions quickly gained didn't paint an especially pretty picture.
Most notably, although Xen certainly seemed to be "production ready" in
terms of capabilities and use and stability, and being well pressed and
used in production in numerous installations, there were still some
notable significant "hazards":
The documentation lagged way behind the operational state, the CentOS
release 5 (Final) distribution included *unstable* Xen ...  not exactly
a great confidence builder regarding what one may then expect to
encounter (okay, sure, Red Hat probably grabbed unstable to get the
more new cool/hot features and capabilities, and theoretically
stabilized it before making it their production release ... the sources
of which CentOS would then in turn be based upon) ... but it was
quickly clear, in any case, that the documentation lagged way behind
the implementation.  (and "read the source" isn't an excuse for lack of
good documentation! ... although at least if the source is available,
it's possible to create the missing documentation.)  Additionally, much
of the materials read strongly implied that one need pay due attention
to the particular distribution/release of operating system one was
dealing with, and also the particular release of Xen - as apparently
many of the details would vary among these - and in many cases quite
critically.

So, ... after much reading and noting various on-line materials, and
trying different stuff out, I was not having a whole lot of Xen
success.  After poking around more in what CentOS includes that's Xen
related, I eventually stumble across some stuff that looks a bit
promising - I find among the Xen related packages and their contents:
virt-manager and virt-install.  Well, ... those look nice and
promising, ... except, only problem ... they're not documented!  No man
pages for them or anything like that.  Ah, ... but they're not
binaries.  But alas, they're Python scripts, ... and I don't grok
Python.  So, anyway, poking a bit more with those, etc., going over
more (mostly on-line) documentation, references, user
reports/suggestions, etc., I find I can set up a CentOS VM under CentOS
... not so useful or interesting for what I want to do, ... but
something anyway.  Also, virt-install and virt-manager don't give me
control of all the stuff I want to configure and set up for the Xen
guest.  But, nevertheless, they give me something I can build and
inspect, to try and figure out how to build something as I actually
want to.  I even made use of strace(1) to try and determine what the
heck virt-install and virt-manager were managing to do, ... and how.
Well, turns out they mostly just communicate with xend ... ah, no
problem, check out the xend manual page, right?  Uhm, ... no manual
page for xend.  Anyway, another thing that I noted from all the various
bits of documentation and information, is that it seems it's a lot
easier for a given distribution, to set up a guest of the same - or
quite similar - distribution under Xen, ... than some other
distribution.  E.g. setting up CentOS/Red Hat/Fedora under CentOS/Red
Hat/Fedora, or Debian/Ubuntu under Debian/Ubuntu, etc.  It also
appeared that moving a Xen guest from host to host - at least with
reasonably similar host architectures, was relatively easy.  So, ... I
decided to change approaches a bit.

On the VMware guest, running Debian ... I installed Xen on that, ... and
built a Xen Debian guest under the Debian VMware guest.  I then "moved"
(copied) that (relatively bare bones Debian Xen guest) over to the
CentOS host, and set it up as a VM under Xen on that CentOS host.  And,
... it worked like a charm (though a rather indirect way of getting
there).  Once that was in place I was able to continue working with that
VM to progress on its installation and configuration (still a work in
progress, but it's pretty far along).

So, ... where does one find all the useful/definitive information on
Xen?  Well, in a few <cough, cough> places.  Some bits are definitive
(uhm, like the source), and there's lots of other stuff of varying
usefulness (some rather to quite useful).  Anyway, in my Xen experience,
these were the resources I noted as at least fairly, to quite useful:
xm(1)
xmdomain.cfg(5)
http://www.howtoforge.com/debian_etch_xen_3.1
http://www.howtoforge.com/debian_etch_xen_from_debian_repository
http://wiki.xensource.com/xenwiki/XenDocs
(and yes, there are books on Xen ... some still to be published ...
from over 300 pages, to over 700 pages ... given what's lacking and
missing entirely in man pages, somehow I find large books on Xen not to
be surprising)
http://wiki.xensource.com/xenwiki/HowTos
http://tx.downloads.xensource.com/downloads/docs/user/
http://tx.downloads.xensource.com/downloads/docs/user/#SECTION04130000000000000000
http://wiki.xensource.com/xenwiki/DebianDomU
http://conshell.net/wiki/index.php/Debian_Sarge_on_Xen
http://home.att.net/~Tom.Horsley/xen-fci.html
virt-manager
virt-install
http://www.howtoforge.com/centos_5.0_xen
http://wiki.debian.org/Xen
http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/user.html
http://www.xensource.com/
http://wiki.xensource.com/xenwiki/
xentop(1) (not all that useful, but nevertheless)