== Using Virtualization to Ease Emulab Upgrades ==
   by  Pat Gunn

=== Intuitions ===
        * Upgrading emulab is hard and risky
        * We sometimes have had downtime longer than we would like
         * Zero downtime/Zero risk would be fantastic

=== Virtualisation as a solution? ===
        * Benefits:
                * Snapshots
                * Upgrades on a forked system
                * Grow the testbed more smoothly
                * Resilience against hardware failures
                * Easily upgrade hardware by allocating more resources from resource pool
        * Issues:
                * Networking
                        * Configuration? Getting the VLANs right
                * Performance
                        * FreeBSD vs Linux - FreeBSD needs recent CPU hardware features to be efficiently virtualized
                        * General performance - Will UDP performance be good enough? Good NFS performance?

=== Solution so far ===
        * VMWare ESX
                *  Expensive, but we already have the license
        * Boss and Ops live in a cloud distant from the machines they manage. We might move them.
        * Systems connect through virtual ports on real switches
        * We can only safely snapshot nodes that are down (On Linux, this is slightly more flexible)
        * Performance tests so far show good UDP and TCP performance
                * We will see how this works in practice as we scale up
        * Imagined upgrade progress:
                * Disable logins and testbed daemons
                * Take testbed down
                * Clone the system
                * Take testbed back up
                * Boot clones of boss/ops, isolated from real versions
                * Upgrade clones
                * If upgrade not successful, delete clones, complain to Utah
                * If upgrade successful, shutdown old boss/ops, massage database and experiment disk state changes into upgraded boss/ops? Not clear on this part.
        * Fallback upgrade progress: Like a normal upgrade, but with a very good backup beforehand
        * Other nice things:
                * Virtual switches, virtual power controllers?
                * Compute nodes in a cloud? (Planetlab-esque?)  Virtualisation technologies are an active area of research at CMU/PDL
                * Boss/ops can run on the same suitably powerful system
                * Storage separate from boss/ops nodes

Our solution is in the very early stages of deployment. The new testbed we're building will be about 120 nodes, big enough to know if it's reasonable as a longer-term solution.

Mitch adds: We're looking into other virtualisation solutions that are more scriptable than VMWare.