May 26, 2009, 8:30 am
Cutting edge virtualization technology has always been one of Fedora’s strong suits and Fedora 11 looks to continue that trend. In an interview with Daniel P. Berrange, Red Hat Virt Team Engineer and Fedora Virtualization guru, we talk about the many key upgrades to virt technology in F11 focusing on areas of usability, performance and security. Fedora 11 will premiere the latest in secure and powerful virtualization technology available to users and developers. With so much to look forward to Fedora 11, it’s sure to make your virtualization dreams a reality.
1. Please introduce yourself, and tell us about your work in virtualization and how you got started.
I’m one of the lead developers for the libvirt project and am actively involved in many related areas of open source development (qemu/kvm, xen, gtk-vnc, virt-manager, to name but a few). I also co-maintain many of these packages in Fedora and RHEL, along with many others in Red Hat’s virtualization team.
More than three years ago (shortly after transferring into Red Hat’s Engineering team, from consulting services) I was working on the OLPC project. We needed a way to easily test the OS images we were building without needing real hardware. As a proof of concept, I hacked up a simple GTK application to run images them under QEMU. At around the same time Daniel Veillard had started the libvirt project and there was a desire for a desktop application to manage Xen using libvirt. So I switched over to the virtualization team, wrote virt-manager for Fedora 6, and my involvement in all areas of open source virtualization grew from there.
2. Many people view the work being done on virtualization as a feature set of major importance and significance. Can you give us a brief overview of some of the changes we can expect to see in Fedora 11?
The open source virtualization development effort is so large now, that it is useful to discuss each stream in turn.
At the lowest layer is obviously the Linux kernel and KVM/QEMU. There has been a major acceleration of development in QEMU and push to merge KVM into the official QEMU source repository. There’s ever continuing work on performance, stability, scalability and reliability in KVM. PCI device passthrough is one new feature we’re highlighting for Fedora 11. The return of Xen Dom0 was not to be, as the Dom0 paravirt_ops merge with the upstream Linux kernel is still an ongoing process.
At the middle layer is libvirt, providing a consistent management API across different virtualization technologies. New features in libvirt, since F10, include PCI device passthrough for Xen and KVM, the sVirt security driver using SELinux to protect KVM guests from each other, thread safety of all libvirt APIs, improved scalability, reliability and debugging for the libvirtd daemon and support for SCSI HBAs and copy-on-write volumes in the storage management APIs.
The top layer covers end user tools such as virt-install and virt-manager. virt-manager is undergoing a significant (and ongoing) overhaul of its user interface. The first improvements arriving for Fedora 11 are in the guest installation process and storage management capabilities. As guest installation is first task most users try, ensuring this is simple and reliable is key to making a good first impression. Guest desktop interaction is another historical pain point which has been a focus for improvements in Fedora 11.
With every release we also try to make a significant step forward in security of the virtualization stack. In Fedora 11 the focus has been on SELinux to protect guests from each other and SASL to authenticate VNC users.
3. There have been some large changes in virt-manager and libvirt, which are at the core of the user experience when it comes to virtualization. Can you talk to us more about those?
The guest installation process and desktop interaction are the most critical areas for making a good first impression. In the virt-manager re-design the wizard used for installing new guests has been streamlined, cutting out three redundant steps. Where possible, it will automatically detect the type of operating system being installed and choose the best configuration options to optimize for this OS, no longer requiring the user to figure this out for themselves.
The installation process now directly utilizes the libvirt storage management APIs to allow easy creation of files in a variety of formats (raw, qcow2, vmdk, etc), allocation of new local disk partitions or LVM volumes and access to LUNs exported by iSCSI targets. This is particularly useful when remotely managing virtualization hosts, allowing regular administrator tasks to be performed from the virt-manager UI without resorting to command line SSH sessions.
The mouse pointer has been a constant source of trouble for virtualization management applications. Getting the guest mouse pointer to track the host pointer is essentially impossible with the standard emulated PS/2 mouse. The solution is to provide a pointer device that supports absolute motion events, instead of relative events that the PS/2 mouse provides. For KVM and Xen, this means adding a USB tablet device, but historically Xorg has not been able to automatically configure this correctly. This is resolved with Fedora 11 guests, finally providing a guest pointer that moves in perfect sync with the host, not requiring the pointer to be confined to the guest window.
Users with non-US layout keyboards have also had a hard time getting their guests to support input of accented/special characters. The VNC protocol has now been extended to allow the hardware keycodes to be passed directly from the VNC client to the guest OS without any intermediate translation step. This should allow the guest OS complete control over the keyboard layout mapping, without a need for any special settings on the host.
The final piece of work was to increase the guest desktop resolution. The real Cirrus video card that QEMU emulates would never have done more than 800×600, but there are tricks that can be done in a virtual world. Thus a simple change to the Xorg cirrus driver allows it to detect that it is using a Cirrus card emulated by QEMU and increase the guest desktop resolution to 1024×768. Still not great by today’s standards, but better than before. Longer term plans involve replacing the cirrus driver in QEMU with something more virtualization friendly.
4. Also, as people should note, there has been a lot of work done surrounding KVM and getting that well integrated into Fedora. How has that work been going, and has anything significant been done in that area in this release?
Fedora was the first major Linux distribution to integrate KVM back in the Fedora 7 release. It became the default virtualization technology in Fedora 9, when it became clear we could no longer maintain the separate Xen host kernel until it was merged in the upstream Linux kernel. The great benefit of KVM from an distro integration point of view, is that it is there by default in all new Linux kernels. All that was required in Fedora was to turn on the module build and make sure the modules are always loaded when compatible CPUs are found. libvirt and virt-manager have also both supported KVM since it was first added to Fedora. Thus there hasn’t been a need for much additional integration work for KVM. The focus has simply been on improving features available to KVM users via libvirt and virt-manager.
5. Glauber Costa has also done significant work merging KVM and QEMU. Can you explain to us what QEMU is, and why the choice was made to merge it with QEMU? How that is of benefit to the user base?
Earlier Fedora releases have suffered from the divergence of QEMU and KVM code bases. Upstream QEMU has had releases almost 1 year apart, while KVM has been releasing at least once a month, if not more, using snapshots of the QEMU source repository. Thus the features available in QEMU were far behind those available in KVM even though they both shared the same fundamental code base and upstream development stream. It also doubled the work package maintainers had to do for security and bug fixes.
Since Fedora 10 though, the upstream QEMU community has accelerated its release schedule significantly and many of the KVM features have been merged back into the main QEMU code base. Thus we judged that the time was right to attempt to ship a single package containing both QEMU and KVM built from a single code base. For users this means that parity of features between QEMU and KVM, while the reduced burden on our Fedora package maintainers, ensures more timely security and bug fixes. Glauber also took the opportunity to split out all the virtual BIOS files and ROMs from QEMU into separate packages and ensure all are fully built from source using appropriate upstream source releases.
6. Virtualization and security are discussed hand in hand these days, and as the ability to create and use virtualized machines expands there are many security risks involved. Can you speak a bit to the work that was done to improve security both at the kernel and user levels?
In each Fedora release we try to make at least one significant step forward in the security of our virtualization technology. In Fedora 8, libvirt gained support for secure remote management using TLS for encryption and x509 client certificates for authentication, while GTK-VNC, QEMU, KVM and Xen were also all extended to add a VNC extension for TLS encryption providing a secure remote desktop. In Fedora 9 libvirt was further extended to support SASL enabling secure remote management with Kerberos for authentication and PolicyKit for local desktop authentication. Fedora 9 and 10 also increased use of SELinux to protect the host operating system from a compromised or malicious QEMU/KVM process.
The latter still did not provide any protection between guests, so one compromised QEMU process would still potentially be able to compromise another on the same host. Thus James Morris started work on a project known as sVirt, the first results of which are appearing in Fedora 11. The focus has been to provide isolation between guests running on a single host. libvirt directly integrates with SELinux to ensure every QEMU process it launches runs within a dedicated security context, only able to access its own assigned disk images. This protection is enabled by default on all Fedora 11 hosts using libvirt for management. As well as the security benefits, the end user experiance is improved because libvirt will automatically manage SELinux labelling for all guest disks, eliminating a major source of bug reports from previous Fedora releases.
7. These features have all evolved over time in previous Fedora releases and Fedora, as a distribution, has always been a leader in the virtualization realm. Let’s talk a bit now about the actual process of developing these features. How many of the improvements and enhancements to virtualization have come about as a direct result of the work done previously? Also, what does the future look like?
Virtualization technology in Fedora is reaping the benefit of very active upstream projects and the significant developer resources of Red Hat’s Virtualization Engineering team. The combination of these provide great opportunities for new features to have their debut in Fedora releases.
The ideas for new features come from many sources, some from Fedora end-user experiences and consequent bug reports, some magically arrive on cue from upstream projects, while others are things that look to be important for future RHEL releases. With the PCI device passthrough feature in F11, the core support was all already done by the upstream KVM community. This is a important feature for future RHEL, so Red Hat put resources into a F11 feature to add support to libvirt for PCI passthrough with KVM and Xen and then expose this in virt-manager.
The feature aiming to improve the guest desktop interaction was a result of the persistent stream of bug reports from Fedora users. We had been considering ways to address this over course of several Fedora releases, but it was not until Fedora 11 that all the pieces of the solution finally came together from the various upstream projects.
The security improvements in virtualization have a different origin. Very few end users ever file explicit bug reports asking for the addition of more authentication / encryption features or to use more SELinux. If anything users ask for ability to more easily turn off existing security features. This is a case where the user is not always right. They do want more security, but they just don’t know it yet!
As a developer it is necessary to be very proactive with security improvements. This can be particularly challenging work because the solutions often involve working across multiple upstream communities.
Take the VNC SASL authentication feature in Fedora 11 as an example. The first step was to write a specification for a new VNC security extension, have it reviewed and get a code for it allocated by RealVNC. Work on QEMU was required to implement the server side. Work on GTK-VNC was needed for client side. For management tools, work on libvirt was required to get the new security type enabled for guests it launches and finally virt-manager was extended such that it knows how to login. That’s five different projects involved for one feature. This is only practical by having a close working relationship with all the upstream communities and carefully coordinating the work there so it all arrives in time for the next Fedora release.
For the future we’re happy that libvirt gained support for managing VirtualBox recently and likely to soon have a driver for the Open Nebular cloud management project. Expect more advancements in sVirt, allowing for tighter controls on what a virtual machine can do, for example, ability to restrict network access of guests. libvirt will also gain the ability to manage host network configuration in Fedora 12, removing the need to manually configure bridge devices. Container based virt may make a more formal appearance in Fedora 12 as the native Linux container (LXC) support improves in the kernel and libvirt. The overhaul of the virt-manager user interface also continues.
8. Working on virtualization must be awfully time consuming and involved. Do you enjoy it? What do you do to get away from the pressures of hacking?
Working on open source virtualization technology is a great experience because it is a really interesting and challenging field, having plenty of talented developers to work with and learn from. There is plenty of work still to be done at all levels of the stack from kernel/hypervisor right through to end user applications and not nearly enough time to do it all. I’m fortunate to be able to spread my work between upstream projects, the Fedora community and RHEL releases and maintenance.
As for free time? What free time? I try to find time for a photography, with 4 out of my 5 cameras still using film, rather than digital.
