Author: David Graham
Corbet started his talk with a brief recap of the Linux kernel development process. According to Corbet, Linux kernels are now on a two- to three-month release cycle. The current Linux kernel version is 2.6.17.6, with 2.6.17.7 expected shortly. All 2.6.x kernels are major releases, with 2.6.x.y kernels being bug-fix releases.
Corbet says that there will not be a 2.7 kernel tree for the foreseeable future, not until there is a major, earth-shattering change that will break everything — and thereby require an unstable kernel tree.
The major release cycle developers use now takes approximately 8 weeks. In week 0, new features are included in the kernel in what is termed the merge window. This is typically in the form of several thousand kernel patches. This process ends when Linus decides there has been enough and the merge window is decreed closed.
The kernel then goes into release candidate mode, with effort going into stabilization and bug-fixing. Release candidate (-rc) kernels are released periodically and by the theoretical 8th week (which usually is a bit later), a major release is released. Subsequently, all bug-fixes and patches to that kernel come in the form of 2.6.x.y version numbers.
The process of merge windows started a year ago, Corbet said, and the result has been the relative predictability of stable kernel releases. New features come out quickly instead of spending years in queue; Distributions are keeping up with more current kernels than they had been.
Corbet showed a graph of kernel patches over time, showing how the number of patches going into the kernel has changed from a more or less straight line to a stair case pattern, with the help of the merge window release cycle now in use.
The quality of the new kernel release cycle has most people happy, Corbet said emphasizing “most.” The perception among some, he said, is that the quality of the kernels is on decline with too much emphasis on few features, and more bugs going in than coming out.
Corbet says that there’s not a firm kernel bug count. As the number of users increases, he noted, so to does the number of bug reports. More code means more bugs, even if the proportion of bugs (bugs per thousand lines of code) drops.
Many bugs being fixed are very old bugs, Corbet says. Of two recent security fixes, one was for a one year old security flaw, and the second was for a three year old security flaw. Fewer bugs, and a single bug database to centralize kernel flaws, would be nice to have; and Corbet says that he expects that progress on that front is on the way. Corbet also pointed out that bug tracking isn’t very helpful if the bugs don’t get fixed.
Kernel developers often lack the hardware needed to fix bugs, and so the bug-fix process can require extensive back and forth exchanges of tests and results. This process is very slow, and often times one party or the other gets bored of the process and the bug remains in the kernel.
Another problem, Corbet says, is that there is no boss to direct bug fixing efforts unless there is a corporate interest in fixing a bug somewhere and that company puts the resources in to getting specific bugs fixed. Kernel developers are also often reluctant to leave their little corner of the kernel, he noted.
Introducing bugs in the first place is becoming harder, said Corbet. Better APIs and more use of automated bug-catching tools are improving the situation. It has also been suggested that the Linux kernel do major releases that are strictly about fixing bugs, not adding features. Another suggestion floating around is the assignment of a kernel bug-master. It would need to be a funded position, he noted.
Future kernel development
Corbet went on to summarize the major changes in the kernel since this time last year, when kernel 2.6.12 was current. Among other specifics about the kernels released since, Corbet noted that Linux kernel 2.6.15 was released January 2nd, 2006, 15 years to the day after Linus bought his first development box to begin work on the kernel.
The kernel has a 15-year history, but it doesn’t have a five-year roadmap. Corbet says that the kernel has no specific timetable for features, or even a specific list of features that will be implemented, and that there’s no way to force development of any particular feature without specific funding. No one knows what hardware will be out down the road, or what users will want. What future we can predict, though, is the next kernel release.
The kernel 2.6.18 merge window has closed, Corbet says, and a number of changes will be in the upcoming release, including a new core time subsystem, and a massive patch set for serial ATA including error handling, and a kernel lock validator. The latter of these changes is designed to help with kernel development. Locks are designed to keep threads apart, he explained, and they’re difficult to get right. He also noted that devfs would be removed from the kernel in 2.6.18, which generated widespread applause.
Corbet went on to discuss challenges with integrating virtualization support with the kernel, noting that the various virtualization programs should not need to maintain their own trees and need to come up with a uniform set of patches into the kernel, to avoid each having its own set. He also spent some time discussion kernel security in the form of SELinux, which he said is acquiring real administrative tools, and AppArmor, SELinux competition recently purchased by Novell.
The Linux kernel is very unlikely to switch to GPL version 3, Corbet noted at the end of his excellent 45 minute talk, as changing the license would require a consensus of all kernel developers, who still individually hold the copyrights on their little bits of the code. This would not be helped, he noted candidly, by the fact that some are dead.
The slides from Corbet’s talk are available here.
Fully automated testing
Later in the day, I attended a talk by Google’s Martin J. Bligh entitled “Fully Automated Testing.” Bligh started by asking: Why? Automated testing, he says, is not just necessary because testing is a boring occupation. With the kernel 2.6 tree’s new development cycle, the rate of change of the kernel is quite scary. Linux is very widely used now, and old methods of bug reporting are no longer adequate.
It used to be that kernels were pushed out, and the developers could wait for feedback from users. Those days are over. Bligh noted that machines are cheap, compared to people, and automated testers don’t disappear. Is automated testing the solution to world peace and hunger? No, but Bligh says it’s part of a solution to kernel bugs. Automated bug testing requires more coders and more regression testing.
The testing is done upstream, he says, so the testing can be done prior to releases instead of after them. The fewer users exposed to a bug, the less pain caused. New code in the kernel can be pushed out of the tree until it’s fixed without causing additional problems, if bugs are found, before other features depend on the new code. The earlier bugs can be found, Bligh says, the better.
Bligh noted that extensive automated testing is done on the kernel twice a day. The test system is written in Python, and he discussed at length why Python was chosen as the language for the system. He also spent some time showing the audience test output from the system, and discussed why other languages were not suitable for the test system.
He described Python as a language that meets the requirements for the task because it is a language that is easy to modify and maintain, that is not write-only, that has exception handling, is powerful but not necessarily fast, is easy to learn, and has wide libraries of modules to leverage.
He described Perl as a write-only language, and said that while people can do amazing things with the shell, it is not appropriate for the purpose. He said with a grin that he has a lot of respect for what people can do in the shell, but none for choosing to do it that way in the first place.
One thing he particularly likes about Python, he said, is its usage of indentation. Unlike other languages, Bligh noted, Python is read by the computer the same way as it is read by a person, resulting in fewer bugs.
Bligh says test.kernel.org is better than it was before, but it is a tiny fraction of what could be done. Blight says that kernel testing would be improved by an open, plugin-able client to share tests. He also called for more upstream testing, and for companies to get involved in testing.
Is it cheaper, Bligh asked, for a company to debug code itself or to help track down bugs for the community to fix before it affects the company? He described the current automated test efforts as the tip of the iceberg. He also encouraged attendees to get involved by downloading the test harness and reading the wiki to get started.
Battery life
Len Brown of the Intel Open Source Technology Centre, and maintainer of the kernel ACPI subsystem, led the next session, “Linux Laptop Battery Life”. By their nature, Brown says, laptops are the source of most innovation in the area of power management.
The first part of his talk centered around how to measure how much power a laptop is using in the first place as a baseline. The first method he suggested is to use an AC watt meter with the battery out of the laptop, if possible. It’s a $100 test, he said, but fails on the AC to DC power conversion and on the fact that most laptops are aware that they are plugged into AC, and therefore unlimited, power and disable some power saving measures used when a battery is active.
The second method is fundamentally the same, but with a DC watt meter to eliminate the power loss caused by the power brick from the math. A more expensive but somewhat more accurate method, he said, is to set up a DC input system through the laptop battery leads.
The simplest solution though, he pointed out, is to simply use the laptop’s built-in information about the battery. Simply run a fully charged battery to fully depleted and see how long it takes. Compare that to the wattage of the battery and you have your power usage. For example, he said, a 53 watt-hour battery that runs a laptop for one hour means the laptop is running at 53 watts. If it lasts two hours, it is only using 26.5 watts, and so forth.
He announced the release of a GPLed program he has been working on which, he emphasis-ed, does not do benchmarks, called the Linux Battery Life Tool Kit. The code is available on his directory on kernel.org.
The first test of the testing program, he says, is the idle test. Idling is a most basic function of a laptop. It even idles between key presses on the keyboard, he noted. The next test he said is a Web read test, which looks at a different Web page every two minutes. He described it as an idle test with eye candy and said the results are indistinguishable from idle.
The next test is an OpenOffice.org-based test that specifically requires version 1.1.4 of OpenOffice.org. The next two tests are DVD playing with Mplayer and a software developer workload test, consisting of browsing and compiling source code.
He gave a number of examples based on his specific laptop, which he indicated vary widely from one laptop to the next, showing the results of his power usage tests under different circumstances.
The results gave both power usage and performance figures for the different circumstances tested. Among the statistics demonstrated was performance and power usage statistics for his dual-core laptop running with one core enabled versus disabled, and what effect a 5400 RPM hard drive had versus a 7200 RPM hard drive.
The speed of the hard drive had a huge performance boost for software development, but not noticeably anywhere else, and cost only a small penalty in power usage. Enabling the second core also cost little in extra power, but provides a significant performance boost. The biggest difference, he noted, was in LCD brightness. From brightest to dimmest setting on his LCD, the difference on his laptop’s battery life was more than 25%.
He also compared Windows XP to Linux performance on his laptop, noting again that performance differences were different from one laptop to the next. On his laptop, DVD playing was noticeably more power efficient in Windows than in Linux. He credited this to WinDVD buffering the DVD instead of just reading it constantly as Mplayer did.
The day was capped off by a reception by Intel, at which there were no speakers, but the winners of an Intel-sponsored essay competition about Linux or open source were announced. The winner, David Hellier, received a very nice laptop for an essay on bridging the digital divide. The runner up received an iPod for his essay, “I can.”
Category:
- News
