Andrew Morton calls for more kernel testing

San Francisco — During the opening keynote at this week’s LinuxWorld Expo, kernel developer Andrew Morton called for more assistance in testing the Linux kernel from users, and predicted that virtualization would be the big thing for the next few years of kernel development.

Melinda Kendall, IDG World Expo’s vice president and general manager, told the audience that the organizers had heard that the keynotes had gotten very commercial. To counter this, and to give some technical meat to the attendees, they scheduled Morton’s appearance for the first day of the conference, when the attendees would be a more technical crowd. Morton, one of the lead kernel programmers, spent a little more than an hour talking to LinuxWorld attendees who showed up for the first day of tutorials. While the room was not packed full — organizers even taped off the back three rows of chairs to encourage attendees to sit up closer to the podium — those who did attend seemed quite interested in what Morton had to say.

One of the themes of Morton’s keynote was the need for more testing of the Linux kernel. He says that “testers are the key” to kernel development, and says that the kernel is now “mainly a maintenance exercise” consisting mostly of bug fixes, performance enhancements, new drivers, and cleaning the codebase to make it easier to maintain.

Despite kernel development being mostly a maintenance project, though, Morton says the pressure for new features is pretty high as well, particularly in the enterprise and server space, but also in desktop and consumer/embedded devices. In the past few years, Morton says three new processor architectures have been added to the kernel — the FRV, AVR32, and the Blackfin — all three targeted at embedded devices.

Despite the fact that Linux is being used in many embedded devices — Morton says that most of the high-end digital televisions being sold today use Linux, for example — Morton says that the embedded developers aren’t well-represented in kernel development. He says that there are a few high-profile embedded developers, but for the most part the Linux kernel team hears little from embedded developers.

Morton says that the likely reasons for this are competition and the perception that embedded developers don’t get the same kind of return on investment from trying to put changes back in the kernel as other developers. Because embedded devices are generally not expected to be updated during their lifecycle, he says there’s little benefit in pushing features used in embedded kernels back into the main kernel tree. While such a practice would benefit other developers — they’re able to track the kernel more easily when their features are in the main tree, not to mention being more likely to get development help from the rest of the kernel community — the embedded manufacturers won’t be updating the kernel shipped in consumer devices.

Morton also says that embedded developers are less likely to be involved in kernel development for fear of their competitors benefitting from their work. Morton says it’s “regrettable” that more embedded developers are not involved, and that he would like to see more participation from embedded developers and would like to have a better bead on what the embedded developers needs are.

It’s not forking, and that’s the end of it

One of the popular memes that pops up in the mainstream press is the “danger” that the Linux kernel might fork. Morton says that this won’t happen, simply for practical reasons.

According to Morton, “the only way a fork could work is if a large group of organizations were to split off and work together” to maintain a fork, which he says is unlikely. Morton says he hasn’t even heard of one company interested in doing this, much less several. He also says that if anyone is unhappy with kernel development “come tell us.”

Of course, many companies want things in the kernel that aren’t in the kernel yet or may never be in the kernel. Morton says that the answer to this is to maintain a private patchset against the kernel, and “everybody does that.”

The ability to follow the vanilla kernel with a patchset “relieves the pressure that there might be for a fork.”

Having dispelled the fork meme, Morton then told the audience “I don’t want to read that [the kernel is going to fork] in the press again, thank you.”

On testing

Morton also spent a fair amount of the talk encouraging people to test the development kernels. He acknowledged that testing kernels was a lot of work, and says that it’s often “not worth your time” because there’s little reward for testing kernels. Even when a user reports a bug in the kernel and helps see it through the process to be fixed, Morton says it’s the maintainer that gets credit in the changelog, rather than the user who reported the bug that the maintainer probably was to blame for in the first place.

However, if users are willing to test kernels “out of the goodness of your heart,” then the best kernel to test would be Linus Torvalds’ development branch, though the kernel developers are willing to take bug reports against kernels from distribution vendors as well.

Worried about a lack of testing skill? Don’t be, says Morton. He says that it’s far better to have a thousand users test the kernel simply by booting it up on all of their hardware to ensure that it can get as far as a login prompt than to have one user run a thousand tests against a kernel. The sheer variety of hardware and workloads that Linux is used on means that kernel developers can never test the kernel on everything, so they need to depend on users to help with that.

One of the hardest things about kernel testing, says Morton, is that it’s difficult for maintainers to reproduce bugs. If you find a bug, it’s imperative to report it and follow through the process to help the developer fix it. “Once you can reproduce a bug, it’s fixed.”

Morton says the kernel already has tens or hundreds of thousands of people testing development kernels. He noted that no commercial software company can match that effort “until they reach beta or early adopter stages, but that’s months or years too late.”

Morton also called out Slashdot commenters, saying that he often reads comments on Slashdot complaining of bugs that have never been reported to the kernel team. It’s an obligation, says Morton, that users have to help keep bugs from propagating. Morton says that if you encounter a kernel bug it’s important to “shout loudly” and not to give up until it’s fixed. “Do not accept it … let me know if you’re not getting help.”

Predicting the future

Of course, everyone wants to know about the kernel roadmap — but, Morton says, there isn’t one. He says neither he nor Torvalds nor other kernel developers have the power to control what anyone else is working on. So Morton didn’t give the audience a deep look into the crystal ball, except to say that there will be a lot of focus on virtualization: containerization, resource management, server consolidation, or whatever one wants to call virtualization or its features.

He says that the OpenVZ project has been working “very patiently” and that the kernel team is slowly moving OpenVZ into the kernel. He also says that getting the various virtualization technologies in has been “a lot of cat herding” but that it is “coming together quite nicely” and should be showing results in the next year or two.

A member of the audience wanted to know about GPLv3 — would the kernel be moving to GPLv3? Unlikely, says Morton. While the final draft of the license is more warmly regarded by Torvalds and others than some of the initial drafts, Morton says that it would be “a lot of discussion and effort and controversy” with no motivation to go through the effort to move the kernel to a new license. Unless there’s some compelling reason to switch to GPLv3, such as some legal issue that can’t be solved by remaining on GPLv2, Morton says it’s not going to happen in the near future.

If you’re not a kernel hacker or organization with enough money and clout to hire kernel developers, how do you get features into the kernel? There’s no guarantee, but Morton says that users should tell kernel developers what they need. In response to one question after the keynote finished, where the attendee wanted to know why he had no success in telling kernel developers that the file permission system should be more like Novell NetWare, Morton responded that the best approach was to tell kernel developers what you need, rather than saying the kernel should be more like another project or product. According to Morton, you’re much more likely to be successful if you tell the kernel team what you need and allow the team to come up with a solution, rather than say the kernel should be more like another OS.

LinuxWorld Expo continues through Thursday at the Moscone Center in downtown San Francisco.