Will we have a new kernel by Christmas? I remember wondering that back in 1998 when 2.2 was on the horizon. Things have changed quite a bit since then in kernel development, and with Greg Kroah-Hartman’s announcement last week about changes in the stable kernel release procedure, it’s a good time to look at how the stable tree works.
A Little Backstory
Kernel development hasn’t always worked this way. If you’re relatively new to Linux, you may not remember the long, long, long wait for development kernels to become stable during the early 2.x series. The kernel developers would maintain a stable kernel with an even minor number (2.0.x, 2.2.x, 2.4.x) and a development kernel with an odd minor number (2.1.x, 2.3.x, 2.5.x).
In the case of 2.2.0, for example, Linus Torvalds was talking about releasing the 2.2.0 kernel in June or July of 1998. Months passed, and the question became “will we have 2.2 by Christmas?” We didn’t. The 2.2.0 kernel came out in January of 1999, quite a while after the 2.0.0 kernel which was released in June of 1996. 2.2.0 carried some big features people were eager to get — like improvements to SMP (multiple processors) and support for PowerPC processors.
It took quite a while for features to make it into stable kernels — so companies would port features from the development series into the stable series, and as Kroah-Hartman says it wasn’t easy:
2.5 was hard. Real hard. We had all these ideas of things we wanted to do, we adopted new tools and were figuring out how to use them, we did all sorts of crazy development as a large number of us were now being paid to do this development, so we had more time to do it.
And, because we all had more time to do this work, it took even longer. 2.6.0 didn’t come out until almost 3 years later, taking 86 releases.
Because of the long development cycle, companies really wanted to use parts of the 2.5 kernel in their “stable” releases. So they backported things from the 2.5 tree into the 2.4 trees, and started calling them “enterprise” kernel releases. These releases were _hard_ to do by the distros and the developers in charge of them. Usually the same developer had to make the changes in the 2.5 kernel and then turn around and try to figure out how to backport the changes to the 2.4 kernel for a distro to use. It was a nightmare for the developers, something that none of them who lived through it ever wanted to do again.
So, a number of changes were gradually introduced after the 2.6.0 kernel was released. Instead of starting a 2.7 development tree, Linus would release an -rc that had gradual changes from the previous kernel. Releases came out rapidly — every two to three months — but there were more modest changes between releases.
This is Kernel Development. There are Rules
The only objection to that style of development was a lack of a “stable” tree that could have bugfixes, security fixes, and so on. Three months is not a long time to wait for SMP — three months is eternity waiting for a security release!
So Kroah-Hartman started up the stable kernel tree to bridge the gap between the development kernel and the stable kernel. Right now, for example, the stable kernel is 2.6.36.1. But also 2.6.35.9, and 2.6.34.7, and several others. If that sounds confusing, it is. More on that in a second.
So what goes into the -stable kernels? There’s a few guidelines that are used to keep disruption in the -stable kernel to a minimum. It can’t be a large patch, it can’t be a new feature or an unnecessary trivial change (like spelling changes), and it has to be accepted by the subsystem maintainer. Security changes are, of course, high priority and appropriate for the -stable tree.
But a couple of kernels have been maintained as stable longer than others. Specifically, the 2.6.27 and 2.6.32 kernels. Why? Well these kernels have been used by major enterprise distributions. 2.6.32, for instance, was used for SUSE Linux Enterprise 11, Red Hat Enterprise Linux 6, and Ubuntu 10.04 LTS, to name a few. So that kernel will be on enterprise systems for quite some time. But the other kernels were maintained somewhat unevenly and there was a bit of confusion over how long kernels would be maintained and which one should be used.
Stable Kernels Now, and Longterm
So Kroah-Hartman has clarified how the releases will be maintained. Kroah-Hartman has introduced the concept of “longterm” kernels — which will include 2.6.27 and 2.6.32, and will be adding 2.6.35 to the list of longterm kernels. That will be maintained by Andi Kleen.
But Kroah-Hartman will only be maintaining the last kernel released as -stable.
Have questions or want to jump in on stable kernel development? There’s now a stable mailing list anyone can subscribe to to watch or work with stable development.
So, overall — not a massive change in the scheme of things. But it’s important, and will further refine the way the Linux kernel is developed and used, and continue the long tradition of open development that Linux users have enjoyed for nearly 20 years.
