As Etch progresses, Debian’s release manager talks turkey

Release managers are not mentioned in the Debian Constitution, yet few positions in Debian are more influential, especially in the final stages of preparation for a release. Recently, Andreas Barth, who shares the release manager position with Steve Langasek, took time from his efforts coordinating the Etch release — tentatively scheduled for early December — to talk about the stages in the release process, the goals for the upcoming release, and the short- and long-range problems that he faces in his role. Contrary to some predictions of disaster, he presents a picture of a distribution that is continuing to evolve without sacrificing the openness for which it is often admired.

The Debian release team for Etch would be a large department in most companies. Besides the two release managers, it includes five release assistants, and at least 20 other people involved in quality assurance, installation, and writing release notes and documentation.

“The release team,” Barth explains, “has to make sure that all of Debian is in a releasable state. This includes speaking with the maintainers, helping people decide which bugs to fix prior to release (sometimes it is better to allow a low-priority bug to stay instead of risking some breakage), [and] making sure nothing is forgotten.”

In addition, the process mobilizes the whole of Debian. “Almost everyone in the Debian community is involved in some way,” says Barth, “starting with the package maintainers who make sure their packages are all in a releasable state at the same time. Releasing is a common act of the whole project.”

Barth estimates that a Debian release involves coordinating the work of some 2,000 people. This coordination is complicated by the fact that the release team has little means to encourage cooperation, since it is working with volunteers working remotely. Most of the time, the release team leads by setting an example. People in the Debian community “want to see the release happen,” Barth says, “so they tend to listen to us — and they know that other people also tend to listen to us.”

The model works, if not always efficiently, because it is consistent with the general philosophy of the Debian community. “Debian’s governance model,” Barth says, “is centered a lot on ‘doing the right thing’ and ‘whoever does a job can make decisions.'” In other words, the release team receives cooperation to the extent that it is responsible and active.

Stages in the release

In some senses, preparation for a Debian release is continuous. New and upgraded packages are continually being added by Debian developers into the unstable repository. After 10 days, if no critical bugs have been filed against an unstable package, and if it does not interfere with other packages, then it is transferred to the testing repository. However, generally speaking, packages are not transferred into the stable repository, which represents the current official release of Debian, until a new minor or major version of the distribution is readied.

As a major release is prepared, Barth says, “We start slowing new upstream changes down to stabilize packages.” As a first step, the release team announces a “soft freeze,” starting with core packages, asking that maintainers start to take extra precautions with any changes. Eventually, a “full freeze” is announced, which means that each change needs to be reviewed by the release team according to the list of standards for the release. Currently, the Etch release is in soft freeze, with a full freeze expected once the number of critical bugs has been reduced.

As the packages are readied, other activities are also under way. The installation program is readied — a stage that was especially important in the last Debian release, since a new installer was being produced. This process promises to be equally important in Etch, given that a graphical installer is being introduced. Release candidates for each hardware architecture supported in the release also need to be prepared, and release notes and documentation must be written or updated.

Throughout all these processes, the release needs to be continually tested on all architectures. Debian has an official Quality Assurance sub-project, but “there are lots of different ways to test that standards are being met,” Barth says. For example, “We have tools that run over the archive to notice some issues, people are running testing and unstable [repositories] and report bugs, some people do checks on whether the packages still compile, and important packages usually have test suites included.”

Only when the number of bugs reaches a level that the release team considers acceptable is the release candidate, and, later, the official release announced. Ideally, this number would be zero, but, in practice, it usually amounts to a minimum of critical bugs in key packages. Without this compromise, the release could be delayed until it became obsolete.

Release goals

To add to the problems of coordination, the Debian release team for Etch divides its goals into two categories: blocking goals, which will delay the release until they are met, and non-blocking, which are considered desirable to fix, but not important enough to cause delays.

For Etch, two of the blocking goals are transitions in software: the migration from version 3.3 of the GNU Compiler Collection to version 4.0, which requires changes in C++ libraries for several hundred packages in order for the release to be successfully built, and the transition from XFree86 to X.org, which can be difficult for some users who upgrade from the current stable version. Other blocking goals include adding the AMD64 architecture to the release, and deciding which documentation meets the Debian definitions of freedom — which means, among other things, separating documents released under the GNU Free Documentation License that do not have invariant sections and can therefore be included from those with invariant sections, which must be omitted.

Non-blocking goals for Etch include Linux Standards Base 3.1 certification and support for SE Linux, pervasive IPv6, Large File Support (LFS), and full NFSRv support.

Problems

The challenge of meeting these goals would be considerable in any circumstances, but preparations for the Etch release are further complicated by a number of problems.

For one thing, as one of the most popular distributions, Debian is struggling to find new structures to accommodate its growth. Debian is a highly politicized community at the best of times, and preparations for the Etch release seem to be occurring against a background of even greater unrest than usual. The announcement of Dunc-Tank, a group of prominent Debian developers who proposed paying the release managers as an experiment to see if further payments would help Debian reach its goals, led to a general resolution about the project’s position on the move. A second, related resolution, called for the resignation of Debian Project Leader Anthony Towns for being in a conflict of interest over his involvement with Dunc-Tank. Still another general resolution called for the removal of kernel files related to proprietary firmware from the main repository, a move that would have made Debian philosophically free at the expense of some gaps in hardware support.

All three resolutions failed to cause a major upheaval, with Debian developers voting to support the Dunc-Tank experiment and to put off any consideration of impeachment or firmware removal until after the Etch release. Now, thanks to Dunc-Tank, Barth is spending November working full-time on the release, as Lagasek did in October. Still, all three resolutions have left resentments that at times threaten to divide the Debian community, to say nothing of a legacy of unwelcome publicity. “Even way smaller organizations have their internal discussions,” Barth says. “It’s just that Debian doesn’t hide them.”

In addition, other problems exist because of other free software projects. In particular, Barth cites “the disappearance of real stable kernel releases” that has come with the elimination of development kernels, and the Mozilla Foundation’s insistence on having veto power over uses of its trademark, a debate that has caused Debian to hastily relabel its implementation of Firefox and Thunderbird, which have been hastily swapped out and replaced by IceWeasel and IceDove, the GNU Project’s substitutes.

“None of these are so bad that I would like to say, ‘if that problem wouldn’t exist, everything would be OK,'” Barth says. “It’s just that the sum of issues is adding up.”

Toward a timely release

The biggest question that hovers over the Etch release is whether, unlike previous Debian releases, it can be completed without long delays. The full freeze of Etch has already been delayed once, which suggests that it could be no more timely than its predecessors.

Barth suggests that Debian’s reputation for long release cycles is partly due to the openness with which the distribution makes decisions. “Even our first forecasts are made in public, so there are no internal dates that we could miss without anybody noticing.”

At the same time, to the extent that the reputation is valid, Barth expresses confidence that the cycle of long releases can be broken with Etch. “I don’t think there is a doubt that we had large issues with Sarge,” he says, referring to the codename of the last release of Debian. “But that is now history. With Etch, we started planning early, and we got good feedback. Most Debian maintainers have started in time to make sure their packages have an adequate quality. Also, the number of persons actively working for the release has drastically increased. So, the situation is good. Perhaps we don’t reach everything we want with Etch, but a change needs to be done step by step.”

As further proof of the improvements in process that have taken place in the preparation for Etch, Barth mentions the creation of a set of criteria that each hardware architecture has to satisfy, and the prioritization of goals by the creation of the blocking and non-blocking categories.

“I think we are in a good way,” Barth concludes. “Of course, there will be also things we need to learn from Etch. Also, the experience we have with Etch has taught us a lot about how to motivate people — and that is one of the important things there is, to give people real responsibility. Overall, I’m quite happy with how well it all works.”

Bruce Byfield is a computer journalist who writes regularly for NewsForge, Linux.com, and IT Manager’s Journal.