Author: Bruce Byfield
QA is often neglected in the free software community. So why is Michlmayr interested enough to center his degree work on it? “I have always been a perfectionist,” he says. “While most people in the community are repeatedly stating how well we are doing, I sit back, acknowledge that we have indeed achieved a lot, but then look at all the things we could do better.
“What I’m particularly interested in,” Michlmayr says, “is the question of how you can maintain high levels of quality when the people involved are mainly volunteers. One reason I’m interested in Debian is that it’s such a large project. We have more than a thousand volunteers, all around the world, and that often leads to certain challenges.
“I’m doing some technical work to ensure quality,” Michlmayr says, referring to his activities with the Debian Quality Assurance project, “but what I’m mainly interested in is how you can manage a project so that the output is of high quality.”
The current state of QA in free software
Traditionally, Michlmayr says, free software communities have excelled in such QA issues as “reliability and code quality.” However, because of free software’s increased popularity among both business and non-technical users in recent years, projects are now being forced to pay more attention to usability, which they have traditionally neglected. As a result, QA teams are becoming increasingly common in free software projects.
Despite this trend, Michlmayr says, “There are problems everywhere, even in those successful FOSS projects we always use as good examples. When you look closer, you see that it’s actually not as good as it seems.”
This sentiment is echoed on the home page for the Debian Quality Assurance project, which states that, “at the moment, there is no real quality assurance for Debian, in a conventional meaning of that term.” “Conventional,” presumably, means what commercial developers would consider necessary. Although Debian has at least half a dozen people working regularly on QA issues, what is missing, Michlmayr believes, “is more coordination between different people and a guarantee that the work is being done and quality levels maintained. The big question is whether we, as a group of volunteers, can ensure high levels of quality. Or is that an area where support from companies would be useful? We often see that quality is very high, but in many cases, there are also problems that remain unsolved for a long time. We talk about security bugs that are fixed within two hours, but we usually don’t talk about those which remain unaddressed after weeks or months.”
Similarly, in observing Linux kernel development, Michlmayr says, “There are some sub-systems that are really well-maintained, but some have maintainers who are not very responsive. One of the ideas of QA is to maintain high levels of quality everywhere. Another problem is the release cycle, where the different needs of users and developers has become quite apparent in recent times, with a number of users saying that the kernel is not stable. Andrew Morton talked about a cycle dedicated to bug fixing, but it didn’t happen for 2.6.18. Andrew also recently mentioned that he’d like to see a dedicated bug master, but, from what I can tell, no company has stepped up to hire someone.”
With the GNU Compiler Collection (GCC), another project that Michlmayr has been monitoring in his research, QA is in an even worse state. “Most of the bugs are triaged by one single person,” he says. “The GCC developers also complain about companies that add specific features, but don’t care what happens once a feature is in. They effectively leave it up to others to fix bugs they introduced.”
“We need, as a community, to put more emphasis on quality,” Michlmayr concludes. “That includes both volunteers and companies that have an interest in the sustainability of [free software].”
Additional problems in free software
During his research, Michlmayr has identified several common problems in free software projects. The first of these is release management. “It’s hard enough to get releases out on time in a commercial environment, but it’s even harder in a distributed environment in which a large number of people are volunteers.” These problems can include lack of coordination and delays caused by people dropping out or being temporarily distracted from their work. However, for Michlmayr, the largest problem is that “many people are only interested in working on some sexy area, and other parts are neglected, even if they are important to the project as a whole.” Too often, he implies, QA is one of the areas neglected. The results can be “out of date or buggy software.”
Another problem is that few people in free software projects are dedicated testers. Even when projects have a QA team in place, Michlmayr says, “Most testers are actually users, and they often don’t get the attention they should because they might not be able to explain a problem in a coherent way,” either because of lack of language skills or because they are trying to communicate in a second language. “In many cases,” he says, “the problem is then just dismissed rather than investigated properly. Often it’s just a problem of lack of resources, but it can also be a problem of attitude.”
By far the greatest problem that Michlmayr identifies is the speed with which projects and their audiences are increasing. As the users of free software increases, developers may be pressured into increasing the speed of development. “Furthermore,” Michlmayr says, “the size of the community has increased, and this makes coordination (people) and integration (software) much harder. It’s largely a matter of complexity and of finding ways to deal with it.”
Looking for solutions
Based on his observations, Michlmayr makes two suggestions for projects that want to improve their QA. In order to deal with the increased complexity, he suggests time-based release management. “By publishing a detailed release plan,” he says, “you effectively allow individual developers to coordinate their own work, and this makes integration before the release easier. GNOME is a project that is doing really well in this area. On the other hand, the Linux kernel has struggled for the last years to find a release model and the frequent changes in their modus operandi show that they’re still working at it.”
Even more importantly, Michlmayr thinks that projects need to change their thinking about the life cycles of their software and the distinction between the cathedral and the bazaar first delineated by Eric S. Raymond. As he explains in detail in a paper online, Michlmayr questions the usual view that the cathedral style of development — in which development is closed and secretive — is typical of proprietary software development and the bazaar style — in which development is open to all — is characteristic of free software. Whatever the case in proprietary software, in free software, he says, “these two styles are really the phases of a project.
“It’s impossible to start a project in the bazaar phase because you first have to develop an initial prototype with which you can attract more developers. And how is that prototype developed? By an individual or a small team, working in isolation and with full control. That’s exactly what the cathedral is all about.
“I believe that many successful projects then make a transition to the bazaar, opening up their project and attracting more volunteers who add more functionality and fix bugs. However, many projects do not reach that stage. That is partly because starting a project and moving it to the bazaar takes two different skill sets. When you start a project, you need good design and programming skills. However, creating a bazaar requires management skills, and that is something which many programmers don’t have. I therefore believe that there are many small projects which could be much more successful.”
In other words, Michlmayr suggests that QA skills are relatively unimportant in the early stages of a free software project, but in order to become a true free software project, as well as to cope with increased popularity and complexity, the project must acquire those skills. An implication is that the cathedral-bazaar dichotomy is so fixed in the minds of people in the free software community that they may fail to see the need for this transition until it is upon them.
“Obviously, being large doesn’t necessarily imply high levels of quality,” Michlmayr says. “In fact, there are plenty of [worthwhile] projects that only have a single or few (but highly competent) developers. [But] I do think that size and popularity are related to quality, even though I am not saying that you necessarily need to be a big project to reach high levels of quality. There are still many open questions related to this [issue], but I think its definitely clear that Raymond’s “The Cathedral and the Bazaar” does not necessarily describe the typical open source project.”
Finishing the dissertation
Michlmayr currently plans to complete his Ph.D. by the summer of 2007. When the dissertation is completed, he plans to make it available on his home page, and to discuss his research at free software conferences. Meanwhile, he is adding a growing list of papers to his curriculum vitae and discussing his preliminary findings as a guest speaker. “At the moment, my talk is an introduction to general quality problems in free and open source projects,” he says. “In the future, I’m planning to give talks specifically about release management.
“You could claim that I’m a bit of a pessimist, in that I focus on the negative sides,” Michlmayr admits, summing up his general opinion of his dissertation topic. “But this [attitude] is often helpful, as it lets you identify problematic issues and then improve them. And, in that sense, I’m quite optimistic, because I think there is major room for improvement and that other people are starting to acknowledge this, as can be seen by the growing number of QA efforts in free software projects.”
Bruce Byfield is a course designer and instructor, and a computer journalist who writes regularly for NewsForge, Linux.com and IT Manager’s Journal.
