November 21, 2009, 12:05 pm
How is it we can know the weight of a person’s soul* but not be able to measure the success of a piece of individual FOSS software with the truly compelling metrics needed to satisfy and influence enterprise adopters and governments.
A recent report (26th August 2009) by IDC indicates that worldwide revenue from Linux operating systems software grew by 23.4% from 2007 to 2008 and by projecting a 16.9% CAGR (compound annual growth rate) over the period 2008-13 the market size will cross $1 billion for the first time in 2012, growing to $1.2 billion by 2013 (www.idc.com).
Whilst this is a useful figure it doesn’t enable us to generate a market share figure for an individual application. Yet metrics are valuable in understanding where a particular product sits in its marketplace, its rate of adoption, its market share, growth or decline, where it sits in a Gartner Magic Quadrant and so on.
OSS2009 in Skovde, Sweden had many interesting papers (and stirring presentations from fellow FossBazaar blogger Stormy Peters and Brian Behlendorf!) A useful attempt to measure the success of a distribution/application came from a paper by Andrea Wiggins, James Howison and Kevin Crowston and looked to quantify success via download metrics to indicate the user base out in the real world. This was a valuable approach to measures and showed how difficult it is to get accurate sizing – something that is likely to hinder enterprise adoption where RFPs and ITTs are likely to request robust numbers on installed base and market share even if only as checklist items.
The quantity of downloads can be in the gazillions (peta, tera territory) yet because the figures don’t directly represent the installed base, only an indicative proxy, further work is required to assess market penetration. SourceForge and other sources can supply an indication but may not include other repositories and means of distribution and hence can be almost anecdotal.
Measuring the size of the of the community involved with a piece of software is also interesting and illuminating. The number of active committers or quality of documentation may be helpful metrics but even these are only indicative of software success. Perhaps a social networking tool is the way to best visualise this.
So although valuable work has been done we have a way to go to find some tangible metrics. So, how do you measure success – by weight, dollars or active user base?
*BTW, the reference to the weight of a person’s soul being 21 grams (and the film of that name) referred to a piece of research done in 1907 with somewhat questionable methodology and sample size – so in truth nobody knows that for a fact either.
Sources:
http://www.idc.com/getdoc.jsp;sessionId=LWSGLGWP4ZRFKCQJAFICFFAKBEAUMIWD…
Heart Beat: Measuring Active User Base and Potential User Interest in FLOSS Projects. Andrea Wiggins, James Howison and Kevin Crowston. (University of Syracuse) http://floss.syr.edu/StudyP/
