to give a common statement about the Forrester report entitled “Is Linux
more Secure than Windows?”. Despite the report’s claim to incorporate a
qualitative assessment of vendor reactions to serious vulnerabilities, it
treats all vulnerabilities as equal, regardless of their risk to users.
As a result, the conclusions drawn by Forrester have extremely limited
real-world value for customers assessing the practical issue of how
quickly serious vulnerabilities get fixed.
The security response teams of GNU/Linux distributors Debian,
Mandrakesoft, Red Hat and SUSE have assisted Forrester in gathering and
correcting data about vulnerabilities in their products. The gathered
data was used at Forrester for a report that became titled “Is Linux more
secure than Windows?”. While the vulnerability data regarding GNU/Linux
which is the basis for the report is considered to be sufficiently
accurate and useful, Debian, Mandrakesoft, Red Hat and SUSE, from now on
referred to as “We”, are concerned about the correctness of the
conclusions made in the report.
We believe that it is in the interest of our usership and the Free
Software community to respond to the Forrester report in the form of a
common statement:
We were approached by Forrester in February 2004 to help them refine their
raw data. Forrester collected data about the vulnerabilities that
affected GNU/Linux during a one year period (June 2002 – May 2003) and
looked at how many days it took us to provide corrections to our users.
Significant efforts have been put in not only making sure that the
underlying dataset for the vulnerabilities was correct, but also to
articulate the special technical and organizational care taken in the
response processes in the professional Free Software security field. This
expertise is greatly appreciated by our usership since it adds a high
value to our products, but we see that most of this value has been ignored
in the methods used for the analysis of the vulnerability data, leading to
erroneous conclusions.
Our Security Response Teams and security specialized organizations of
respectable reputation (such as the CERT/DHS, BSI, NIST, NISCC) exchange
information about vulnerabilities and cooperate on the measures and
procedures to react to them. Each vulnerability gets individually
investigated and evaluated; the severity of the vulnerability is then
determined by each of the individual teams based on the risk and impact as
well as other, mostly technical, properties of the weakness and the
software affected. This severity is then used to determine the priority
at which a fix for a vulnerability is being worked on weighed against
other vulnerabilities in our current queues. Our users will know that for
critical flaws we can respond within hours. This prioritization means
that lower severity issues will often be delayed to let the more important
issues get resolved first.
Even though the Forrester report claims so, it does not make that
distinction when it measures the time elapsed between the public knowledge
of a security flaw and the availiability of a vendor’s fix. For each
vendor the report gives just a simple average, the “All/Distribution days
of risk”, which gives an inconclusive picture of the reality that users
experience. The average erroneously treats all vulnerabilities as equal,
regardless of the risk they pose. Not all vulnerabilities have an equal
impact on all users. An attempt has been made to allocate a severity to
vulnerabilities using data from a third party, however the classification
of “high-severity” vulnerabilities is not sufficient: The mere
announcement of a vulnerability by a particular security organization does
not necessarily make the vulnerability severe – similarly, the ability to
exploit a weakness over the network (remote) is often irrelevant to the
vulnerability’s severity.
We believe the report does not treat vendors of Free Software and the
single closed source vendor in the same way. Free Software is known for
its variety and its freedom of choice amongst the standards it defines.
Multiple implementations of these standards are typically offered for both
desktop and server use, which gives users the freedom to select software
based on their own criteria rather than those of the vendor. The
openness, transparency and traceability of the source code is added value
in addition to the larger variety of software packages available.
Finally, the claim that one software vendor had fixed 100% of their flaws
during the period of the report should be incentive for a closer
investigation of the conclusions the report presents.
signed,
Noah Meyerhans, Debian
Vincent Danen, Mandrakesoft
Mark J Cox, Red Hat
Roman Drahtmüller, SUSE
Additional Information:
Javier Fernández-Sanguino Peña composed a survey in 2001[*] and discovered
that it has taken the Debian security team an average of 35 days to fix
vulnerbilities posted to the Bugtraq list. However, over 50% of the
vulnerabilities where fixed in a 10-days time frame, and over 15% of them
where fixed the same day the advisory was released! For this analysis,
all vulnerabilities were treated the same, though.
He has rerun the survey based on vulnerabilities discovered between June
1st 2002 and May 31st 2003 and found out that the median value of delays
between the disclosure and releasing an advisory including a correction
was 10 days (average is 13.5 days). Again, for this analysis advisories
were not classified with different priorities.
* http://lists.debian.org/debian-security-0112/msg00257.html
http://people.debian.org/~jfs/debconf/security/data/
Category:
- Linux
