Wednesday, July 11, 2007

Open Source versus Commercial Software Quality

I was reading through questions asked on LinkedIn and came across this question:

Can we consider that for a given application one will find fewer bugs in open source than in a branded proprietary application? (Link)

A fascinating question with a lot of opinion-based answers. I don't think there are many software products out there that don't incorporate open source in some way, so it seems like an important question to understand.

OSS proponents often use the argument that the more eyes on a piece of code, the less bugs in that piece of code (usually stated as Linus' Law, "Given enough eyeballs, all bugs are shallow."). In fact, this is really considered to be a self-evident truth in that community.

Personally, I don't like "self-evident" truths. I also don't like subjective answers like "OSS programmers are coding what they love, so they make better code" or "OSS projects have many, many programmers and, according to Brook's law, that's where bugs cluster." Give me real numbers, not theoretical possibilities.

The reality is that there is little hard data on the subject. A couple of items I've found:
  • The best data I could find on the comparison of open source and closed source is in this paper published by Stamelos, et al, from Aristotle University of Thessaloniki. It isn't a true side-by-side comparison, but rather a comparison of open source code analysis to the tool's industry "standard" recomendations. The opinion: open source software is better than an OSS detractor might expect, but it still isn't up to the industrial "standard" of the tool. The conclusion: more data is still needed.
  • Michlmayr, Hunt, and Probert of the University of Cambridge discuss open source processes and quality management and their effect on OSS quality in their paper. The conclusion is that OSS presents some significant challenges around ensuring the quality of software produced. Unless an OSS project has a specific focus on quality and process, it likely will suffer in that regard. Once again, this is a small sample set and more data is needed.
My experience with software would lead me to believe the results of the Michlmayr paper. It is difficult to get people to volunteer to focus on the corner cases and the boundary conditions; that isn't "fun." In addition, to polish a product takes a good amount of governance to make sure that fixes aren't introducing more risk than what they are fixing. This also seems difficult to achieve in a volunteer army. Not impossible, just difficult.

The question of quality is but one question is determining whether to use open source or closed source software. Other factors that need to be thought about include support costs (both in terms of finances and time), opportunity costs (or, potentially, wins), licensing costs, and training costs. It is only when all of these items (there are probably others, too) are calculated that you can begin to see the value of open source versus closed source to your project.

I would love to read more definitive research on this topic. If you know something I don't, let me know!

No comments: