“With enough eyes, all bugs are shallow:” Modelling massive diversity in testing
Open source software, diversitySummary
The quotation in the title (it comes from Eric Raymond) neatly encapsulates a widely claimed reason for the claimed efficacy of Open Source software: that the exposure of the code to very many people’s scrutiny allows even quite obscure bugs to be identified (and thus removed). In DIRC we are particularly interested in whether – and if so, how – such a process delivers greater dependability than other ways of developing software.
The intuition behind the claim comes, of course, from the belief that in situations like this diversity is a good thing. The different ‘eyes’ will have different perspectives on the problem, and it might be expected that what is missed by some people will be picked up by others. Compared with other research on diversity in DIRC (e.g. on multi-version fault tolerance), the distinguishing feature here is the massive diversity represented by potentially hundreds or thousands of different users in the OS community.
Our approach has concentrated on reliability, and our aim has been to give formal expression, via probabilistic modeling, to some of the conjectured laws that support the claims made. Specifically, we have concentrated on modelling the reliability growth of software while in use after release. In this context we address a single conjecture of interest: that the diversity of users involved in software fault reporting in an OS environment may give it an advantage, from a reliability growth viewpoint, over comparable software that lacks such diversity.
The diversity we examine here is between the usage profiles of the different users in the population. By usage profile we mean the set of probabilities of each possible demand being chosen by that user. These profiles will differ from one user to another, and claims for efficacy of the OS approach concern how different are the profiles within a population of users: the hypothesis is that the more difference there is here – i.e. the more diverse the usage across the population – the better.
Our headline result in this work is that, broadly, diversity is indeed a good thing: all things being equal, it is better for users to have diverse demand profiles than for them to have the same profile. Any user should prefer the previous exposure of the software to have been diverse rather than uniform, because diversity gives them higher reliability. Somewhat surprisingly, this is true no matter what the profile of the user actually is.
Bev Littlewood (City)
|Page Maintainer: firstname.lastname@example.org||Credits||Project Members only||Last Modified: 6 July, 2006|