Using moments to analyse software product risk and quantify impact of defects and field data (16-Dec-2009)
This article proposes a quantitative method for measuring software product quality. The quantitative result allows comparisons between different releases of a product as well enabling a user profile specific quality measurement. The method accounts not only for defects discovered during testing, but also incorporates field data from user reported problems. The results provide a useful measure of testing effectiveness and enabling testing prioritisation as well as indicating the impact of known problems for different user profiles.
Using moments to analyse software product risk and quantify impact of defects and field data
There are many aspects to a software system and several 'industry standard' ways to measure these. The notion of software quality is dependent on a given solution context, and - as a result - it is very difficult to quantify software product quality universally. There are a number of reasons for this:
The total number of problems (defects) in the code are unknown
1.
The resulting impact of a given defect will vary significantly for different users
2.
The widespread usage of defect severity indicators is flawed: A large number of
3.
low severity defects may have an impact on overall product quality which is just as significant as a small number of high severity defects.
It is impractical to create a set of tests that will cover every aspect of the software
4.
system.
When testing a large complex system, it is unlikely that all planned testing will be
5.
completed within the time allocated: There will always be some partially complete and unattempted test cases, and it will be very difficult to assess the impact of this situation.
It is generally assumed that all test cases are equal in terms of a quality
6.
assessment. So if test case A fails and B passes, how can we determine the overall quality of the system?
Conventional methods of measurement mean that a test phase will only be
7.
considered successful if it completes in good time and having raised a large volume of defects. This is not a quality based measurement.
The most common solution is to measure the fraction of code that is exercised successfully by test cases. This approach has two major drawbacks:
It is expensive to collect this information for a large software product.
1.
The data provides no indication as to the relative importance of different
2.
components of the software.
As a result, it is possible to deliver a test phase which will cover a large percentage of code (indicating a low risk, high quality deliverable), whilst missing a small number of code paths which are of vital importance i.e. delivering a potentially high risk, low quality deliverable. Furthermore, it is theoretically possible to have a high severity defect escape the attention of the project team if it does not directly impact a test case.
Another approach to measuring product risk is to analyse the total number of known product defects. The disadvantage of this approach is that the reduced risk from known test results (whatever the outcome) is not reflected, and - as a result - it is not possible to compare disparate releases of a product accurately.
As can be seen, existing solutions do not provide a reliable basis for quantitative assessment of software product quality. Furthermore, test tracking processes tend to compound the problem and 'overlook' product quality indicators; they...
Find More Documents Like This
Download this Disclosure
Click here if you want to sign-in now.
