Test effort
The effort (the costs) incurred for testing is considered to be the test effort . The test effort is part of the so-called quality costs of the product to be tested. Pol, Koomen, Spillner in count the test costs as part of the effort for preventive measures, i. H. to the cost of avoiding errors. These also include the costs for the quality in software development (see "static testing" ). Likewise, the actively executed tests (= "dynamic test types" ) - from the point of view of the project goal "go live" - are part of preventing errors.
This preventive measure is offset by the cost of reactive measures and the cost of troubleshooting. In the narrower sense, they are not part of the test effort, but rather the implementation effort . Even so, if they occur during testing, they are often counted as testing effort. This also includes costs and damage that (due to undetected errors) only occur after going live. They can also have an impact outside the company (with customers, in the environment, etc.) and (in dramatic cases) even endanger the very existence of the company.
The following rule of thumb applies: "The later errors are discovered, the more time-consuming it is to correct them ", which also leads to the reverse conclusion: "Quality must be implemented (throughout the course of the project), not 'tested' .
Factors that influence the test effort are: maturity level of the development process, quality and testability of the test objects , the test infrastructure, employee qualifications, quality goals and the test strategy .
This article describes the topic of "test effort" mainly in connection with software tests .
Approaches to estimating the test effort
Analyzing all factors is difficult because most of the factors influence one another. The following approaches can be used for estimation: metric-based (in English: top-down estimation) and expert-based (in English: bottom-up estimation) approaches.
The metrics-based techniques (as controlling instruments) are formula-based and are usually specified relative to the development effort: these include the function point method (FPA) and test point analysis (TPA). Corresponding information (e.g. effort per project phase or per test object, plan / actual deviation, number of errors found, number of required retests, etc.) can be determined and displayed in order to prove the quality and efficiency of the test process .
The expert-based techniques are based on detailed information and often involve experts. The following techniques include: Work Breakdown Structure (in English: Work Breakdown Structure (WBS)) and Wideband Delphi (WBD).
Test effort from the literature
The test effort (in software projects) is between 20% and 70% of the total costs, depending on the author. Pol, Koomen and Spillner put it at 30% to 40% of the total budget. The spread is determined by project-specific circumstances.
If the test effort for individual phases of the test process is considered, it is distributed differently: With around 40% each, the test specification and the test execution are often involved. The rest would be used for planning, evaluation and completion of the test.
According to the "Principles of Software Testing " ( ISTQB ), Principle 2, full testing is not possible (*) ; because testing can indeed show the presence of errors, but not their absence (Principle 1) - (*) because all influencing variables, in all possible combinations, are practically impossible to test (except for trivial functions).
Therefore, the test activities must be strategically planned in order to select the test intensity, thus also the test effort, "appropriately". According to Juran, referenced in, the test effort should be so high that the total costs (test effort plus potential failure costs) are as low as possible. As the diagram shows schematically, if the test intensity is too low, disproportionately high follow-up costs arise, if the test intensity is too high (the costs of which are linear), the test costs are too high in relation to the expected reduction in follow-up costs (from still existing defects).
According to ISTQB, the amount of test effort should be such that the tests provide enough information to be able to decide on the release (of the software) .
literature
- Andreas Spillner, Tilo Linz: Basic knowledge of software testing - Basic knowledge of software testing: Training and further education to become a Certified Tester: Foundation Level according to the ISTQB standard. 3rd, revised and updated edition, dpunkt.verlag GmbH, Heidelberg 2005, ISBN 3-89864-358-1 .
- Martin Pol, Tim Koomen, Andreas Spillner: Management and optimization of the test process , ISBN 978-3-89864-951-3
- Erik van Veenendaal (editor and co-author): The Testing Practitioner. 3. Edition. UTN Publishers, CN Den Bosch, Netherlands 2005, ISBN 90-72194-65-9 .
- German Testing Board eV & Swiss Testing Board (Ed.): Certified Tester - Foundation Level Syllabus. German-language edition, Möhrendorf 2005, International Software Testing Qualifications Board (ISTQB), ( PDF; 0.266 MB ).
- Andreas Spillner, Tilo Linz, Thomas Roßner, Mario Winter: Practical knowledge of software testing - test management: Training and further education to become a Certified Tester: Advanced Level according to the ISTQB standard. 1st edition. dpunkt.verlag GmbH, Heidelberg 2006, ISBN 3-89864-275-5 .
- Harry Sneed , Manfred Baumgartner, Richard Seidl : The System Test - From Requirements to Quality Proof . 3. Edition. Carl Hanser Verlag , 2011, ISBN 978-3-446-42692-4 .
- Harry Sneed , Richard Seidl, Manfred Baumgartner: Software in numbers - the measurement of applications . 1st edition. Carl Hanser Verlag , 2010, ISBN 978-3-446-42175-2 .