Test coverage

from Wikipedia, the free encyclopedia

As test coverage is the ratio of actual statements made a test against the theoretically possible treffbaren statements or the amount of the desired treffbaren statements. The test coverage plays a major role as a metric for quality assurance and for increasing quality, especially in mechanical engineering and software technology .

In practice, the test coverage is influenced by various criteria. Test coverage can be improved by increasing the number of measurements , samples, and test cases . In practice, however, test coverage is limited by the costs associated with each test.

Test coverage in mechanical engineering

Depending on the type, effort and benefit of the tests, some tests are carried out on a random basis and others in full. A test that can be carried out easily and automatically is carried out on every product, since its costs only slightly increase production costs. However, a crash test with a vehicle is of course only carried out with random samples, since the tested product then becomes unusable.

For 1000 vehicles produced this could e.g. B. mean that particularly complex tests and crash tests are carried out only with a single vehicle, while less complex tests are carried out with a larger number or even all vehicles.

Necessary but complex tests are varied in their frequency and thus the test coverage. If a test delivers predominantly or exclusively positive results, its number is reduced. If a test delivers negative results, it is used more frequently until the changes in production lead to a significant increase in positive results and thus to higher product quality again.

The cost-benefit calculation of such tests is carried out with the help of stochastics . Is z. For example, if a test is only carried out on 5 out of 1000 vehicles to determine whether the electric window lifters are working properly, the statistical relevance and the probability of an incorrect assessment can be calculated based on the test result with the aid of stochastics .

Test coverage in software technology

For test coverage in software technology (test coverage or code coverage), stochastics play practically no role, since computer programs are not series-produced individual products in which tests are carried out with random samples. Instead, tests are defined on the basis of the specification (properties of the interface) or the internal structure of a software unit to be tested.

In software technology, the test coverage is determined for different areas of the software. These include the coverage of the code, the data or the subject matter. In order to achieve the highest possible test coverage, different test cases must be written depending on the area to be covered. It is not possible to specify a measure for test coverage in software tests for all control flow-oriented test procedures , since it is often not possible to determine the number of possible test cases for real problems.

A complete test coverage of the subject matter is an exception, because the number of possible test cases becomes extremely large very quickly (due to combinatorial explosion ). A full function test for a simple function that receives two 16-bit values ​​as arguments would already mean 2 ^ (16 + 16), i.e. approx. 4 billion test cases, to fully test the specification. Instead, you limit yourself to a selection of tests that appear reasonable for borderline cases. Example: A root function for rational numbers could e.g. B. with all elements of the set {-10; -1; -0.0000001; 0; 0.0000001; 1; 2; 3; 4; 5.25; 9; 10000} to be tested. A reasonable selection of test cases for adequate test coverage is generally valid of various types of valid arguments, and additional boundary elements for components with robustness requirements (arguments that are still valid and arguments that are currently invalid). It has also proven successful to include the argument that triggered the error in the set of test elements in the event of an error.

A largely complete test coverage of the code, on the other hand, is often the goal for module tests or component tests : Due to high test coverage for 'small' functional units, the number of total test cases required results only from the addition of these test cases and not from the combinatorics of the larger functionality. But here too this goal cannot be 100% achieved in most cases because of residual risks (unexpected 'remote effect' of errors) and also for cost-benefit reasons.

Since a high code coverage can also be achieved with tests that do not check anything, the code coverage has only limited significance for the quality of the tests. To ensure the quality of tests, mutation tests are usually used. For this purpose, errors or mutations are automatically built into the code to be tested and the tests are then carried out. If tests then fail that previously worked, the mutation was recognized. The quality of the tests can be derived from the percentage of mutations recognized. The test coverage calculated in this way therefore determines the coverage of the code which, in the event of a mutation, led to a failure of one of the tests.

Norms

The code coverage is also used by various international standards for recommendations and minimum requirements:

  • IEEE 1008 (" Software Unit Testing "): 100% statement coverage as a requirement for completing the module tests, 100% branch coverage for modules with critical code or insufficient specification.
  • ISO 26262 (" Road vehicles - Functional safety "): Depending on the criticality, appropriate test coverage for module tests and integration tests is recommended or strongly recommended.
  • IEC 61508 (" Functional Safety of Electrical / Electronic / Programmable Electronic Safety-Related Systems "): Recommends or strongly recommends a 100% coverage of inputs, statements, branches, conditions depending on the safety requirement level , for example 100% for the lowest level Test coverage of the inputs strongly recommended that of the statements, branches and conditions recommended.
  • DO-178B (" Software Considerations in Airborne Systems and Equipment Certification "): Requires 100% test coverage of statements, decisions, changed decisions depending on the impact of a system error. For example, if there is a risk of injury to passengers, 100% test coverage of the statements is required.

Code coverage measurement tools

Mutation testing tools

  • PIT - Java

Web links

See also