Benchmarkings are standardized measurement and evaluation procedures that can be used to determine the performance of EDP systems or system classes and to compare them with one another according to certain criteria. The benchmark tests for hardware performance in computer systems are well known. Benchmarks are also used to compare the performance of programming languages, including interpreters and compilers and their runtime systems on the same hardware. Then one speaks of a software benchmark. A benchmark in general is always a program that solves a problem (often from mathematics, complex). Examples are solving linear systems of equations ( LINPACK ) or the calculation of Mersenne prime numbers .
- 1970s: measurement in MIPS and FLOPS
- Early 1980s: Measurement in Dhrystones and Whetstones
- Since 1989: Standard Performance Evaluation Corporation (SPEC) application benchmark tests . It is a series of applications (e.g. neural networks, finite elements, video coding, etc.) which are divided into integer-related (SpecInt) and floating point-related (SpecFp) programs. Accordingly, the final result of the tests always forms two numbers that indicate the performance in relation to a reference machine (benchmark). With Spec2000 this is e.g. B. a Sun UltraSparcIIi / 256MB.
Hardware benchmarks traditionally only include CPU and memory access performance. Graphics card, mass storage, etc. are not tested.
The SPEC update took place in August 2006: SPECcpu 2006. A new version is being planned. A “SPEC CPU Benchmark Search Program” was started in 2008 for this purpose.
The best known benchmarks are:
Recently the focus of benchmarks has expanded. In particular, the inclusion of energy consumption as a metric has become established and is used in various SPEC (e.g. SPECpower) and TPC benchmarks.
In addition, it deals with how one can capture current trends in IT with the help of benchmarks. For example, there are currently several projects that deal with cloud benchmarking or benchmarking peer-to-peer systems.
Hardware benchmark example
Table of some Dhrystone values as an example for hardware benchmarks :
|Type||microprocessor||operating system||Compiler||Dhrystones / sec., No reg||Dhrystones / sec., Reg|
|IBM PC / XT||8088-4.77Mhz||PC / IX||cc||257||287|
|VAX 11/780||UNIX 5.2||cc||1,515||1,562|
|Compaq PIII / 450||NT4||TCC 3.1 / 286/287||71,428|
|Compaq PIII / 450||NT4||wcc 10.5 / -otexan-fp5||250,000|
|FMS AthlonXP1800||SUSE 8.2||gcc||2,016,129|
|FMS AthlonXP1800||SUSE 8.2||gcc -O3||5,050,505|
Software benchmarks are used to compare the performance of different programming systems in terms of execution speed. When creating a software benchmark, the same algorithm is implemented in different programming languages and the execution times of the programs are compared with one another. It is necessary to bring together experts for the various programming languages who can implement the algorithm as optimally as possible for the respective programming language. The Ackermann function is a classic example of such benchmarks .
Application benchmarks are used to assess the performance of computer systems when all hardware and software components interact. For this purpose, realistic application scenarios are created. The runtime of these automatically reproduced application scenarios allows a uniform comparison of different computer systems. Important parameters such as energy consumption and battery life for mobile devices can also be determined in a comparable way.
Examples of application benchmarks are
- SysMark, MobileMark, EEcoMark
- 3DMark, PCMark, PowerMark
- Most computer games have a benchmark function.
If the benchmark software used is known, there is scope for manipulation. In practice, device drivers , compilers or even instruction sets can be optimized in such a way that common benchmarks run particularly quickly.
- Frequently Asked Questions on the Linpack Benchmark and Top500 netlib.org , August 5, 2007, Accessed October 22, 2013
- Sysmark 2012 - Bapco announces new benchmark that AMD doesn't like. In: golem.de , June 22, 2011. Accessed October 10, 2013
- New 3DMark appears in: heise.de , January 31, 2013. Accessed October 10, 2013
- Tricky - Oddities in VideoLogic's Apocalypse 3D In: c't 4/97. Retrieved from heise.de on October 10, 2013
- Benchmark - Internet & eCommerce Lexicon
- The Computer Language Benchmarks Game (English)
- SPEC CPU Benchmark Search Program (English)
- List of the 500 fastest computer systems in the world based on the LINPACK benchmark (English)