In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it.[1]

The term benchmark is also commonly utilized for the purposes of elaborately designed benchmarking programs themselves.

Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems (DBMS).

Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures. Benchmarking as a part of continuous integration is called Continuous Benchmarking.[2]

Purpose

As computer architecture advanced, it became more difficult to compare the performance of various computer systems simply by looking at their specifications. Therefore, tests were developed that allowed comparison of different architectures. For example, Pentium 4 processors generally operated at a higher clock frequency than Athlon XP or PowerPC processors, which did not necessarily translate to more computational power; a processor with a slower clock frequency might perform as well as or even better than a processor operating at a higher frequency. See BogoMips and the megahertz myth.

Benchmarks are designed to mimic a particular type of workload on a component or system. Synthetic benchmarks do this by specially created programs that impose the workload on the component. Application benchmarks run real-world programs on the system. While application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks are useful for testing individual components, like a hard disk or networking device.

Benchmarks are particularly important in CPU design, giving processor architects the ability to measure and make tradeoffs in microarchitectural decisions. For example, if a benchmark extracts the key algorithms of an application, it will contain the performance-sensitive aspects of that application. Running this much smaller snippet on a cycle-accurate simulator can give clues on how to improve performance.

Since around 1995, the SPEC collection of benchmarks have becomer widely used.

Computer companies are known to tune their systems to improve performance on benchmark tests that are not representative of real world usage. Benchmarks have been improved to more closely mimic real world usage, so that any tuning carries over to application performance.

Software vendors also use benchmarks in their marketing, such as the "benchmark wars" between rival relational database makers in the 1980s and 1990s. Companies commonly report only those benchmarks (or aspects of benchmarks) that show their products in the best light. They also have been known to mis-represent the significance of benchmarks, again to show their products in the best possible light.[3][4]

When performance is critical, the only benchmark that matters is the target environment's application suite.

Functionality

Features of benchmarking software may include recording/exporting the course of performance to a spreadsheet file, visualization such as drawing line graphs or color-coded tiles, and pausing the process to be able to resume without having to start over. Software can have additional features specific to its purpose, for example, disk benchmarking software may be able to optionally start measuring the disk speed within a specified range of the disk rather than the full disk, measure random access reading speed and latency, have a "quick scan" feature which measures the speed through samples of specified intervals and sizes, and allow specifying a data block size, meaning the number of requested bytes per read request.[5]

Challenges

Benchmarking is not easy and often involves several iterative rounds in order to arrive at predictable, useful conclusions. Interpretation of benchmarking data is also extraordinarily difficult. Here is a partial list of common challenges:

Benchmarking principles

There are seven vital characteristics for benchmarks.[8] These key properties are:

  1. Relevance: Benchmarks should measure relatively vital features.
  2. Representativeness: Benchmark performance metrics should be broadly accepted by industry and academia.
  3. Equity: All systems should be fairly compared.
  4. Repeatability: Benchmark results can be verified.
  5. Cost-effectiveness: Benchmark tests are economical.
  6. Scalability: Benchmark tests should work across systems possessing a range of resources from low to high.
  7. Transparency: Benchmark metrics should be easy to understand.

Types of benchmark

  1. Real program
  2. Component Benchmark / Microbenchmark
    • core routine consists of a relatively small and specific piece of code.
    • measure performance of a computer's basic components[9]
    • may be used for automatic detection of computer's hardware parameters like number of registers, cache size, memory latency, etc.
  3. Kernel
    • contains key codes
    • normally abstracted from actual program
    • popular kernel: Livermore loop
    • linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language)
    • results are represented in Mflop/s.
  4. Synthetic Benchmark
    • Procedure for programming synthetic benchmark:
      • take statistics of all types of operations from many application programs
      • get proportion of each operation
      • write program based on the proportion above
    • Types of Synthetic Benchmark are:
      • Whetstone
      • Dhrystone
    • These were the first general purpose industry standard computer benchmarks. They do not necessarily obtain high scores on modern pipelined computers.
  5. I/O benchmarks
  6. Database benchmarks
    • measure the throughput and response times of database management systems (DBMS)
  7. Parallel benchmarks
    • used on machines with multiple cores and/or processors, or systems consisting of multiple machines

Common benchmarks

Industry standard (audited and verifiable)

Open source benchmarks

Microsoft Windows benchmarks

Others

See also

References

  1. Fleming, Philip J.; Wallace, John J. (1986-03-01). "How not to lie with statistics: the correct way to summarize benchmark results". Communications of the ACM. 29 (3): 218–221. doi:10.1145/5666.5673. ISSN 0001-0782. S2CID 1047380.
  2. Grambow, Martin; Lehmann, Fabian; Bermbach, David (2019). "Continuous Benchmarking: Using System Benchmarking in Build Pipelines". 2019 IEEE International Conference on Cloud Engineering (IC2E). pp. 241–246. doi:10.1109/IC2E.2019.00039. ISBN 978-1-7281-0218-4.
  3. "RDBMS Workshop: Informix" (PDF) (Interview). Interviewed by Luanne Johnson. Computer History Museum. 2007-06-12.
  4. "RDBMS Workshop: Ingres and Sybase" (PDF) (Interview). Interviewed by Doug Jerger. Computer History Museum. 2007-06-13.
  5. Software: HDDScan, GNOME Disks
  6. Krazit, Tom (2003). "NVidia's Benchmark Tactics Reassessed". IDG News. Archived from the original on 2011-06-06.
  7. Castor, Kevin (2006). "Hardware Testing and Benchmarking Methodology". Archived from the original on 2008-02-05.
  8. Dai, Wei; Berleant, Daniel (December 12–14, 2019). "Benchmarking Contemporary Deep Learning Hardware and Frameworks: a Survey of Qualitative Metrics" (PDF). 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI). Los Angeles, CA, USA: IEEE. pp. 148–155. arXiv:1907.03626. doi:10.1109/CogMI48466.2019.00029.
  9. Ehliar, Andreas; Liu, Dake. "Benchmarking network processors" (PDF). {{cite journal}}: Cite journal requires |journal= (help)
  10. Transaction Processing Performance Council (February 1998). "History and Overview of the TPC". TPC. Transaction Processing Performance Council.

Further reading

Wikimedia Commons has media related to Benchmarks (computing).