Benchmark Design

Purpose

A benchmark’s job is to measure real, steady-state performance — several common mistakes instead produce a number that looks precise but measures something else entirely.

Common mistakes and fixes

// Vulnerable: no warm-up, single run, unused result
function badBenchmark(fn) {
  const start = performance.now();
  fn(); // may be eliminated if the result is never used
  return performance.now() - start;
}

// Better: warm-up iterations, many measured iterations, result observed
function benchmark(fn, warmupRuns = 100, iterations = 1000) {
  for (let i = 0; i < warmupRuns; i++) fn();

  let sink = 0;
  const start = performance.now();
  for (let i = 0; i < iterations; i++) sink += fn();
  const elapsed = performance.now() - start;

  console.log(sink); // discourage the engine from eliminating the loop as dead code
  return elapsed / iterations;
}

Behavior notes

• Warm-up runs let the engine reach steady-state optimized execution before the timed portion begins — a cold run measures JIT compilation overhead, not real performance.
• Dead-code elimination: an engine that can prove a computed result is never observed may legally skip the work — using the result (accumulating it, logging it) defends against this.
• Representative input: a benchmark on tiny, uniform, or otherwise unrealistic data can favor an approach that loses on real production data.
• Variance: reporting a single number without a sense of spread across many runs risks mistaking noise for a real difference — running the whole benchmark multiple times and reporting a range (or a statistical summary) is more trustworthy than one number.

Common mistakes

• Comparing two benchmarks run under different conditions (different warm-up, different iteration counts, different machine load) as if the numbers were directly comparable.
• Treating a lab benchmark result as a final verdict without confirming it matters against field data — recall lab versus field data from Performance and Diagnostics.

Runtime notes

Applies to benchmarking JavaScript in any engine — the specific pitfalls (JIT warm-up, dead-code elimination) are consequences of how modern engines optimize, not unique to one runtime.