Introduction

Measuring algorithm runtime in Python is easy to start and easy to do badly. A single timing often reflects background noise, cache state, or unlucky input more than the algorithm itself. Good measurements come from using the right clock, running repeated trials, and keeping the comparison conditions stable.

Use the Right Timer for Elapsed Time

For wall-clock benchmarking, time.perf_counter() is the standard choice. It is monotonic and has better resolution than older timing functions.

1import time
2
3
4def insertion_sort(values):
5    items = values[:]
6    for i in range(1, len(items)):
7        key = items[i]
8        j = i - 1
9        while j >= 0 and items[j] > key:
10            items[j + 1] = items[j]
11            j -= 1
12        items[j + 1] = key
13    return items
14
15
16data = list(range(3000, 0, -1))
17start = time.perf_counter()
18insertion_sort(data)
19elapsed = time.perf_counter() - start
20print(f"elapsed: {elapsed:.6f} seconds")

This is fine for quick checks, but one run is not enough to support any serious conclusion.

Repeat the Measurement and Report Robust Statistics

Operating system scheduling, Python memory management, and background processes introduce noise. Run the same benchmark many times and report at least the median.

1import statistics
2import time
3
4
5def benchmark(fn, values, repeats=25):
6    samples = []
7    for _ in range(repeats):
8        start = time.perf_counter()
9        fn(values)
10        samples.append(time.perf_counter() - start)
11    return {
12        "median": statistics.median(samples),
13        "mean": statistics.mean(samples),
14        "min": min(samples),
15        "max": max(samples),
16    }
17
18result = benchmark(insertion_sort, data)
19print(result)

The median is often more informative than the mean because it is less affected by occasional slow runs.

Use timeit for Small, Focused Benchmarks

When the code under test is short, timeit is a better tool than hand-written loops. It reduces measurement boilerplate and is designed for repeat execution.

1import timeit
2
3seconds = timeit.timeit(
4    stmt="sorted(data)",
5    setup="data = list(range(5000, 0, -1))",
6    number=200,
7)
8print("total:", seconds)
9print("per run:", seconds / 200)

timeit is especially useful for comparing small expressions or verifying whether a local refactor changed a hot code path.

Compare Algorithms on Equivalent Inputs

A fair comparison means each algorithm sees the same logical input. That matters most for in-place algorithms, because the first run can mutate the data and make later runs look faster.

1import time
2
3
4def benchmark_once(fn, base_values):
5    start = time.perf_counter()
6    fn(base_values[:])
7    return time.perf_counter() - start
8
9base = list(range(2000, 0, -1))
10print("insertion_sort:", benchmark_once(insertion_sort, base))
11print("built-in sorted:", benchmark_once(sorted, base))

Copying the input before each run avoids accidental bias.

Measure Across Input Sizes and Shapes

Runtime analysis is not only about one absolute number. The interesting question is how the runtime grows as the input grows and how the algorithm reacts to different data distributions.

1import random
2
3for size in [200, 400, 800, 1600]:
4    reversed_data = list(range(size, 0, -1))
5    random_data = random.sample(range(size), size)
6    print(size, benchmark_once(insertion_sort, reversed_data), benchmark_once(insertion_sort, random_data))

An algorithm may look acceptable on nearly sorted input and collapse on reversed or random input. If production data has a specific shape, benchmark that shape explicitly.

Use Profiling When the Benchmark Is Too Coarse

Timing only tells you how long the whole operation took. It does not tell you where the time went. If you need to optimize real code, pair timing with profiling.

1import cProfile
2import pstats
3
4profiler = cProfile.Profile()
5profiler.enable()
6insertion_sort(data)
7profiler.disable()
8
9pstats.Stats(profiler).sort_stats("cumtime").print_stats(10)

This helps you avoid spending time optimizing code that is not actually the bottleneck.

Control the Environment Enough to Trust the Result

For serious measurements, reduce avoidable noise. Run on a consistent power profile, close heavy applications, avoid timing through a debugger, and record the Python version. If you are comparing runs over time in CI or across teammates, note the CPU and operating system as well.

Do not chase false precision. Reporting many decimal places does not make a noisy benchmark more reliable. Good methodology matters more than fine-grained formatting.

Common Pitfalls

A common mistake is reporting one benchmark result as if it were definitive. Repeated trials are required if you want a trustworthy number.

Another issue is benchmarking unrealistic toy inputs. If production data is mostly sorted, mostly random, or much larger than the test input, your benchmark should reflect that reality.

Developers also accidentally measure setup work, logging, or printing instead of the algorithm itself. Keep the timed section focused on the computation you actually care about.

Finally, many comparisons are invalid because one function mutates the input and the next function receives an already-processed version. Always copy shared benchmark inputs when mutation is possible.

Summary

• Use time.perf_counter() for elapsed-time measurements.
• Run repeated trials and report median or similar robust statistics.
• Use timeit for short microbenchmarks and cProfile for hotspot analysis.
• Compare algorithms on equivalent inputs and across realistic sizes.
• Treat benchmark setup as part of the engineering work, not as an afterthought.