In the realm of software optimization, a frequently encountered yet perplexing phenomenon is when a code runs faster despite employing more threads than available CPU cores. At first glance, this counterintuitive outcome raises questions on how resources are managed and utilized, and challenges the conventional wisdom of sticking to one thread per core for optimal performance. In this article, we delve into the underlying factors that contribute to this scenario, examining technical aspects and concepts that make it possible for more threads to lead to better performance.

Understanding Threads and Cores

Before diving into the core of the topic, it’s critical to understand the relationship between threads and cores. A CPU core is a physical processing unit, while a thread is the smallest sequence of programmed instructions that can be managed independently by a scheduler. Modern CPUs typically support a feature known as Simultaneous Multithreading (SMT), which allows a single core to execute multiple threads concurrently. This is commonly referred to as Hyper-Threading in Intel CPUs.

Contributing Factors to Increased Performance

1. Parallelism vs. Concurrent Execution

In systems where the workload can be broken down into discrete independent tasks, increasing the number of threads can improve throughput even when the thread count exceeds core count. Here's why:

• I/O-Bound Workloads: If a significant portion of a task involves waiting for I/O operations (disk access, network communication), additional threads can make use of idle CPU time more efficiently. While one thread is waiting for I/O operations, another thread can utilize the CPU to perform computation.
• Task Switching Overhead: The cost of switching between threads is reduced, especially if the threads are lightweight. When threads exceed the number of cores, the operating system effectively schedules them based on priority and availability, utilizing CPU cycles that might otherwise be underused.

2. Improved Cache Utilization

The performance gain can also be attributed to enhanced cache utilization:

• Temporal Locality: More threads can lead to a higher chance of utilizing cached data if the data requested by different threads is stored in memory close in time.
• Data Pre-fetching: Modern processors can pre-fetch data into the cache based on predictable access patterns, improving data availability across threads.

3. Overcoming Blocking Situations

In many applications, especially those involving shared resources or locking mechanisms, threads might get blocked frequently:

• Lock Contention and Deadlocks: More threads can minimize the impact of other threads being blocked by locks, allowing threads to continue executing unrelated tasks while waiting for resources to become available.

4. JIT Compiler Optimizations

In environments like Java or DotNet, just-in-time (JIT) compilation can significantly optimize multi-threaded applications at runtime:

• Adaptive Optimization: The JIT compiler may optimize the code paths taken by different threads based on how frequently they are executed, thus better utilizing CPU capabilities.

5. Real-time Systems Consideration

In real-time systems where minimizing response time is crucial, having more threads than cores can provide buffer latency. This is particularly beneficial in handling high-throughput concurrent requests with low response time demands.

Example Scenario

Consider an application that processes data from multiple sensors in a factory. Each sensor streams data independently, and the data is processed in parallel to generate real-time analytics. In this case, although the CPU has 8 cores, launching 16 threads can drastically reduce the time individual data streams wait for their share of CPU time, thus reducing latency and increasing throughput.

Example: