A service that handles 10,000 concurrent connections comfortably and falls over at 12,000 has hit a specific limit, and there are only a handful of candidates. Knowing which one you are near is the difference between buying capacity you do not need and changing one number.

Descriptors, the limit most people meet first

Every socket a process holds occupies a file descriptor, and descriptors are a bounded per-process resource.

The default soft limit on many systems is small, historically 1,024 and on modern distributions often a few thousand, which is far below what a network service needs. When a process exhausts its descriptors, accept fails with a too-many-open-files error, and the common symptom is a server that stops accepting while using almost no CPU and almost no memory.

The limit is raised per process, and the detail that catches people is that it is inherited from whatever launched the process, so raising the shell's limit does nothing for a service started by an init system that sets its own. Descriptors are also consumed by files, pipes and the listening socket itself, so the ceiling for connections is slightly below the configured number.

Memory per connection, and why it is larger than it looks

Each connection costs kernel memory for its socket buffers plus whatever the application holds alongside it.

The kernel side is the send and receive buffers, which are sized dynamically between a configured minimum and maximum. A server tuned for high throughput with large maximums will happily allocate megabytes per connection under load, so a tuning change that improves single-stream throughput can cut the sustainable connection count by an order of magnitude. The two goals are in direct tension and the resolution is to set the maximum from the delay product rather than from optimism.

The application side is usually the larger half: a per-connection buffer, a parsed request, a session object, and in a thread-per-connection model an entire thread stack, which defaults to megabytes. Ten thousand connections at a two megabyte stack is twenty gigabytes of virtual address space before any application data, which is why thread-per-connection models do not scale to high connection counts.

Connection tracking, the limit that is easy to forget

Any stateful firewall or address translation in the path keeps a table entry per connection, and that table has a fixed size.

On a host with connection tracking enabled, a full table means new connections are dropped and the kernel logs a table-full message, which is one of the few limits that announces itself clearly. The default size is derived from memory and is frequently in the low hundreds of thousands, which sounds generous until a service with short-lived connections fills it with entries waiting out their timeout.

The important subtlety is that the table counts connections that are no longer active. A closed connection's entry persists for a timeout, and the default timeout for an established entry is often days, so a service churning connections accumulates entries far faster than its concurrent count suggests. Either raise the table size, shorten the timeouts, or exempt the traffic from tracking entirely where the firewall does not need it.

The accept queue and the application's own ceiling

Beyond the kernel's limits, the application decides how many connections it will actually serve.

A completed handshake sits in the accept queue until the application calls accept, and that queue has a configured depth. If the application is slow to accept, the queue fills and further connections are dropped or refused, which presents as connection failures even though the server has ample descriptors and memory.

Above that sits the application's concurrency model: a worker pool of 200 threads serves at most 200 requests simultaneously regardless of how many connections are open, and an event-driven server with a single loop is bounded by that loop's throughput. A high connection count with low concurrency is perfectly reasonable for mostly-idle connections and fails badly if those connections are all active.

Separating a connection limit from a throughput limit

The diagnosis that matters is whether you are out of connections or out of capacity to serve them, because the fixes are unrelated.

A connection limit looks like failures at accept time, with healthy latency for connections that do get through and low resource utilisation. A throughput limit looks like rising latency across all connections, with CPU, memory or a downstream dependency saturated, and no accept failures at all.

Conflating them produces the classic wrong move: raising the descriptor limit on a server that was actually CPU-bound, which lets more connections in and makes latency worse for everyone. The queue depth and the accept failure counters tell you which story you are in.

The numbers worth knowing for your own service

Capacity planning here is empirical and the measurements are cheap to take.

Measure the memory cost of one idle connection and one active connection, which gives the memory-bound ceiling directly. Measure the maximum connection establishment rate, since that is often the binding constraint rather than the concurrent count. Measure the concurrent count at which latency begins to rise, which is the useful ceiling as opposed to the failure ceiling.

Those three numbers, taken once and recorded, turn capacity questions into arithmetic. Without them, every discussion about whether the service can handle an event is speculation.

The concurrency model decides the shape of everything

How a server waits for input determines which of the limits above it will ever reach.

A thread or process per connection is simple and bounded by memory, since each costs a stack and a scheduler entry, and context switching between thousands of threads consumes real CPU. The practical ceiling is low thousands.

An event-driven model keeps one thread per core and multiplexes connections over a readiness interface, so a connection costs a descriptor and a buffer, perhaps a few kilobytes, and hundreds of thousands are achievable. The cost is that any blocking call stalls every connection on that loop, which is why a single synchronous database driver can destroy an otherwise excellent event-driven server.

The hybrid most modern runtimes use, lightweight tasks scheduled over a small thread pool, gives the event model's density with the thread model's programming style, and is the right default unless there is a reason otherwise.

Interrupts, offload and the per-packet cost

At high connection counts the per-packet work becomes visible, and it lands unevenly.

A network interface raises interrupts for arriving packets, and by default those may all land on one core, which saturates that core while the others idle. Spreading receive processing across cores by hashing on the connection tuple is standard and is sometimes not enabled, which presents as a throughput ceiling with one core at 100 percent and overall utilisation low.

Offload features let the interface coalesce segments and compute checksums, which reduces per-packet CPU substantially for bulk traffic. They occasionally interact badly with capture tools, producing impossibly large frames in a trace, which is a known artefact rather than a problem.

The diagnostic habit worth forming is to look at per-core utilisation rather than the average, because every one of these problems is invisible in an aggregate.

Where the limits are configured

A brief orientation, since the hardest part is often finding the right knob.

Per-process descriptor limits come from the service manager's unit configuration or the container runtime, not from a login shell. System-wide descriptor and socket maximums, buffer sizes, the ephemeral range, the backlog maximum and the tracking table size are kernel parameters, which must be set persistently rather than only at runtime or they revert on reboot.

In a container, some of these are namespaced per container and some are shared with the host, and which is which is a frequent source of confusion: descriptors are per process, the tracking table may be shared, and buffer settings are usually the host's. Knowing which layer owns a setting saves changing the wrong one repeatedly.