Exactly once is the most oversold phrase in stream processing, and most of the confusion it causes comes from the fact that it is describing something narrower and more useful than the words suggest.

What the phrase actually promises

Exactly once does not mean that each record is read once, or that your function runs once per record, or that nothing is ever retried. All of those things happen many times in a healthy pipeline. A task fails, the job restarts from its last checkpoint, and records that were already read are read again and pushed through your function again. That is not a bug in the guarantee. That is how the guarantee is implemented.

What exactly once promises is that each record affects the result once. The state the job holds, and the data the job has written out, end up as though every record had been processed exactly one time. Your map function may have been called five times for the same record across five recovery attempts. If four of those calls had their effects thrown away and one of them stuck, the guarantee holds.

This is why many engineers prefer the phrase effectively once. It is less marketable and much more accurate. The machinery does not prevent repeated processing. It makes repeated processing invisible in the output.

Why delivery cannot be exactly once

There is a harder claim people sometimes make, which is that a message was delivered exactly once across a network. That one is impossible, and it is worth knowing why, because the impossibility is what forces the design of everything else in this lesson.

A sender transmits a message and waits for an acknowledgement. The acknowledgement does not arrive. The sender cannot distinguish between two situations. In the first, the message was lost and never processed. In the second, the message arrived and was processed and the acknowledgement was lost on the way back. The sender has exactly two choices. It can resend, which risks processing the message twice. It can give up, which risks never processing it at all.

That is the whole problem, and no protocol removes it. You get at least once or at most once at the delivery layer, and that is a fact about networks rather than a limitation of any particular engine.

Exactly once processing is built on top of at least once delivery. The engine delivers duplicates and then arranges for the duplicates not to matter. Once you see it that way, the configuration options stop looking like magic and start looking like the three specific places duplicates have to be neutralised.

The three places duplicates appear

Walk a record through a pipeline and there are exactly three points where a retry can leave a visible mark.

The first is the source. After a restart the job rereads records it already read, because the only position it knows about is the one it recorded at its last checkpoint. Everything between that position and the moment of failure comes through a second time.

The second is operator state. If a counter was incremented by a record before the failure and that record arrives again after the failure, the counter is now wrong unless the counter itself was rolled back to the checkpointed value. State and position have to rewind together or the rewind makes things worse rather than better.

The third is the sink. If the job wrote a row to a table before failing, and then reprocesses the record that produced that row, there is now either a duplicate row or a corrected row depending entirely on what the sink does with a repeated write.

End to end exactly once requires all three to be handled. A job with checkpointed state and a replayable source but a naive sink has exactly once state and at least once output, which is a perfectly reasonable configuration as long as somebody has said out loud that it is the configuration.

What at least once actually costs

The honest way to decide whether you need the machinery is to measure what you are living with now.

Take a pipeline reading twelve partitions at five thousand records a second, which is eighteen million an hour. It checkpoints every forty five seconds. Over one month it failed over eleven times, for the ordinary reasons, which were two node evictions, three deploys that were not drains, four transient broker errors and two out of memory kills.

Each failover replays from the last successful checkpoint. On average the failure landed twenty three seconds into the checkpoint interval, so each one replayed about twenty three seconds of data, which at five thousand a second is a hundred and fifteen thousand records. Eleven failovers produced roughly one and a quarter million duplicate records.

Against a month of just under thirteen billion records, that is about ninety eight duplicates per million. Whether ninety eight per million is a catastrophe or a rounding error depends entirely on what the pipeline computes. For a dashboard of request rates, nobody will ever notice. For a count of distinct paying users, nobody will ever notice either, because the deduplication happens naturally in the aggregation. For a sum of transaction amounts that reconciles against a ledger, ninety eight per million is a daily unexplained discrepancy and somebody will eventually have to explain it.

The operations that are sensitive and the ones that are not

The reason the same duplicate rate can be harmless or fatal is that aggregations differ in how they respond to a repeated input.

Some are naturally immune. A maximum does not change when you add the same value twice. A minimum does not either. A set union, a distinct count on exact semantics, and an assignment of the latest value by key all absorb duplicates without noticing them.

Some are mildly sensitive. An average shifts slightly, by an amount proportional to the duplicate rate, which at ninety eight per million is far below the noise in almost any real metric.

Some are fully sensitive. A sum is wrong by the duplicated amount. A count is wrong by the duplicated count. An increment against an external counter is wrong and, crucially, is wrong permanently, because nothing in the pipeline remembers that it over counted.

This is the single most useful classification in the lesson, because it means the question is never whether the pipeline needs exactly once. It is which of the pipeline's outputs need it.

A worked example of the distinction

One job in the estate computes three things from the same stream. It maintains a per user last seen timestamp, a per hour request count, and a per account running revenue total.

Under at least once with a replayable source and checkpointed state, the last seen timestamp is exactly right, because the operation is idempotent by nature. Replaying a record sets the same timestamp again. The request count is wrong by about ninety eight per million, which is invisible in a chart whose vertical axis is in thousands. The revenue total is wrong by about ninety eight per million of revenue, which is a real number of real currency and accumulates forever because nothing recomputes it.

So the correct design is not to turn on exactly once for the job. It is to leave the first two alone and move the third one onto a sink that can absorb a repeat, or onto a transaction. Doing that for one of three outputs is a fraction of the latency cost of doing it for all three.

Why the vendors all say it anyway

It is worth understanding why a phrase this misleading survived. When the first generation of stream processors shipped, the realistic alternative was a job that silently lost records on every restart, because position and state were checkpointed separately and could disagree. Going from losing data to not losing data was an enormous improvement, and exactly once was the shortest phrase that described the difference to somebody who was not going to read the design document.

The phrase then became a checkbox on comparison tables, which is where the damage happened. A checkbox cannot express that the guarantee is conditional on a replayable source, that it costs the checkpoint interval in output latency when the sink is transactional, or that it says nothing at all about the external system you call from inside a map function. All three of those conditions are load bearing, and all three are invisible on a feature matrix.

So read the claim as a statement about the engine's internal state machine, which is where it is true, and verify the rest yourself. The rest is the part that breaks.

The rule worth remembering

Exactly once is a property of the effect, not of the execution. The engine will run your code more than once and your job is to make sure the second run is either prevented from landing or indistinguishable from the first. Everything that follows is two ways of doing that, one expensive and general, one cheap and conditional.