At-Least-Once vs Exactly-Once: There Is No Third Option

There are three delivery semantics people talk about and only two of them really exist. At-most-once is fire and forget. The producer sends a message and never retries. If the network drops it, the message is gone. This is fine for telemetry where losing a few percent of data points does not matter, and terrible for anything that touches money. At-least-once retries until it gets an acknowledgment. The producer is willing to send the same message twice, three times, or thirty times, because losing the message is worse than delivering it twice. This is what most production systems actually do.

Exactly-once is the one that sounds different but is not. The wire is still at-least-once. What changes is that the consumer keeps a deduplication table keyed by some identifier, and when it sees the same identifier twice it processes the message exactly once. The "exactly-once" property is not a network guarantee, it is a processing guarantee built on top of at-least-once plus idempotency. Kafka transactions are a special case where the dedup table is the broker's internal state and the side effect is a write to another Kafka topic. As soon as your side effect leaves Kafka, you are back to needing your own dedup.

The production failure I have personally cleaned up: a billing pipeline used at-least-once delivery from a payment processor and skipped the idempotency layer because "the network is reliable in the same AZ." A 30-second TCP blip during a deploy caused the producer to retry, and roughly 0.2% of charge events were processed twice. Sounds small. It was several hundred customers double-charged, which means support tickets, refunds, chargebacks, and a postmortem the CFO reads.

The fix is to make every consumer idempotent. Define an idempotency key as source_id + sequence_number or whatever uniquely identifies the logical event, and store seen keys in a dedup table with a TTL longer than your maximum retry window. Before doing the side effect (charging the card, writing the row, sending the email) check the table. If you have seen the key, return the cached result. If not, do the work and record the key in the same transaction as the side effect, otherwise you reintroduce the duplicate window you just closed.

The mental model: at-least-once is what the network gives you. Exactly-once is what you build on top of it. Anyone selling you exactly-once without showing you the dedup table is selling you at-least-once and hoping for the best.