The Other Half of the Outbox: Consumer-Side Deduplication

If you have already added idempotency keys to your API, it is tempting to think the duplicate problem is solved. It is half solved. The outbox pattern, broker retries, and consumer restarts all live downstream of your API, and they all produce duplicates on a different axis.

The contract of any reliable event pipeline is at-least-once delivery. The broker retries until the consumer acknowledges. If a consumer crashes after processing an event but before committing the offset, the next pod will see the same event again. This is not a bug to chase. It is the guarantee you bought.

That means you need two independent layers of idempotency, and they protect against different things.

The first layer is request idempotency. The client sends an Idempotency-Key header. The service stores it, deduplicates retries from flaky clients or load balancers, and returns the cached response. This is what most teams mean when they say "we added idempotency."

The second layer is event idempotency, and it is the one most teams discover the hard way. Every event in the outbox carries an event_id, usually a UUID generated when the producer wrote the outbox row. Downstream consumers must treat that ID as the deduplication key. Before applying an event, the consumer asks: have I seen this event_id before?

There are two common implementations.

• Inbox table. A table in the consumer's own database keyed by event_id. The consumer's transaction does the business write and the inbox insert together. If the event is replayed, the inbox insert hits a unique constraint and the whole transaction rolls back. The side effect happens at most once even though the event was delivered twice.
• Redis dedup set. A SET of recent event_id values with a TTL. Faster, but not transactional with the business database, so it only works when the side effect itself is idempotent (an upsert, a counter increment with a known version, a write to an idempotent sink).

This is the exactly-once-effects pattern. You cannot achieve exactly-once delivery on a real network. You can achieve exactly-once observable effect, by combining at-least-once delivery with idempotent application.

A few production failure modes are worth knowing.

TTL too short, and duplicates re-emerge after expiry, usually weeks later when a consumer is replayed from an old offset. Size the TTL to the maximum replay window you actually use.

Wrong dedup key. Using a business identifier like order_id instead of event_id collapses legitimate state transitions ("OrderPlaced" then "OrderShipped") into one. The key must identify the event, not the entity.

Non-atomic check-then-write. Two pods race, both see "not yet processed," both apply the side effect. The dedup write and the business write must share a transaction, or the check must use an atomic primitive like SET NX.

Get those right and the stream finally behaves the way the diagram suggested.