The Transactional Outbox: Publishing Events Without Losing Them

The bug looks innocent until it costs you a customer.

A service writes a row to orders, then publishes an OrderPlaced event to Kafka. Most of the time both succeed. Some of the time the database commit returns OK, the process dies before the publish, and the event is gone. The order exists. The downstream services that needed to know about it never will. You will not see this in tests. You will see it in a support ticket six weeks later.

The cause is that you have two independent side effects pretending to be one. You cannot make a database commit and a broker publish atomic. They sit on different systems, with different failure modes, and there is no transaction manager that spans both.

The transactional outbox pattern fixes this by moving the second side effect inside the first. Instead of publishing directly, the service writes two rows in the same database transaction. One row goes to the business table. One row goes to an outbox table, with the event payload, a unique event_id, and a status column.

If the transaction commits, both rows are durable. If it aborts, neither exists. There is no window where the order is saved but the event is missing.

A separate process is responsible for turning outbox rows into broker messages. There are two common shapes:

• A relay that polls the outbox table on an interval, publishes pending rows, and marks them sent. Simple, easy to run, slightly higher latency.
• A change-data-capture pipeline that tails the database write-ahead log directly. Tools like Debezium read Postgres logical decoding or MySQL binlog and stream inserts to Kafka with no polling overhead.

Either way, publishing is decoupled from the business transaction. The service does one thing: write the database. The relay handles delivery, with retries, backoff, and observability you control.

The catch most teams miss is the production failure mode after the relay restarts. If it crashes between publishing to the broker and updating the row status, the same event will be republished on recovery. This is not a bug to fix. It is a guarantee to embrace. The outbox gives you at-least-once delivery, never exactly-once.

That is why every system using this pattern also requires consumers to deduplicate on event_id. Without that, the outbox solves your producer problem and creates a consumer problem. With it, the entire pipeline is finally reliable.

When you see Sagas, microservice choreography, or event-sourced services that survive real outages, the outbox is almost always underneath.