Exactly-Once Kafka Without Magical Thinking
What exactly-once guarantees do—and do not—mean across brokers, databases, retries, and downstream side effects.
Exactly-once is a carefully bounded guarantee. The engineering work begins where the marketing phrase ends.
EXECUTIVE THESIS
Exactly-once processing is never a property of the entire business workflow by default. It is a bounded guarantee created by aligning broker transactions, application state, idempotency, and external side effects around an explicitly defined invariant.
Define the boundary
Kafka can make a consume-transform-produce flow transactional, but an external database or API sits outside that transaction unless you deliberately bridge the boundary.
Build for replay
Use stable event identifiers, idempotency keys, monotonic state transitions, and reconciliation jobs so replay becomes an ordinary operation rather than an incident.
Bridge external side effects
When a consumer must update a database and publish an event, an outbox table gives both changes one local transaction. A relay can publish the outbox record repeatedly because downstream consumers deduplicate on the event identifier. The same discipline applies to email, payment, and partner API calls.
Test the ugly timeline
Correctness tests should include a crash after commit but before acknowledgement, delayed partitions, consumer rebalances, duplicate delivery, and a replay from an older offset. If the state converges in every timeline, the guarantee is meaningful beyond the broker configuration.
SYSTEM DESIGN
Decision framework
Name the guarantee boundary
A Kafka consume-transform-produce loop can be transactional. A database update, payment call, email, or partner API is outside that boundary until the application introduces an outbox, inbox, idempotency key, or compensating workflow.
Prefer convergence over suppression
Deduplication reduces repeated work, but replay-safe state transitions provide the stronger guarantee. Model commands and entity versions so applying an event twice converges on the same valid state.
Budget for disorder
Duplicates, rebalances, delayed partitions, poison messages, and partial outages are ordinary operating conditions. Design the data model and operational tools around those timelines before optimizing throughput.
FROM DESIGN TO DELIVERY
Implementation playbook
- 1
Define the invariant
Decide whether uniqueness belongs to an event ID, payment attempt, order version, or business command. Persist that key with the state transition it protects.
- 2
Bridge local state
Write the business update and outbox record in one database transaction. Publish the outbox asynchronously and let consumers maintain an inbox or naturally idempotent state transition.
- 3
Control retries
Classify transient and permanent errors, use bounded exponential backoff, isolate poison events, and retain enough context in the dead-letter record to reproduce the decision.
- 4
Prove replay
Restore a production-shaped snapshot, rewind offsets, inject duplicates and crashes, and verify that balances, notifications, and derived views converge without manual edits.
Production review
Use these two lists during architecture review, readiness assessment, and post-incident learning.
COMMON FAILURE MODES
- Assuming broker configuration covers external side effects
- Deduplicating in a cache that can expire before replay
- Retrying permanent validation failures indefinitely
READINESS CHECKLIST
- Documented transaction boundary
- Stable event and business identifiers
- Atomic outbox or equivalent state bridge
- Bounded retry and dead-letter policy
- Automated replay and reconciliation tests