Designing Idempotent Event Consumers
Reliable patterns for deduplication, state transitions, retry safety, and recovery in event-driven systems.
Duplicates are normal in distributed delivery. A consumer should make repetition safe rather than hope it never happens.
EXECUTIVE THESIS
Duplicate delivery is a normal property of distributed systems. An idempotent consumer makes repeated, delayed, and reordered input safe by protecting a clearly defined business invariant with atomic state transitions and evidence-based retention.
Choose the invariant
Decide whether uniqueness belongs to an event, command, entity version, or business operation. Persist that decision with the state it protects.
Test disorder
Exercise duplicate, delayed, missing, and out-of-order events in integration tests. Recovery behavior deserves the same rigor as the happy path.
Store the decision atomically
A deduplication marker written separately from the business update leaves a crash window. Put the processed-event record and protected state transition in the same local transaction, or use a naturally idempotent compare-and-set keyed by the business version.
Expire with evidence
Deduplication records cannot grow forever, but retention must exceed the longest plausible replay window. Base expiry on broker retention, backup restoration, partner retry policy, and incident procedures. If an old replay is still possible, deleting its key changes correctness.
SYSTEM DESIGN
Decision framework
Choose the idempotency key
Event IDs protect message processing, while order versions, payment attempts, and business command IDs protect domain operations. Select the key that represents the effect that must occur once.
Store the decision atomically
Write the deduplication record and business state in the same local transaction, or use a conditional update that naturally rejects an already-applied version. Separate writes create a crash window.
Model ordering explicitly
If order matters, carry entity versions and reject, buffer, or reconcile stale events. Idempotency prevents duplicate effects but does not automatically repair out-of-order state.
FROM DESIGN TO DELIVERY
Implementation playbook
- 1
Define the state machine
List valid transitions and make impossible transitions explicit. A repeated command should return the already-established outcome rather than start a second side effect.
- 2
Persist inbox evidence
Store the key, handler version, result, entity reference, and processing time with the protected update. This supports audit, replay, and safe handler evolution.
- 3
Classify failures
Retry timeouts and transient dependencies, dead-letter invalid or unsupported messages, and expose a controlled replay path that preserves the original identity.
- 4
Test disorder
Inject duplicates before and after commit, delayed versions, consumer crashes, partition rebalances, and replay after backup restoration. Assert convergence, not only message acknowledgement.
Production review
Use these two lists during architecture review, readiness assessment, and post-incident learning.
COMMON FAILURE MODES
- Using a short-lived cache as the only deduplication store
- Marking an event processed before the business update commits
- Assuming idempotency also solves entity ordering
READINESS CHECKLIST
- Business-meaningful idempotency key
- Atomic inbox and state transition
- Explicit version and ordering policy
- Retention longer than every replay window
- Duplicate, crash, and out-of-order integration tests