Designing Idempotent Kafka Consumers That Never Double-Process

When you process millions of events, retries are not an edge case — they're the steady state. The question is never if an event gets redelivered, but what happens when it does.

The problem with retries

Kafka gives you at-least-once delivery. Combine that with consumer restarts, partition rebalances, and the occasional 3am incident, and the same message will land in your handler more than once. If that handler isn't idempotent, you double-charge a customer or double-send an email.

Exactly-once delivery is mostly a marketing phrase. What you actually want is effectively-once processing — and you build that yourself.

Dedup keys

The pattern I reach for: derive a deterministic dedup key from the event's natural identity, then check it against a fast store before doing any side effect.

async def handle(event):
    key = f"order:{event.order_id}:{event.version}"
    # SET NX returns False if the key already exists
    if not await redis.set(key, "1", nx=True, ex=86400):
        return  # already processed — skip silently
    await charge_customer(event)

The transactional outbox

Dedup alone isn't enough when you also need to publish a downstream event. If the DB write commits but the publish fails, you're inconsistent. The outbox pattern makes the write and the publish atomic:

• Write the business row and an outbox row in the same DB transaction.
• A separate relay reads the outbox and publishes to Kafka.
• Mark the outbox row sent only after the broker acks.

This turns 'exactly-once' (impossible) into 'effectively-once' (achievable), and it survives every failure mode the broker can throw at you.