Idempotency and Exactly-Once: The Honest Truth About Delivery Guarantees

TL;DR: Exactly-once delivery on a network with crashes is impossible. The sender cannot distinguish a lost message from a lost acknowledgement, so it must either retry (risking duplicates) or give up (risking loss)^[1]. What you actually build is at-least-once delivery combined with idempotent processing to achieve "effectively exactly-once." Idempotency keys are the single most valuable pattern for safe APIs: Stripe retains them for 24 hours^[2], Kafka's transactional producer adds only ~3% throughput overhead^[3], and SQS FIFO deduplicates within a 5-minute window^[4]. Every critical endpoint should accept an idempotency key. Kafka's "exactly-once" only covers Kafka-to-Kafka pipelines.

Learning Objectives#

After this module, you will be able to:

• Explain why at-most-once and at-least-once are the only honest network delivery guarantees
• Design idempotency keys for HTTP APIs (the Stripe pattern)
• Implement deduplication at the consumer (message ID store with TTL)
• Use Kafka's idempotent producer and transactional producer correctly
• Model "exactly-once side effects" on non-transactional systems (SMS, payments, webhooks)

Intuition#

Two generals sit on opposite hills. They need to coordinate an attack, but every messenger they send through the valley might be captured. General A sends "Attack at dawn." Did General B receive it? A does not know. B sends an acknowledgement back. Did A receive the ack? B does not know. They can send acks of acks forever and never reach certainty.

Now replace the generals with your checkout service and your payment gateway. You submit a charge. The network times out. Did the charge go through? You do not know. If you retry, you might double-charge the customer. If you do not retry, the payment might be lost.

This is the fundamental problem. No protocol can solve it on an unreliable network. What you can do is make retries safe. If the payment gateway recognizes "I already processed this exact request" and replays the original response, your retry is harmless. That is idempotency: applying an operation twice produces the same result as applying it once.

The rest of this chapter teaches you how to build that safety net at every layer of a distributed system.

Theory#

The Two Generals problem#

The Two Generals problem proves that two parties communicating over a lossy channel cannot reach common knowledge of a decision in finite rounds^[1:1]. When service A sends a request to service B and waits for an acknowledgement, a timeout could mean:

• The request was lost in transit
• B received and processed the request, but the ack was lost
• B crashed after processing
• B is simply slow

A cannot distinguish these cases. To be safe, A retries, which risks duplicate delivery. To avoid duplicates, A can refuse to retry, which risks loss. Tyler Treat's 2015 essay frames this as the core impossibility: "You cannot have exactly-once delivery"^[1:2].

This gives us the delivery taxonomy:

• At-most-once: send and forget. Messages may be lost, never duplicated. (Kafka acks=0, UDP fire-and-forget.)
• At-least-once: retry until acknowledged. Messages are never lost, but may be duplicated. (The practical default for every production system.)
• Exactly-once processing: at-least-once delivery combined with an idempotent consumer, so duplicates produce no visible side effects. This is what Kafka, Pulsar, and Stripe actually provide^[3:1][5].

The phrase "exactly-once delivery" is marketing. What exists is exactly-once processing.

Delivery guarantee taxonomy#

The distinction between delivery and processing is not pedantic. It determines where you place the deduplication logic:

• At-most-once is appropriate for metrics, telemetry, and best-effort notifications where losing a few events is acceptable.
• At-least-once is the default for any system where data loss is unacceptable. Every major message broker (Kafka, SQS, RabbitMQ, Pulsar) defaults to this mode.
• Effectively exactly-once requires the consumer to be idempotent. The broker delivers at-least-once; the consumer ensures that processing a message twice has the same effect as processing it once.

Apache Pulsar provides "message deduplication" via producer sequence IDs rather than claiming "exactly-once delivery"^[5:1]. Kafka uses "exactly-once semantics" but the Confluent documentation carefully notes this only covers Kafka-to-Kafka pipelines^[3:2].

Idempotency fundamentals#

An operation f is idempotent if f(f(x)) = f(x). Applying it twice produces the same observable effect as applying it once.

Natural idempotency requires no bookkeeping:

• PUT /users/42 {name: "Alice"} replaces the resource regardless of how many times you call it
• DELETE /orders/99 deletes the order; calling it again returns 404 but changes nothing
• SET balance = 100 assigns a fixed value; repeating it is harmless

Non-idempotent operations need help:

• POST /charges creates a new charge per call
• INCREMENT counter BY 1 changes the total on every retry
• "Send SMS to +1-555-0100" produces a new message each time

HTTP semantics assign idempotency to GET, HEAD, OPTIONS, PUT, and DELETE, but not to POST or PATCH^[6]. When your operation is inherently non-idempotent, you need induced idempotency: a client-provided key that the server uses to deduplicate.

The idempotency key pattern#

The canonical implementation comes from Stripe^[7]. The client generates a unique value (typically a UUIDv4) per logical request and sends it in the Idempotency-Key HTTP header. The server stores the tuple (key, request fingerprint, response, status) and replays the stored response on retry instead of re-executing the work.

The IETF draft draft-ietf-httpapi-idempotency-key-header-07 (October 2025, working-group draft; not yet a finalized RFC) codifies this pattern with specific status codes^[6:1]:

• 409 Conflict for concurrent retries on the same key (another request is still in-flight)
• 422 Unprocessable Entity for a key reused with a different request body
• 400 Bad Request when a required key is missing

Stripe retains idempotency keys for 24 hours^[2:1]. A request that returns a 400 error replays the same 400 on retry with the same key; to recover, the client must mint a new key^[2:2].

The client retries the same request with the same Idempotency-Key; the server replays the cached response without re-executing the side effect.

The server stores a request fingerprint (hash of method + path + body), not the whole body, to detect reuse with a different payload while keeping storage small^[8]. Brandur Leach's reference implementation adds a recovery_point field that supports resumable multi-step workflows: a crashed request resumes from the last completed step rather than replaying all steps^[8:1].

Consumer-side deduplication#

On the consumer side, you record each message's ID and skip processing if the ID has been seen. Typical stores:

• Redis SET with TTL: SET NX dedup:{msg_id} 1 EX {ttl}. Fast, bounded memory, language-agnostic.
• PostgreSQL unique constraint: INSERT INTO processed_messages (msg_id) VALUES ($1) ON CONFLICT DO NOTHING. ACID guarantee, no separate store to operate.
• Bloom filter: low-memory approximate dedup. Never produces false negatives (safe), but false positives silently drop legitimate messages. Pair with a ground-truth store for positives.

Consumer checks a Redis set for the message ID before processing; the TTL must outlive the producer's maximum retry window.

The critical design decision is TTL sizing. PayPal IPN retries for up to 4 days^[9]. If your Redis TTL is 1 hour, late retries produce duplicates. Size the TTL to at least the producer's maximum retry window. For long-tail scenarios, use a tiered approach: hot Redis with 24-hour TTL plus a PostgreSQL unique index for durable long-tail dedup.

Kafka exactly-once semantics#

As of Kafka 0.11 (KIP-98, 2017), Kafka provides two levels of deduplication^[10]:

Idempotent producer (enabled by default since Kafka 3.0^[11]): every producer gets a Producer ID (PID). Each message carries (PID, epoch, sequence number) per partition. The broker rejects a produce if the sequence is not exactly last_committed + 1 for that (PID, partition), eliminating duplicates caused by producer retries. Throughput penalty: negligible versus at-least-once^[3:3].

Transactional producer: adds a user-supplied transactional.id that survives process restarts. The broker writes begin markers, produce RPCs, consumer offset commits via sendOffsetsToTransaction, and finally COMMIT or ABORT markers. Consumers using isolation.level=read_committed buffer messages until they see the commit marker, then emit^[10:1].

Idempotent producer deduplicates retries on a single partition via sequence numbers; transactional producer adds atomic multi-partition writes plus offset commit.

Key configuration: transaction.timeout.ms defaults to 60 seconds, max.transaction.timeout.ms to 15 minutes, and transactional.id.timeout.ms to 7 days^[10:2]. Kafka Streams with processing.guarantee=exactly_once and a 100 ms commit interval sees 15-30% throughput overhead; with a 30-second commit interval, overhead drops to near zero^[3:4].

External side effects: the hard part#

Any mutation outside your ACID boundary (sending an SMS, charging a card, posting to a webhook) cannot be rolled back. Kafka EOS does not cover it. This is where the intent-row pattern comes in^[8:2]:

1. Atomic phase 1: Insert an "intent" row locally (status=pending, idempotency_key=event_id)
2. External call: Call the foreign API with an idempotency key the foreign system accepts
3. Atomic phase 2: Mark the intent row as done, store the response

On retry, the consumer checks the intent row: if done, skip and replay the cached response. If pending with an expired lock, re-acquire and retry the external call. If pending with a held lock, return 409.

The intent row is written before the external call; on retry, the consumer checks the intent and either resumes, skips, or re-executes.

This pattern composes because Stripe, Twilio, Adyen, PayPal, and Square all expose idempotency key mechanisms on their APIs^[6:2]. You derive a deterministic key from your local event ID (e.g., payment-intent-{event_id}) and pass it to the external service. Both sides deduplicate independently.

Real-World Example#

Stripe's idempotency keys at scale.

Stripe's public API requires an Idempotency-Key header on every mutating POST request. The system has been battle-tested since at least 2015 and forms the foundational safety net for millions of API calls per day^[7:1].

The server lookup on each request follows this path: load the idempotency key row for (user_id, idempotency_key). If missing, insert with recovery_point = started and proceed. If present and recovery_point = finished, replay the cached (response_code, response_body). If present and locked by another in-flight request, return 409 Conflict^[8:3].

Brandur Leach's reference implementation (rocket-rides-atomic) demonstrates the full pattern in PostgreSQL^[8:4]:

CREATE TABLE idempotency_keys (
    id              BIGSERIAL   PRIMARY KEY,
    idempotency_key TEXT        NOT NULL,
    user_id         BIGINT      NOT NULL,
    recovery_point  TEXT        NOT NULL,
    locked_at       TIMESTAMPTZ DEFAULT now(),
    request_params  JSONB       NOT NULL,
    response_code   INT         NULL,
    response_body   JSONB       NULL
);
CREATE UNIQUE INDEX ON idempotency_keys (user_id, idempotency_key);

The recovery_point column acts as a state machine (started, ride_created, charge_created, finished). Each step wraps in a SERIALIZABLE transaction. If the process crashes between steps, the next retry resumes from the last committed recovery point rather than replaying the entire workflow^[8:5].

Background jobs are transactionally staged: a staged_jobs row is inserted in the same transaction as the domain write. A separate enqueuer process moves staged jobs to the work queue only after the transaction commits. This prevents the "email sent but transaction rolled back" inversion^[8:6].

Key design decisions that make this work at Stripe's scale:

• Per-user uniqueness: the unique constraint is (user_id, idempotency_key), so two different users can reuse the same key value without collision
• 24-hour retention: keys expire after 24 hours, bounding storage growth^[2:3]
• Deterministic downstream keys: when calling external APIs (card networks, banks), Stripe derives a child idempotency key from the parent (e.g., stripe-charge-{key_id}) so the external system also deduplicates

Trade-offs#

Common Pitfalls#

Exercise#

You are building a webhook receiver that processes payment notifications from a provider. The provider retries on any non-2xx response for up to 3 days. Your service may crash mid-processing. Design the processing pipeline so that each logical payment is applied exactly once to the user's balance, even with duplicate webhooks, out-of-order delivery, and crashes.

CREATE TABLE webhook_intents (
    provider    TEXT NOT NULL,
    event_id    TEXT NOT NULL,
    status      TEXT NOT NULL DEFAULT 'pending',
    locked_at   TIMESTAMPTZ,
    result      JSONB,
    PRIMARY KEY (provider, event_id)
);

UPDATE balances
SET amount = amount + $credit,
    last_event_sequence = $seq
WHERE user_id = $uid AND last_event_sequence < $seq;

Key Takeaways#

• Exactly-once delivery on a network is impossible. What you build is at-least-once delivery + idempotent processing = effectively exactly-once.
• Idempotency keys are the single most valuable pattern for safe APIs. Every critical mutating endpoint should accept one.
• Consumer-side dedup via a store + TTL handles producer duplicates without changing producers. Size the TTL to the producer's maximum retry window.
• Kafka's exactly-once only covers Kafka-to-Kafka. External side effects (HTTP calls, SMS, payments) always need their own idempotency mechanism.
• The intent-row pattern makes external side effects safe: write intent locally, call externally with a derived key, mark done atomically.
• Design APIs assuming clients will retry. They will. Make retries safe by default.
• Natural idempotency (PUT, DELETE) is free. Use it whenever the operation fits. Reserve idempotency keys for inherently non-idempotent operations (POST /charges).

Further Reading#

• Designing robust and predictable APIs with idempotency (Stripe, 2017) - the canonical engineering post that established the idempotency key pattern; start here if you read nothing else.
• Implementing Stripe-like Idempotency Keys in Postgres (Brandur Leach, 2017) - full working reference implementation with schema, state machine, recovery points, and Ruby code; the best "how to actually build this" resource.
• Exactly-Once Semantics Are Possible: Here's How Kafka Does It (Confluent, 2017, updated 2025) - the definitive walkthrough of Kafka's idempotent and transactional producer with throughput benchmarks.
• KIP-98: Exactly Once Delivery and Transactional Messaging (Apache, 2017) - the primary design document with full API, configuration, and message-format details.
• You Cannot Have Exactly-Once Delivery (Tyler Treat, 2015) - the blog post that reset the industry's framing; read this to internalize why the impossibility matters.
• The Idempotency-Key HTTP Header Field (IETF draft-07, October 2025) - the emerging standard with status codes, security considerations, and implementation status across Stripe, Adyen, PayPal, and Twilio.
• Life Beyond Distributed Transactions (Pat Helland, 2007/2016) - the foundational argument for building with idempotent messages instead of distributed transactions; essential reading for anyone designing event-driven systems.
• Exactly-once processing in Amazon SQS (AWS) - FIFO deduplication mechanics, the 5-minute window, and content-based vs explicit dedup IDs.

Flashcards#

References#