Idempotent Consumer: Turn Redelivery Into One Business Effect

Introduction

The Transactional Outbox article ended with a deliberate gap: the relay may publish the same Integration Event more than once. That is not a defect in the pattern. It is the safe result of publishing before marking the outbox row complete.

The receiving service now owns the next guarantee:

Repeated delivery of one logical message must produce one acceptable business effect.

That is the Idempotent Consumer pattern. It does not promise that the handler runs once. It makes repeated execution safe through message identity, atomic persistence, and domain-specific outcome rules.

The Production Problem

An inventory consumer receives orders.order-placed.v1, reserves stock, saves the reservation, and then completes the broker message.

The database commit succeeds. Before the completion reaches the broker, the process stops. The lock expires and another instance receives the same message.

Without idempotency, the second delivery can:

• reserve the same stock twice;
• charge a payment twice;
• append a duplicate ledger entry;
• send another email;
• publish another downstream command.

Acknowledging before the database commit avoids the duplicate but allows loss: the process can stop after acknowledgment and before applying the business change.

At-least-once delivery intentionally prefers possible duplication over silent loss. The application must make duplication harmless.

Core Concepts

Delivery identity is not business identity

A message ID identifies one logical delivery contract instance. A business key identifies a domain object or operation.

Use the message ID to detect broker redelivery. Use a business idempotency key when several different messages can represent the same business request. For example, two independently produced ChargePayment commands may have different message IDs but the same PaymentAttemptId.

Do not deduplicate only by OrderId unless the contract says an order can produce only one relevant event forever. OrderPlaced, OrderCancelled, and OrderReopened can legitimately share an order ID.

Idempotency is an outcome property

An operation is idempotent when repetition leaves the system in an acceptable equivalent state.

SET Status = 'Reserved' looks idempotent, while Quantity = Quantity - 1 does not. But an older “set” event can overwrite a newer status. Natural idempotency does not solve ordering.

The consumer must protect:

• repeated delivery of the same message;
• concurrent delivery to multiple instances;
• stale or out-of-order messages when the domain cares;
• downstream side effects that are outside its local transaction.

Deduplication and the business update share one transaction

The strongest common implementation inserts a processed-message record and applies the business state change in the same local transaction.

If the transaction rolls back, neither is visible and redelivery can retry. If it commits, a unique constraint prevents a second transaction from repeating the work.

This closes the consumer’s database boundary. It does not atomically cover an external payment gateway, SMTP server, or another broker. Use that external system’s idempotency key, a local outbox, or both.

The deduplication scope includes the consumer

Two independent consumers may both need to process the same event. A key of only MessageId would let one suppress the other if they share a database.

Use a unique key such as:

(ConsumerName, MessageId)

Keep ConsumerName stable across deployments. Renaming it without migrating records can replay retained messages.

How It Works

1. Receive under a broker lock or manual acknowledgment mode.
2. Validate the envelope and contract version.
3. Start a local database transaction.
4. Insert (ConsumerName, MessageId) into a deduplication table.
5. If the unique key already exists, commit no new business work and complete the delivery.
6. Otherwise apply the business change and, if needed, write downstream messages to a local outbox.
7. Commit the database transaction.
8. Complete or acknowledge the broker message.

If step 8 is ambiguous, redelivery repeats the algorithm and stops at the unique key.

For a permanently invalid contract, do not record it as successfully processed. Dead-letter it with a bounded reason. For a transient database failure, abandon or allow redelivery.

Architecture Diagram

.NET Implementation

Model the processed-message record

public sealed class ProcessedMessage
{
    public string Consumer { get; private set; } = null!;
    public Guid MessageId { get; private set; }
    public DateTimeOffset ProcessedAtUtc { get; private set; }

    private ProcessedMessage() { }

    public ProcessedMessage(
        string consumer,
        Guid messageId,
        DateTimeOffset processedAtUtc)
    {
        Consumer = consumer;
        MessageId = messageId;
        ProcessedAtUtc = processedAtUtc;
    }
}

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<ProcessedMessage>()
        .HasKey(x => new { x.Consumer, x.MessageId });
}

The database key, not a read-before-write check, resolves concurrent delivery. Two consumers can both observe “not present”; only one insert can commit.

Keep the local effects atomic

public sealed class OrderPlacedConsumer(
    InventoryDbContext dbContext,
    TimeProvider clock)
{
    private const string ConsumerName =
        "inventory.order-placed.v1";

    public async Task<ConsumeResult> HandleAsync(
        MessageEnvelope<OrderPlacedIntegrationV1> envelope,
        CancellationToken cancellationToken)
    {
        await using var transaction =
            await dbContext.Database.BeginTransactionAsync(
                cancellationToken);

        dbContext.ProcessedMessages.Add(
            new ProcessedMessage(
                ConsumerName,
                envelope.MessageId,
                clock.GetUtcNow()));

        try
        {
            var reservation = await dbContext.Reservations
                .SingleOrDefaultAsync(
                    x => x.OrderId == envelope.Body.OrderId,
                    cancellationToken);

            if (reservation is null)
            {
                reservation = InventoryReservation.Create(
                    envelope.Body.OrderId,
                    envelope.Body.Lines);

                dbContext.Reservations.Add(reservation);
            }

            await dbContext.SaveChangesAsync(cancellationToken);
            await transaction.CommitAsync(cancellationToken);

            return ConsumeResult.Processed;
        }
        catch (DbUpdateException exception)
            when (IsProcessedMessageConflict(exception))
        {
            await transaction.RollbackAsync(cancellationToken);
            return ConsumeResult.AlreadyProcessed;
        }
    }
}

IsProcessedMessageConflict must inspect the selected database provider’s error code and constraint name. Do not treat every DbUpdateException as a duplicate; a foreign-key failure, timeout, or disk error needs different recovery.

The broker adapter completes both Processed and AlreadyProcessed. It abandons transient exceptions and dead-letters only classified permanent failures.

Handle external effects

Do not send email or publish another message between the database commit and broker acknowledgment. That creates another dual write.

Instead:

• write a downstream Integration Event to this consumer’s local outbox;
• call a payment provider with a persisted business idempotency key;
• record a durable work item for a nontransactional adapter;
• reconcile external state when an API cannot provide idempotency.

The processed-message table proves that the local transaction ran. It cannot prove what an unrelated external API did.

Common Mistakes

Read before insert

AnyAsync(MessageId) followed by an insert races under concurrency. Keep the unique constraint and handle its exact violation.

Acknowledge before commit

This changes the mode toward at most once. A process failure can lose the business effect permanently.

Store the ID in a different transaction

Committing the dedupe record before the business update suppresses recovery after an update failure. Committing it afterward permits duplicate effects.

Assume the broker’s duplicate detection is enough

Broker deduplication is usually scoped by producer, entity, partition, session, or time window. It does not cover redelivery after consumer failure or duplicate messages outside the configured window.

Retain dedupe records forever without a plan

An unbounded table increases storage and index cost. Retention must exceed the longest possible message replay and broker retention window. If old messages can be replayed indefinitely, archive keys or use business-state idempotency rather than deleting proof blindly.

Treat every failure as retryable

Malformed payloads and unsupported versions do not heal through immediate redelivery. Classify them and dead-letter after preserving diagnostic metadata.

Ignore ordering

A duplicate of an older event can be harmless while a first delivery of that older event is harmful. Track an aggregate version, sequence, or business timestamp when state transitions require freshness.

Best Practices

• Make message ID mandatory and immutable across producer retries.
• Use a database uniqueness constraint as the concurrency boundary.
• Commit deduplication, business state, and local outbox messages together.
• Complete the broker delivery only after commit.
• Bound processing time within the broker lock or renew it deliberately.
• Keep prefetch and handler concurrency within database and downstream capacity.
• Classify duplicate, transient, permanent, and stale outcomes separately.
• Define dedupe retention from actual broker and replay policies.

Observability

Record:

• delivery result: processed, duplicate, stale, transient failure, permanent failure;
• stable consumer and contract names;
• message, correlation, and causation IDs;
• processing duration and settlement outcome;
• retry or delivery count when the broker exposes it.

Measure duplicate rate by producer and contract. A sudden increase can indicate publisher-confirm ambiguity, relay crashes, short lock durations, or consumer timeouts. Alert on permanent failures and growing processing lag, not on every expected duplicate.

Create a consumer processing span when application handling starts, not when the client library merely prefetches. Link it to the producer context when the instrumentation follows OpenTelemetry messaging conventions. Keep message IDs out of metric labels because they create unbounded cardinality.

Verification

Run the same message concurrently against two consumer instances. Expect one business change, one processed-message record, and successful settlement of both deliveries.

Then stop a consumer after database commit but before settlement. Expect redelivery, a duplicate classification, and no repeated downstream business effect. Repeat with the database transaction forced to roll back; expect no dedupe record and successful processing on redelivery.

Decision Matrix

Consumer operation	Mechanism
Naturally repeatable write with no ordering risk	Stable state assignment may be sufficient; still observe duplicates
Database update under at-least-once delivery	Processed-message key plus business update in one transaction
Multiple contracts share one business request	Add a business idempotency key in addition to message ID
Consumer publishes another message	Add a local Transactional Outbox
External API supports idempotency keys	Persist and reuse one business key on every attempt
External API has no idempotency or query API	Avoid automatic retry or add reconciliation/manual recovery
Old events can overwrite new state	Add version or sequence validation
Occasional duplicate effect is explicitly acceptable	Simpler at-least-once handler may be enough

Relationship to Other Patterns

Transactional Outbox makes producer publication reliable but permits duplicates. Idempotent Consumer completes that boundary on the receiving side.

Domain Events describe internal facts; deduplication belongs at the Integration Event consumer boundary, not inside an aggregate’s in-memory event list.

Circuit Breaker, Retry, and Backoff manage transient dependency failures. They can increase attempts, so they make idempotency more important rather than replacing it.

Saga participants need idempotent forward and compensation commands because either can be redelivered. A Dead Letter Queue handles messages that cannot complete after classification; it is not a duplicate store.

Decision checklist

• Is every logical message assigned one stable ID?
• Is the consumer identity part of the dedupe key?
• Do deduplication and the business update commit together?
• Does a database constraint resolve concurrent delivery?
• Is broker settlement performed only after commit?
• Are downstream external effects protected by an outbox or business idempotency key?
• Are stale messages detected separately from duplicates?
• Does retention cover the complete replay horizon?
• Can tests reproduce the commit-before-ack failure window?

Key takeaways

• Handler execution can repeat even when the desired business effect cannot.
• A unique database key is stronger than a read-before-write check.
• Producer-side duplicate detection is useful but scoped and insufficient.
• Local deduplication cannot make an unrelated external API transactional.
• Duplicate rate is an operational signal, not merely noise.