Exactly Once Is a Myth: Define the Guarantee at Every Boundary

Introduction

The series has deliberately used “at least once,” stable message identity, and idempotent business effects. That language can sound weaker than a platform feature labeled “exactly once.”

The problem is not that infrastructure guarantees are false. Kafka idempotent production, transactional stream processing, Azure Service Bus duplicate detection, and broker transactions can provide valuable scoped guarantees.

The problem is scope. A guarantee about records in one broker is not automatically a guarantee about:

• a producer database;
• a consumer database;
• an HTTP payment provider;
• an email recipient;
• a business operation spanning all of them.

“Exactly once” becomes useful only after naming exactly what, where, and for how long.

The Production Problem

A consumer receives PaymentRequested, charges a gateway, writes Paid to SQL, publishes PaymentCompleted, and commits its broker offset.

Any boundary can become ambiguous:

1. The gateway accepts the charge, but the response is lost.
2. SQL commits, but the consumer stops before offset commit.
3. The output message commits, but a downstream consumer repeats its database update.
4. Broker deduplication expires before an old producer retry arrives.

The platform may guarantee exactly one output record for a broker transaction while the customer is still charged twice.

The required production invariant is usually not “the handler executes once.” It is:

One approved payment attempt produces at most one captured charge and one durable, reconcilable ledger outcome.

That is an application guarantee composed from infrastructure features, durable identity, local transactions, and external-system contracts.

Core Concepts

Delivery, processing, and effect are different

• Delivery count: how many times a broker presents a message.
• Handler execution count: how many times application code runs.
• State transition count: how many committed changes occur.
• External effect count: how many payments, emails, shipments, or device actions occur.

Exactly-once delivery is neither necessary nor sufficient for exactly one business effect. Redelivery with idempotent state can be correct. One delivery followed by a repeated external API call inside the handler can still be wrong.

At most once

The system may lose work but does not intentionally redeliver it. Examples include receive-and-delete or acknowledging before processing.

Use it only when occasional loss is acceptable, such as some ephemeral telemetry.

At least once

The system retries until ownership transfers, so duplicate delivery is possible. Manual acknowledgment and lock-based brokers commonly provide this behavior.

This is a practical baseline for business messaging because loss is visible and duplicates can be controlled.

Effectively once

This informal phrase usually means delivery may repeat, but idempotency makes the observed business result equivalent to one successful execution.

It is useful only when the equivalence is defined. Sending the same email twice may not be equivalent. Setting a projection to version 12 twice may be.

Infrastructure-scoped exactly once

Examples:

• an idempotent Kafka producer prevents duplicate records from its retry sequence under its producer protocol;
• a Kafka transaction can atomically write output records and consumer offsets inside Kafka;
• Azure Service Bus duplicate detection suppresses repeated message IDs within a configured entity and time window;
• a database uniqueness constraint allows one committed row for one key.

Each guarantee has boundaries, configuration requirements, retention, failure semantics, and performance costs.

Application-scoped correctness

The application defines the business identity and composes:

• one producer state change plus Outbox intent;
• stable message ID;
• broker durability and settlement;
• one consumer state transition plus Inbox identity;
• downstream Outbox;
• external API idempotency or reconciliation.

This design can provide one acceptable business effect even though messages and handlers repeat.

How It Works

Start with a guarantee ledger for one workflow:

Boundary	Mechanism	Guarantee	Residual failure
Orders DB → Outbox	Local transaction	State and intent commit together	Relay may publish repeatedly
Outbox → Broker	Confirms plus retry	At-least-once publication	Confirmation can be ambiguous
Broker storage	Durable entity configuration	Broker-specific persistence	Retention, routing, or operator configuration
Broker → Consumer	Lock plus completion	At-least-once delivery	Handler may repeat
Consumer DB	Inbox key plus local transaction	One committed local effect per message	External effects remain separate
Consumer → Payment API	Business idempotency key	Provider-specific duplicate suppression	Window, contract, or reconciliation limits

Review the weakest row. The end-to-end claim cannot be stronger than an uncovered boundary.

Architecture Diagram

.NET Implementation

Express business identity

Do not let a transport-generated delivery tag define the business operation.

public sealed record CapturePayment(
    Guid MessageId,
    Guid PaymentAttemptId,
    Guid OrderId,
    Money Amount);

MessageId deduplicates one message. PaymentAttemptId identifies the business attempt across messages and HTTP retries.

Enforce the consumer’s local guarantee

public async Task HandleAsync(
    MessageEnvelope<CapturePayment> envelope,
    CancellationToken cancellationToken)
{
    await using var transaction =
        await db.Database.BeginTransactionAsync(
            cancellationToken);

    db.ProcessedMessages.Add(new ProcessedMessage(
        ConsumerName,
        envelope.MessageId,
        clock.GetUtcNow()));

    var payment = await db.Payments.SingleOrDefaultAsync(
        x => x.PaymentAttemptId ==
             envelope.Body.PaymentAttemptId,
        cancellationToken);

    if (payment is null)
    {
        payment = Payment.Start(
            envelope.Body.PaymentAttemptId,
            envelope.Body.OrderId,
            envelope.Body.Amount);

        db.Payments.Add(payment);
    }

    await db.SaveChangesAsync(cancellationToken);
    await transaction.CommitAsync(cancellationToken);
}

Production configuration should enforce unique keys on both:

(ConsumerName, MessageId)
PaymentAttemptId

The first handles redelivery. The second prevents semantically duplicated commands from starting two payments.

Protect the external call

If the payment provider accepts an idempotency key, persist and reuse PaymentAttemptId:

using var request = new HttpRequestMessage(
    HttpMethod.Post,
    "/payments/captures")
{
    Content = JsonContent.Create(payload)
};

request.Headers.Add(
    "Idempotency-Key",
    payment.PaymentAttemptId.ToString("N"));

The provider’s documented semantics determine whether this is enough. Check:

• uniqueness scope;
• retention window;
• behavior when the same key has a different payload;
• response replay;
• concurrent requests;
• timeout and reconciliation APIs.

Without provider idempotency, store an uncertain state after ambiguous timeout and query before retrying. Do not guess that the first attempt failed.

Use broker transactions only within their scope

For Kafka consume-transform-produce flows, transactions can atomically commit output records and consumed offsets in Kafka when the producer and consumers are configured correctly. A SQL update or payment API call does not join that Kafka transaction.

In .NET Kafka clients, use the selected client’s documented transaction APIs and compatible broker configuration. Do not wrap those calls in an unrelated TransactionScope and claim cross-system atomicity.

Common Mistakes

Using “exactly once” without a noun

Exactly once delivery, record append, state transition, and payment charge are different claims.

Trusting a deduplication window forever

Azure Service Bus duplicate detection is time-bounded and entity-scoped. A retry after the window can be accepted.

Confusing Kafka idempotence with database atomicity

Idempotent production prevents a class of duplicate broker writes. It does not atomically commit an independent relational database.

Generating a new ID on retry

Infrastructure cannot recognize a duplicate if the application changes its identity.

Treating natural idempotency as ordering safety

Setting a value twice may be safe, but applying version 8 after version 9 is stale.

Ignoring side effects outside durable state

Email, payment, files, and device commands require their own identity and recovery contract.

Claiming handler execution is once

Processes can restart, messages can redeliver, and callbacks can repeat. Design around committed effects rather than invocation count.

Best Practices

• Replace “exactly once” with a boundary-specific statement.
• Assign stable transport and business identities.
• Keep atomic changes within one resource transaction.
• Use Outbox and Inbox patterns across resource boundaries.
• Verify broker features, windows, partitions, and client configuration.
• Make stale-message handling explicit.
• Reconcile external systems after ambiguous outcomes.
• Treat configuration and retention as part of the guarantee.

Observability

Measure each boundary:

• producer attempts, confirms, and ambiguous sends;
• broker duplicate suppression when exposed;
• deliveries and redeliveries;
• inbox duplicates and business-key conflicts;
• stale messages;
• external idempotency hits and uncertain outcomes;
• reconciliation backlog and corrections.

Trace IDs correlate work but are not idempotency keys. Sampling can omit a trace; business correctness cannot depend on telemetry retention.

Verification

Build a failure matrix and inject a stop:

• before and after producer database commit;
• before and after broker confirmation;
• before and after consumer database commit;
• before and after settlement;
• before and after external API acceptance.

For every stop, record the expected replay and prove the business invariant with database and external-system queries. A messaging dashboard alone cannot validate an external effect.

Decision Matrix

Claim	Accurate replacement
“The queue is exactly once”	“Duplicate MessageIds are suppressed for this entity and configured window”
“Kafka makes the workflow exactly once”	“Kafka atomically commits these offsets and output records under this transaction configuration”
“The consumer runs once”	“Repeated delivery produces one committed local state transition per message ID”
“Payments are exactly once”	“One PaymentAttemptId maps to at most one provider capture, with reconciliation for uncertain outcomes”
“Outbox guarantees exactly once”	“State and intent commit atomically; publication is at least once”
“Idempotency solves everything”	“The named repeated operation preserves the defined business invariant”

Relationship to Other Patterns

Transactional Outbox provides atomic producer intent and repeated publication. Idempotent Consumer turns redelivery into one local effect.

Saga defines acceptable cross-service outcomes through forward and compensating actions. Distributed Transactions can provide a stronger atomic boundary only for compatible enlisted resources.

Retry increases attempts. Circuit Breaker bounds correlated failure. Neither defines business uniqueness.

Decision checklist

• Exactly once what: record, delivery, transition, or external effect?
• Which resource owns the atomic boundary?
• What identity remains stable across every retry?
• What is the deduplication scope and retention window?
• Can messages arrive out of order?
• Which effects sit outside broker or database transactions?
• How are ambiguous outcomes reconciled?
• Can failure injection prove the business invariant at every boundary?

Key takeaways

• Scoped infrastructure guarantees are real but do not expand automatically.
• Delivery count and business-effect count are different.
• Stable business identity is required across transport boundaries.
• Trace IDs are not deduplication keys.
• State the observable invariant instead of saying “exactly once.”