Transactional Outbox: Close the Dual-Write Gap Without Distributed Transactions

Introduction

The previous article followed a Domain Event from an aggregate to application handlers and then stopped at the service boundary. That boundary hides a dangerous production question:

How does a committed business fact become a durable message without losing it or publishing a fact that never committed?

Consider an order service that changes an order to Placed and publishes orders.order-placed.v1. The database and message broker are independent systems. A normal transaction cannot commit both atomically.

If the service saves the order first, it can crash before publishing. The order exists, but fulfillment never hears about it. If it publishes first, the database commit can fail. Fulfillment acts on an order that does not exist. Retrying either sequence moves the failure window; it does not remove it.

The Transactional Outbox pattern changes the unit of atomic work. The request writes the business change and a message record to the same database transaction. A separate relay later publishes that record to the broker.

The pattern provides atomic persistence, not exactly-once delivery. A relay can publish the same message more than once. Consumers must therefore be idempotent.

The Production Problem

The dual-write gap

The direct implementation looks reasonable:

order.Place(clock.GetUtcNow());

await dbContext.SaveChangesAsync(cancellationToken);
await messagePublisher.PublishAsync(
    OrderPlacedIntegrationV1.From(order),
    cancellationToken);

It contains two commits:

the database commits the order;
the broker confirms the message.

No ordering makes both outcomes correct under every failure.

Sequence	Failure window	Observable result
Save, then publish	Process stops after database commit	Business state exists; message is missing
Publish, then save	Database commit fails after broker confirmation	Message describes state that never committed
Run both concurrently	Either operation can win or fail independently	Outcome is nondeterministic and recovery is unclear
Retry the request	The first attempt may have committed partially	Duplicate orders or duplicate messages unless every boundary is idempotent

This is not primarily a broker availability problem. Retry and Circuit Breaker can control how the publisher reacts to a broker outage, but neither can infer whether an independent database transaction committed.

Why not use one distributed transaction?

Two-phase commit can coordinate resources only when every participant and the deployment environment support the required transaction protocol. It also extends lock duration, couples availability, complicates recovery, and is often unavailable across managed databases and brokers.

The outbox accepts eventual publication instead. It keeps the atomic boundary inside one service-owned database, where local transactions are mature and observable.

That trade is useful when:

a database change must eventually produce an external message;
the database and broker cannot share a transaction;
delayed delivery is acceptable;
duplicate delivery can be handled safely.

It is not a free reliability layer. The relay, backlog, message contract, retention, and consumer deduplication become production responsibilities.

Core Concepts

An outbox row is a durable publication intent. It normally contains:

a stable message identifier;
a stable contract name and version;
the serialized payload;
the occurrence time;
optional correlation, causation, tenant, and trace context;
publication state such as attempt count, next-attempt time, and processed time.

The row belongs in the same database and local transaction as the business state. Writing it to another database recreates the dual-write problem.

A Domain Event is not automatically an Integration Event

The Domain Events article separated internal business facts from external contracts. The outbox is where that distinction becomes operational.

OrderPlaced can be an internal event shaped around the domain model. orders.order-placed.v1 is a versioned contract for other services. Map between them before persistence. Do not serialize an aggregate, an EF Core entity, or a CLR assembly-qualified type name into a public message contract.

public sealed record OrderPlacedIntegrationV1(
    Guid MessageId,
    Guid OrderId,
    Guid CustomerId,
    decimal TotalAmount,
    DateTimeOffset OccurredAtUtc);

The MessageId identifies the logical message across retries. It must not change on each publication attempt.

The relay is an asynchronous delivery mechanism

The request does not wait for the broker. A relay:

selects unpublished rows;
claims a bounded batch;
publishes each message;
waits for the broker client’s success confirmation;
marks the row processed;
retries failures with bounded backoff.

The relay can run in the application process, a separate worker, or database-specific change-data-capture infrastructure. The choice changes deployment and scaling, not the atomicity rule.

Publication is at least once

There is an unavoidable failure window:

the broker confirms the publish;
the relay stops before marking the row processed.

After restart, the relay publishes the row again. Marking it processed before broker confirmation creates the opposite failure: a message can be lost.

Therefore:

the producer guarantees that a committed intent remains retryable;
the broker provides its configured durability and delivery semantics;
the relay provides repeated publication attempts;
the consumer must make repeated processing safe.

No component in that list independently provides end-to-end exactly-once business effects.

Ordering is scoped, not global

Some workflows need messages for one aggregate to retain commit order. A global sequence is usually unnecessary and expensive.

If order matters, define the scope explicitly—commonly aggregate, account, tenant, or broker partition key—and ensure the relay and broker preserve ordering within that scope. Multiple relay instances, retries, and broker partitions can reorder messages unless the design constrains them.

Consumers should still reject stale state transitions or use versions when business correctness depends on order. Transport ordering alone is not a business invariant.

How It Works

Request path

The application loads or creates an aggregate.
The aggregate changes state and records a Domain Event.
The application maps externally relevant facts to versioned Integration Events.
EF Core inserts the business changes and outbox rows in one SaveChangesAsync transaction.
The request returns after the local commit. Broker availability is not on the request’s commit path.

EF Core wraps one SaveChanges call in a transaction when the provider supports transactions. If the application spans several SaveChanges calls or mixes other database operations, it must define and test the transaction boundary explicitly.

Relay path

A worker claims a small batch of due rows.
It publishes with the persisted message ID and contract metadata.
It marks a row processed only after the broker client reports successful publication.
On failure, it records a bounded error summary, increments attempts, and schedules another attempt.
After a configured terminal policy, it quarantines the row for operator review rather than retrying forever at full speed.

Claiming must be atomic when more than one relay instance runs. The mechanism is database-specific: row locks with skip-locked semantics, an update that returns claimed rows, or a lease with an expiry are common options. A plain SELECT followed by an UPDATE lets two workers publish the same row concurrently. Consumer idempotency limits the damage, but correct claiming prevents unnecessary duplicates and load.

Recovery path

Recovery should not depend on reconstructing messages from application logs. The outbox is the durable work queue:

a broker outage grows the backlog;
a relay restart resumes due rows;
an expired lease makes abandoned work claimable;
an operator can inspect and replay quarantined rows under an audited procedure.

The service must retain rows long enough to diagnose delayed consumers and replay safely, then delete or archive processed rows in bounded batches.

Architecture Diagram

Transactional Outbox request and relay flow

The two transactions are intentionally separate. The producer transaction proves that business state and publication intent agree. The consumer transaction must prove that deduplication and the consumer’s state change agree.

.NET Implementation

The following implementation shows the important boundaries. It uses EF Core for local atomicity and an application-owned publisher abstraction for the broker-specific client. Broker acknowledgments, confirms, transactions, and routing differ by product and must be configured through the chosen SDK.

Persist a stable envelope

public sealed class OutboxMessage
{
    public Guid Id { get; private set; }
    public string Contract { get; private set; } = null!;
    public string Payload { get; private set; } = null!;
    public DateTimeOffset OccurredAtUtc { get; private set; }
    public DateTimeOffset AvailableAtUtc { get; private set; }
    public DateTimeOffset? ProcessedAtUtc { get; private set; }
    public int AttemptCount { get; private set; }
    public string? LastError { get; private set; }

    private OutboxMessage() { }

    public static OutboxMessage Create(
        Guid id,
        string contract,
        string payload,
        DateTimeOffset occurredAtUtc) =>
        new()
        {
            Id = id,
            Contract = contract,
            Payload = payload,
            OccurredAtUtc = occurredAtUtc,
            AvailableAtUtc = occurredAtUtc
        };
}

Use Id as a primary key. Index the relay query, normally on publication state and AvailableAtUtc. Keep the contract name stable and independent of .NET namespace or assembly changes.

Map and save in one EF Core unit

public interface IHasDomainEvents
{
    IReadOnlyCollection<IDomainEvent> DomainEvents { get; }
    void ClearDomainEvents();
}

public sealed class OrdersDbContext(
    DbContextOptions<OrdersDbContext> options) : DbContext(options)
{
    public DbSet<Order> Orders => Set<Order>();
    public DbSet<OutboxMessage> OutboxMessages => Set<OutboxMessage>();

    public override async Task<int> SaveChangesAsync(
        CancellationToken cancellationToken = default)
    {
        var aggregates = ChangeTracker
            .Entries<IHasDomainEvents>()
            .Select(entry => entry.Entity)
            .Where(entity => entity.DomainEvents.Count > 0)
            .ToArray();

        var events = aggregates
            .SelectMany(entity => entity.DomainEvents)
            .ToArray();

        var alreadyTracked = ChangeTracker
            .Entries<OutboxMessage>()
            .Select(entry => entry.Entity.Id)
            .ToHashSet();

        foreach (var domainEvent in events)
        {
            var message = Map(domainEvent);

            if (alreadyTracked.Add(message.Id))
                OutboxMessages.Add(message);
        }

        var result = await base.SaveChangesAsync(cancellationToken);

        foreach (var aggregate in aggregates)
            aggregate.ClearDomainEvents();

        return result;
    }

    private static OutboxMessage Map(IDomainEvent domainEvent) =>
        domainEvent switch
        {
            OrderPlaced placed => OutboxMessage.Create(
                placed.EventId,
                "orders.order-placed.v1",
                JsonSerializer.Serialize(
                    new OrderPlacedIntegrationV1(
                        placed.EventId,
                        placed.OrderId,
                        placed.CustomerId,
                        placed.TotalAmount,
                        placed.OccurredAtUtc)),
                placed.OccurredAtUtc),
            _ => throw new UnmappedDomainEventException(
                domainEvent.GetType())
        };
}

The stable event ID makes a retried save deterministic. The alreadyTracked check prevents the same DbContext from adding a second entity with the same key if an execution strategy replays SaveChangesAsync. The database primary key provides the final uniqueness boundary.

This override intentionally fails for an unmapped event. Silently dropping an externally relevant fact is worse than rejecting the transaction. In a larger system, an explicit mapper registry can replace the switch, but it must retain deterministic IDs and stable contract names.

Do not combine a manually started transaction with an EF Core retrying execution strategy without following EF Core’s execution-strategy transaction procedure. The complete transaction must be replayed as one unit, and an ambiguous commit outcome requires verification.

Run a scoped relay

BackgroundService is registered as a singleton. Create a dependency-injection scope for each iteration before resolving an EF Core DbContext, which is normally scoped.

public sealed class OutboxRelay(
    IServiceScopeFactory scopeFactory,
    TimeProvider clock,
    ILogger<OutboxRelay> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await using var scope = scopeFactory.CreateAsyncScope();
            var dispatcher = scope.ServiceProvider
                .GetRequiredService<OutboxDispatcher>();

            var count = await dispatcher.DispatchDueBatchAsync(
                batchSize: 100,
                stoppingToken);

            if (count == 0)
                await Task.Delay(
                    TimeSpan.FromMilliseconds(500),
                    clock,
                    stoppingToken);
        }
    }
}

The dispatcher should depend on two explicit ports:

public interface IOutboxStore
{
    Task<IReadOnlyList<ClaimedOutboxMessage>> ClaimDueBatchAsync(
        int batchSize,
        TimeSpan leaseDuration,
        CancellationToken cancellationToken);

    Task MarkProcessedAsync(
        Guid messageId,
        CancellationToken cancellationToken);

    Task ScheduleRetryAsync(
        Guid messageId,
        DateTimeOffset nextAttemptAtUtc,
        string errorCode,
        CancellationToken cancellationToken);
}

public interface IIntegrationEventPublisher
{
    Task PublishAsync(
        string contract,
        Guid messageId,
        ReadOnlyMemory<byte> payload,
        CancellationToken cancellationToken);
}

These are application contracts, not claims about a broker SDK. The IOutboxStore implementation must use the selected database’s atomic claim or lease mechanism. PublishAsync must return only after the chosen broker client reports the configured success condition.

Do not hold a database transaction open while waiting on the broker. Claim or lease quickly, commit the claim, publish, then update the row. A lease must expire so another worker can recover work after a crash.

Retry without creating a hot loop

Retry transient publish failures with exponential backoff and jitter, bounded by an operational policy. Persist AttemptCount and AvailableAtUtc; an in-memory delay is lost on restart.

Classify failures:

transient transport, throttling, or broker availability failures are retryable;
invalid contracts, oversized messages, missing destinations, and authorization failures usually need quarantine or configuration repair;
cancellation during shutdown should release or let the lease expire without incrementing a permanent-failure counter.

A Circuit Breaker can stop a relay instance from continuously attacking an unavailable broker. It does not replace persisted scheduling because process restarts erase in-memory circuit state.

Common Mistakes

Publishing inside the database transaction

The broker does not participate in the local transaction. Holding database locks while awaiting the network increases contention and still leaves an ambiguous commit window.

Persist the intent in the transaction; publish after it commits.

Marking processed before confirmation

If the process stops after marking but before the broker accepts the message, the row appears complete and is never retried.

Mark processed only after the broker client’s configured success confirmation. Accept that a crash immediately afterward creates a duplicate.

Treating the outbox as exactly-once delivery

The outbox prevents a committed business change from having no durable publication intent. It does not prevent repeated publication, broker redelivery, or repeated consumer execution.

Use stable message IDs and idempotent consumers. If the consumer changes a database, record the processed message ID and apply the business change in the same consumer transaction.

Generating a new message ID on every retry

Consumers cannot recognize duplicates if the logical message receives a new identifier each time. Create the ID with the Domain Event or outbox row and preserve it through every attempt.

Using CLR type names as contracts

Renaming a namespace or assembly then breaks deserialization across services. Use an explicit contract name and version, and test compatibility independently of the producer’s internal model.

Polling without an index or a bound

An unbounded scan of a growing outbox competes with business queries and increases lock pressure. Query a bounded batch through a supporting index, and purge processed rows incrementally.

Scaling relays without atomic claims

Two workers can select and publish the same row at the same time. Implement provider-specific locking or leasing before scaling replicas. Consumer idempotency remains required because atomic claims cannot close the confirmation-to-mark crash window.

Retrying poison messages forever

A permanently invalid message can consume every batch and hide newer work. Apply a terminal policy, quarantine it with its original payload and metadata, alert the owner, and provide an audited replay path.

Hiding publication lag behind request success

The request succeeding means the business state and outbox intent committed. It does not mean downstream services have processed the event. API contracts and user-facing state must not imply synchronous cross-service completion.

Best Practices

Make the guarantee explicit

Document the contract in operational terms:

business state and outbox intent commit atomically;
publication is asynchronous and at least once;
consumers must deduplicate;
ordering is guaranteed only for named keys, if at all;
maximum acceptable publication lag and retention are service objectives, not pattern defaults.

Observe the queue as a production subsystem

Logs should be structured and low-cardinality. Include message.id, contract, attempt count, result, lease identifier, and a bounded error code. Do not log full payloads by default; they may contain personal or confidential data.

At minimum, measure:

count of pending and quarantined rows;
age of the oldest pending row;
publish attempts, successes, and failures by contract and reason;
claim-to-confirm latency;
end-to-end event age at publish time;
cleanup duration and rows removed.

Backlog count alone is insufficient. A busy healthy system may always have pending rows. Oldest-message age reveals whether delivery is falling behind. Alert on sustained age beyond the service objective, terminal failures, and absence of relay activity when eligible work exists.

Preserve trace continuity

Persist correlation and causation identifiers with the outbox message. If trace context is propagated, create producer-side messaging spans when the relay sends and let consumers create processing spans or links according to the chosen OpenTelemetry instrumentation.

Do not keep the original HTTP request span open until asynchronous delivery. The request commit, relay publication, and consumer processing are separate operations connected by message context.

OpenTelemetry messaging semantic conventions are still evolving. Pin instrumentation versions and treat emitted attribute names as a versioned telemetry contract.

Test the failure windows

Unit tests are not enough. Run integration tests against the actual database provider and broker client:

Atomicity: force the database transaction to fail and verify that neither business state nor outbox row persists.
Broker outage: commit a business change with the broker unavailable; restore it and verify eventual publication.
Confirmation crash: stop the relay after broker confirmation but before marking; verify duplicate publication with the same message ID and one consumer business effect.
Concurrent claims: run multiple relays and verify that leases prevent concurrent ownership while expired claims recover.
Poison message: publish an invalid row and verify bounded retry, quarantine, alerting, and continued progress for later rows.
Retention: purge a realistic volume in batches and observe lock time, log growth, and query latency.

The third test proves the boundary of the guarantee: duplicates are expected, but duplicate business effects are not.

Operate replay as a controlled change

Replaying can repeat external effects, overload consumers, and violate ordering. Require a defined message range, reason, owner, rate limit, and verification query. Preserve the original message ID when replaying the same logical event. Use a new ID only when publishing a new corrective event with different business meaning.

Decision Matrix

Situation	Use Transactional Outbox?	Reason or alternative
One database update must eventually notify another service	Yes	It closes the database-to-broker dual-write gap
Domain Event triggers only local work in the same transaction	Usually no	Execute the local handler inside the transaction; no external publish exists
Loss is acceptable for ephemeral telemetry	No	Direct best-effort export is simpler; do not turn diagnostics into business state
The source of truth is already an append-only event log	Maybe	Publish from the log or use its subscription mechanism instead of duplicating it
Database change capture is available and operationally owned	Maybe	Change-data capture can relay committed rows without application polling, but contract mapping and consumer idempotency remain
The command requires synchronous downstream confirmation	No, not by itself	Use a synchronous call with explicit timeout and failure semantics; an outbox cannot provide immediate confirmation
Consumer cannot tolerate or deduplicate duplicates	No	Redesign the consumer contract or choose a truly transactional boundary; the outbox is at least once
One process and one database own all affected state	Usually no	A local transaction is simpler and stronger
Global ordering across all messages is mandatory	Usually no	Revisit the requirement; global serialization creates a throughput and availability bottleneck

Use the pattern when the cost of a missing message exceeds the cost of operating a durable relay and idempotent consumers.

Do not use it merely because the system has microservices. If no database change and broker publication must agree, there is no dual write to close.

Relationship to Other Patterns

Domain Events define the internal facts selected for publication. Idempotent Consumer owns duplicate-safe processing after at-least-once delivery.

Retry, Timeout, and Backoff bounds one publish attempt; persisted outbox scheduling survives restarts and prolonged outages. Saga uses an outbox at each committed workflow transition but owns compensation and deadlines separately.

Dead Letter Queue covers consumer-side quarantine. Exactly Once explains why local atomicity, stable identity, idempotency, and reconciliation must be composed across the full path.

Decision checklist

Does one committed database change require an eventual external message?
Can both records be written in the same local database transaction?
Is delayed delivery acceptable and visible to callers?
Does every message have a stable ID and explicit versioned contract?
Can consumers apply duplicate deliveries without duplicate business effects?
Is relay claiming safe with multiple instances and recoverable after a crash?
Are retry, quarantine, retention, and audited replay policies defined?
Can operators alert on oldest pending age and verify end-to-end recovery?
Is required ordering defined by a bounded key rather than assumed globally?
Is a local transaction, direct publish, or change-data-capture feed simpler for this case?

Key takeaways

The outbox closes the producer’s dual-write gap; it does not create exactly-once delivery.
Broker confirmation must precede marking a row processed, so duplicates are unavoidable.
Stable IDs and idempotent consumers turn duplicate delivery into one business effect.
Publication lag is part of the service’s correctness and must be measured.
Database-specific claiming and broker-specific confirmation semantics require integration testing.