Saga: Choose Choreography or Orchestration for Long-Running Consistency

Introduction

The previous articles established reliable facts, publication intent, and duplicate-safe consumption. Those mechanisms move one message safely. They do not answer a larger question:

Who owns a business operation that spans several services and may take minutes, hours, or human intervention to finish?

An order workflow may create an order, reserve inventory, authorize payment, and arrange shipment. Each service commits independently. If shipment fails after payment succeeds, no database rollback can cross those boundaries.

A Saga models the workflow as a sequence of local transactions plus explicit recovery actions. The main architecture choice is whether participants react through choreography or a dedicated component directs them through orchestration.

The Production Problem

Assume an API returns 202 Accepted for order placement. During processing:

1. Orders creates Order.
2. Inventory reserves stock.
3. Payments authorizes funds.
4. Shipping rejects the address.

The system is now partially complete. The order exists, stock is reserved, and funds are authorized. “Roll back” is not a valid technical instruction:

• each service owns its database;
• another transaction may have observed the intermediate state;
• external providers may have accepted irreversible work;
• compensation can fail or produce a different business result.

The system needs durable workflow state, time limits, repeatable commands, and a defined terminal outcome.

Core Concepts

A saga is not an ACID transaction

A Saga is a distributed state machine made of local transactions. It provides eventual movement toward defined outcomes. It does not provide isolation across services.

Other requests can observe intermediate states. Concurrent sagas can conflict. Protect domain invariants with reservations, versions, semantic locks, or explicit pending states rather than pretending the whole workflow is one serializable transaction.

Compensation is a new business action

Compensation does not erase history. Releasing inventory after a failed order is a new fact. Voiding an authorization may differ from refunding a captured payment. Cancellation fees or nonrefundable work may apply.

Every forward step needs a failure policy:

• retry the same operation;
• compensate prior work;
• choose an alternative;
• cross a point of no return;
• wait for human resolution.

Compensation can also fail and must be idempotent, observable, and resumable.

Choreography distributes control

In choreography, each participant consumes an event, performs a local transaction, and publishes the next event.

Benefits:

• no central workflow runtime;
• participants remain loosely coupled to a controller;
• simple, linear flows can stay small.

Costs:

• workflow state is spread across logs and services;
• event chains become difficult to reason about;
• timeouts, compensation, and global status need explicit ownership;
• a new participant can create hidden cycles or ambiguous completion.

Choreography is not “no orchestration.” Control still exists; it is encoded across event reactions.

Orchestration centralizes workflow decisions

An orchestrator stores saga state and sends commands to participants. Participants return results through replies or events.

Benefits:

• one explicit state machine;
• clear timeout, compensation, and status ownership;
• easier operational queries and controlled evolution.

Costs:

• the orchestrator becomes critical workflow infrastructure;
• it knows participant command contracts;
• poor design can pull business rules out of participant domains;
• throughput and availability require durable state and partitioning.

The orchestrator owns process policy. Participants still own their invariants.

Pivot and irreversible steps

Some workflows have a point after which compensation is no longer meaningful. Place irreversible work late when possible. Before the pivot, use compensable steps. After it, prefer idempotent retryable steps that drive the saga to completion.

This is business design, not framework configuration.

How It Works

Choreographed flow

1. Orders commits Pending and an OrderSubmitted outbox message.
2. Inventory consumes it idempotently, commits a reservation, and publishes InventoryReserved.
3. Payments reacts and publishes PaymentAuthorized or PaymentRejected.
4. Orders or another explicit owner interprets terminal events and updates customer-visible status.
5. Failure events trigger compensating reactions.

The architecture must name who detects missing events and elapsed deadlines. A collection of event handlers does not automatically form a recoverable saga.

Orchestrated flow

1. The orchestrator creates a durable saga instance with a correlation ID.
2. It sends ReserveInventory.
3. On InventoryReserved, it persists the transition and sends AuthorizePayment.
4. On a rejection or timeout, it persists compensation state and sends ReleaseInventory.
5. It reaches a terminal state only after required forward or compensation replies arrive.

Each state transition and outgoing command should commit atomically, usually through an outbox owned by the orchestrator.

Architecture Diagram

.NET Implementation

The durable state model matters more than the mediator or broker library.

public enum OrderSagaStatus
{
    ReservingInventory,
    AuthorizingPayment,
    ArrangingShipping,
    CompensatingPayment,
    CompensatingInventory,
    Confirmed,
    Rejected,
    ManualReview
}

public sealed class OrderSaga
{
    public Guid Id { get; private set; }
    public Guid OrderId { get; private set; }
    public OrderSagaStatus Status { get; private set; }
    public int Version { get; private set; }
    public DateTimeOffset DeadlineUtc { get; private set; }
    public string? FailureCode { get; private set; }

    public void On(PaymentAuthorized message)
    {
        if (Status == OrderSagaStatus.ArrangingShipping)
            return;

        if (Status != OrderSagaStatus.AuthorizingPayment)
            throw new InvalidSagaTransition(Status, message.GetType());

        Status = OrderSagaStatus.ArrangingShipping;
        Version++;
    }

    public void On(ShippingRejected message)
    {
        if (Status != OrderSagaStatus.ArrangingShipping)
            throw new InvalidSagaTransition(Status, message.GetType());

        FailureCode = message.ReasonCode;
        Status = OrderSagaStatus.CompensatingPayment;
        Version++;
    }
}

Persist an optimistic concurrency token so two replies cannot advance the same instance silently. Store deadlines as data; an in-memory timer disappears during deployments.

An orchestration handler should commit the state transition and next command together:

public async Task HandleAsync(
    MessageEnvelope<PaymentAuthorized> envelope,
    CancellationToken cancellationToken)
{
    if (await inbox.HasProcessedAsync(
        ConsumerName,
        envelope.MessageId,
        cancellationToken))
        return;

    var saga = await dbContext.OrderSagas.SingleAsync(
        x => x.Id == envelope.Body.SagaId,
        cancellationToken);

    saga.On(envelope.Body);

    outbox.Add(new ArrangeShipping(
        MessageId: Guid.NewGuid(),
        SagaId: saga.Id,
        OrderId: saga.OrderId));

    inbox.Record(ConsumerName, envelope.MessageId);
    await dbContext.SaveChangesAsync(cancellationToken);
}

In production, the inbox record, saga state, and outbox message must share one local transaction. A unique inbox constraint resolves concurrent redelivery. The saga version resolves conflicting valid messages. The outbox relay sends the next command after commit.

Timeout processing is another message:

public sealed record SagaDeadlineReached(
    Guid MessageId,
    Guid SagaId,
    int ExpectedVersion,
    DateTimeOffset DueAtUtc);

When handled, compare ExpectedVersion and current status. A late timeout must not compensate a saga that already advanced.

Common Mistakes

Calling compensation “rollback”

Compensation has its own failure modes and business meaning. Record it as durable forward progress.

Building choreography without a workflow owner

If no component can answer “Where is order 123?” or detect a missing step, the workflow is not production-operable.

Centralizing participant business rules

The orchestrator may decide to request a refund. Payments decides whether refunding is valid and how its ledger changes.

Using request/response timeouts as saga deadlines

An HTTP timeout bounds one call. A saga deadline is durable business state that survives restarts and may trigger compensation hours later.

Reusing one retry rule for business rejection

CardDeclined is a completed business decision, not a transient fault. Route it to an alternate or compensation path instead of retrying blindly.

Ignoring late and duplicate replies

Compensation can race with a delayed success. Validate state and version on every transition. Define whether late success is ignored, compensated, or sent to manual review.

Assuming events imply loose coupling

Participants remain coupled to contract meaning, order, timing, and compensation behavior. Asynchronous transport changes when coupling is observed, not whether it exists.

Best Practices

• Model the saga as an explicit state machine with named terminal states.
• Persist correlation ID, current state, version, deadlines, failure code, and last transition.
• Make every forward and compensation command idempotent.
• Use inbox and outbox transactions at each participant boundary.
• Place irreversible steps late and document the pivot.
• Prefer orchestration when the workflow has many branches, deadlines, compensation, or audit requirements.
• Prefer choreography for small, stable flows with clear terminal ownership.
• Provide a manual-review state instead of infinite retries.

Observability

Log saga ID, business ID, transition, prior state, next state, message ID, attempt, and failure code. Avoid full message payloads.

Measure:

• active sagas by state;
• age of the oldest saga in each nonterminal state;
• transition latency and failure rate;
• compensation count, duration, and failure;
• deadline expirations;
• manual-review backlog.

Trace each message send and process operation. Use the saga ID as searchable structured context, not a high-cardinality metric label. Long-running sagas should not be represented as one open span; connect asynchronous spans with propagated context and links.

Verification

Test the state machine independently, then run broker-and-database integration tests for:

• duplicate and out-of-order replies;
• crash after state commit but before command publication;
• concurrent replies against one saga version;
• timeout racing with success;
• compensation failure and resumption;
• deployment while deadlines remain pending.

The expected result is one valid terminal business outcome or a visible manual-review state—never silent limbo.

Decision Matrix

Condition	Choreography	Orchestration
Two or three linear, stable steps	Strong candidate	Often unnecessary
Many branches and compensations	Becomes difficult to follow	Strong candidate
Durable deadlines and human steps	Requires a separate owner	Natural fit
Participants must evolve independently	Reduces controller coupling	Version command contracts carefully
Central workflow visibility required	Must build correlation views	State is explicit
Very high partitionable throughput	Event flow can distribute naturally	Partition orchestrator state by saga ID
Workflow policy changes frequently	Changes spread across services	Policy changes concentrate in orchestrator
One service already owns the process	Choreography may obscure ownership	Usually clearer

Do not use a Saga when all required state can remain inside one service and one local transaction. Fix an incorrect service boundary before adding distributed workflow machinery.

Relationship to Other Patterns

Domain Events capture participant facts. Transactional Outbox publishes each committed transition reliably. Idempotent Consumer makes repeated commands and replies safe.

Circuit Breaker, Retry, Timeout, and Backoff handle technical faults within a step. They do not decide business compensation.

Two-phase commit tries to keep all participants inside one atomic outcome. A Saga instead exposes intermediate states and resolves failure through domain actions. A Dead Letter Queue can hold a failed transport message, but the saga must remain in a corresponding recoverable state.

Decision checklist

• Does the workflow truly cross service-owned transaction boundaries?
• Are intermediate states acceptable and visible?
• Is each forward step paired with retry, compensation, alternative, or manual handling?
• Are irreversible steps placed deliberately?
• Can every command and compensation run more than once safely?
• Are saga state, inbox, and outbox committed atomically?
• Are deadlines durable and protected from late-message races?
• Can operators identify every nonterminal saga and its next action?
• Is choreography still understandable, or does orchestration better match ownership?

Key takeaways

• Compensation is a new business transaction, not erasure.
• Choreography distributes control; orchestration makes it explicit.
• Neither approach provides isolation across participant databases.
• Durable state and deadlines matter more than the chosen messaging framework.
• A visible manual-review state is safer than indefinite automated retry.