Production Messaging Checklist: Prove Reliability Before Release

Introduction

The series began with Domain Events and followed a production fact through Circuit Breaker, Transactional Outbox, Idempotent Consumer, Saga, resilience policies, transaction boundaries, exactly-once claims, and Dead Letter Queue recovery.

This final article turns those patterns into one release decision:

Can the team prove what happens when a message is duplicated, delayed, reordered, rejected, or stranded between independent systems?

A checklist is useful only when every item has evidence. “The broker is reliable” is not evidence. A configuration export, uniqueness constraint, fault test, dashboard, alert, and rehearsed replay procedure are.

The Production Problem

The happy path is easy to demonstrate:

1. API accepts an order.
2. Producer publishes an event.
3. Consumer updates a projection.
4. Dashboard shows success.

Production failures happen between those steps:

• the database commits but publication is delayed;
• broker confirmation is lost;
• the consumer commits but settlement fails;
• an old message arrives after a new one;
• a poison message retries continuously;
• credentials expire;
• a replay repeats external effects;
• telemetry says the queue is empty while the DLQ grows.

No single pattern covers the full path. Release readiness requires an explicit guarantee and recovery model at every boundary.

Core Concepts

Start with the business invariant

State the outcome in testable language:

• “One PaymentAttemptId produces at most one captured payment.”
• “Every committed order reaches Confirmed, Rejected, or Manual Review within the service objective.”
• “Projection state never moves to an older aggregate version.”

Do not begin with “exactly once,” “event-driven,” or a broker product name.

Define ownership transfer

At each hop, name when responsibility moves:

• application to outbox after database commit;
• relay to broker after configured confirmation;
• broker to consumer after successful settlement;
• consumer to downstream outbox after local commit.

Ambiguous transfer requires retry plus idempotency or reconciliation.

Treat configuration as code or evidence

Durability, retention, delivery count, lock duration, partitioning, ordering, dead-letter routing, and duplicate detection are part of the application guarantee. Capture and review them with the same care as C#.

Recovery is part of correctness

A system is not reliable merely because it retries. It must converge or expose owned manual work. Backlog queries, DLQ procedures, Saga state, and reconciliation close the path.

How It Works

Use the checklist during design, before release, after broker or client upgrades, and after incidents.

For each control:

1. mark Pass, Gap, or Not applicable;
2. link one piece of current evidence;
3. name the owner;
4. record the consequence of failure;
5. create a bounded remediation for every Gap.

Do not approve production based on document presence alone. Execute the failure tests against the actual database provider, broker tier, client SDK, and deployment topology.

Architecture Diagram

.NET Implementation

Contract envelope

A bounded envelope makes operational metadata explicit:

public sealed record MessageEnvelope<T>(
    Guid MessageId,
    string Contract,
    int ContractVersion,
    DateTimeOffset OccurredAtUtc,
    string CorrelationId,
    string? CausationId,
    string? TraceParent,
    T Body);

Do not put credentials, mutable runtime types, or an entire aggregate into the envelope. Validate maximum size and required fields before domain handling.

TraceParent helps telemetry correlation. It is not a business key. MessageId identifies the logical message. A separate field such as PaymentAttemptId identifies a business operation.

Consumer outcome model

Avoid one generic catch block:

public enum ConsumeOutcome
{
    Processed,
    Duplicate,
    Stale,
    RetryableFailure,
    PermanentFailure,
    ManualReview
}

Map each outcome to explicit broker settlement and telemetry. The adapter, not the domain handler, should translate outcomes into complete, abandon, retry scheduling, or dead-letter operations.

Health is more than connectivity

A broker connection health check can pass while no messages progress. Expose operational health from:

• oldest eligible outbox age;
• oldest unprocessed consumer message age when observable;
• last successful relay and consumer activity;
• DLQ count and age;
• Saga instances beyond deadline;
• reconciliation backlog.

Do not make every backlog threshold a pod liveness failure. Restarting healthy workers during broker outage can amplify instability. Use readiness and alerting according to the failure the signal represents.

Common Mistakes

Reviewing only producer code

Reliability is end to end. Broker and consumer settlement configuration can invalidate producer assumptions.

Treating defaults as stable contracts

SDK retry, idempotence, acknowledgment, prefetch, and timeout defaults change across versions. Pin packages and test effective behavior after upgrades.

Using queue depth as the only metric

Depth can be zero while messages sit in a DLQ, a transfer DLQ, an outbox, or an in-flight lock. Age and terminal-state metrics reveal stuck work.

Mixing technical and business failures

PaymentDeclined is not a transport exception. It should advance business state, not consume transient retries.

Unbounded cardinality

Message IDs, order IDs, customer IDs, exception text, and raw destinations should not become metric labels.

Replaying without reconciliation

Queue drain proves transport movement. It does not prove correct payments, reservations, shipments, or projections.

No capacity for recovery

After an outage, live traffic and backlog compete. Without replay rate limits and headroom, recovery causes another incident.

Best Practices

1. Business and ownership

• The workflow has one testable business invariant.
• Every producer, consumer, Saga, DLQ, and reconciliation job has an owner.
• Terminal success, rejection, expiry, and manual-review states are defined.
• Customer-visible status does not claim downstream completion prematurely.

Evidence: architecture decision, state model, ownership entry, and query showing all nonterminal work.

2. Contracts and identity

• Contract name and version are explicit and independent of CLR type names.
• Message ID remains stable across producer retries and replay of the same logical message.
• Business idempotency keys are separate from transport identity.
• Compatibility, size, required fields, and unknown-field behavior are tested.
• Schema evolution and retirement have a consumer migration plan.

Evidence: contract tests, sample envelopes, compatibility matrix, and broker size configuration.

3. Producer atomicity

• Business state and publication intent commit in one local transaction when they must agree.
• Outbox IDs are deterministic and unique.
• Relay claims are atomic across replicas and recover after lease expiry.
• Processed outbox rows have retention and bounded cleanup.
• Publish is marked complete only after the broker’s documented confirmation.

Evidence: database constraints, provider-specific claim test, confirmation test, and oldest-outbox dashboard.

4. Broker configuration

• Durability, replication, retention, time to live, maximum delivery count, and dead-letter routing are explicit.
• Publisher confirmation and consumer settlement modes match the loss-versus-duplicate decision.
• Partition or session key matches required ordering scope.
• Prefetch and concurrency stay within consumer capacity and lock duration.
• Authentication, authorization, secret rotation, and network failure behavior are tested.

Evidence: exported configuration, policy-as-code, permission test, and broker failover exercise.

5. Consumer correctness

• Consumer assumes redelivery.
• Inbox identity and local business update commit together.
• A unique constraint resolves concurrent duplicates.
• Stale and out-of-order messages have a domain policy.
• External effects use idempotency keys, local outbox, or reconciliation.
• Broker completion occurs only after durable success.

Evidence: concurrent duplicate test, commit-before-settlement crash test, version test, and external-effect reconciliation query.

6. Retry and overload

• Transient, permanent, business, stale, and unknown failures are classified.
• Retry count and delay fit a total deadline or durable workflow objective.
• Backoff uses jitter and honors bounded server delay guidance.
• Retry ownership across SDK, application, mesh, and upstream caller is known.
• Concurrency limits and Circuit Breaker protect dependencies.
• Recovery traffic is rate-limited.

Evidence: effective client configuration, timing tests, attempt-amplification dashboard, and fault-load result.

7. Saga and long-running work

• Saga state and version are durable.
• Forward and compensation commands are idempotent.
• Deadlines survive process restart.
• Late replies and timeout races are tested.
• Irreversible steps and the point of no return are explicit.
• Failed compensation reaches visible manual review.

Evidence: state-machine tests, persisted deadline query, compensation drill, and manual-review runbook.

8. Dead-letter and replay

• DLQ is distinct from automatic delayed retry.
• Stable reason codes and safe descriptions are recorded.
• Alerting includes count, rate, and oldest age.
• Replay requires a repaired cause, bounded filter, owner, and rate.
• Same logical messages preserve their IDs.
• Transform, discard, and replay decisions are audited.

Evidence: injected poison-message test, alert, canary replay record, and business reconciliation.

9. Observability

• Logs include contract, message ID, correlation, attempt, outcome, and stable failure code.
• Metrics separate logical messages from delivery or publish attempts.
• Outbox age, consumer lag, duplicate rate, DLQ age, Saga age, and reconciliation backlog are visible.
• Trace context crosses message boundaries using supported instrumentation.
• Metrics avoid high-cardinality identifiers.
• Alerts map to an owned response procedure.

Evidence: dashboard links, alert tests, sample trace, and label-cardinality review.

10. Recovery and change

• Backup and restore effects on message identity and offsets are understood.
• Broker or SDK upgrade behavior is tested before rollout.
• Rollback does not reintroduce an incompatible consumer.
• Reconciliation can find missing, duplicate, stale, and stranded business state.
• Capacity exists to drain backlog while serving live traffic.
• Incident exercises cover ambiguous send, commit, and settlement outcomes.

Evidence: upgrade test, rollback plan, reconciliation output, recovery capacity test, and incident exercise record.

Decision Matrix

Release state	Decision
Business invariant, ownership, and failure outcomes are undefined	Do not release
Producer dual write exists without Outbox or supported transaction	Do not release for loss-intolerant workflows
At-least-once consumer lacks idempotency	Do not release for duplicate-sensitive effects
No DLQ owner or replay procedure	Release only if loss is explicitly acceptable and documented
Ordering assumed but no partition/version design exists	Do not release ordering-sensitive state
Fault tests pass but dashboards and alerts are missing	Limited rollout only with active manual observation
All critical controls have current evidence and recovery has been exercised	Eligible for production rollout
Control is not applicable with a documented reason	Accept; do not add unused machinery

Relationship to Other Patterns

• Domain Events define meaningful internal facts.
• Circuit Breaker limits correlated dependency failure.
• Transactional Outbox closes the producer dual-write gap.
• Idempotent Consumer turns redelivery into one local business effect.
• Saga coordinates durable cross-service outcomes.
• Retry, Timeout, and Backoff bounds transient attempts.
• Distributed Transactions explains the stronger but coupled 2PC boundary.
• Exactly Once Is a Myth replaces slogans with scoped guarantees.
• Dead Letter Queue defines quarantine and controlled replay.

Use the smallest combination that covers the actual boundaries. A single database does not need a Saga. Ephemeral telemetry may not need an Outbox. A duplicate-sensitive payment does.

Decision checklist

• Is the business invariant stated without infrastructure slogans?
• Does every commit-to-message boundary have atomic intent?
• Does every at-least-once consumer have idempotent effects?
• Are ordering, deadlines, and compensation explicit where required?
• Are broker and SDK configurations captured and tested?
• Can telemetry locate old, duplicate, dead-lettered, and stranded work?
• Can operators replay or reconcile without repeating unsafe effects?
• Has the team injected failure at every ownership-transfer point?
• Does each Gap have an owner and release decision?

Key takeaways

• Configuration and operations are part of message correctness.
• Queue depth alone cannot prove workflow health.
• Every retry and replay needs identity and capacity limits.
• Recovery must verify business state, not only transport state.
• “Pass” requires current evidence.