Circuit Breaker: Prevent Cascading Failures Before They Take Down Your Microservices

Why It Matters

A downstream service does not have to be completely unavailable to take down its callers. It only has to become slower than the rate at which new work arrives.

Assume an API normally completes 500 payment lookups per second in 80 ms. The payment service degrades and starts responding in 8 seconds. Requests continue to arrive. Each caller retains an in-flight HTTP operation, request state, buffers, sockets, and often a database connection or other upstream resources. Queues grow. Cancellation and timeout work increases. Thread-pool starvation can appear when any layer blocks synchronously, while purely asynchronous code can still exhaust connection pools, memory, concurrency limits, and request deadlines.

The payment service is now one incident. The caller’s resource exhaustion turns it into several incidents:

1. Payment-dependent endpoints slow down.
2. Unrelated endpoints compete for the same process resources.
3. Callers retry, multiplying traffic against the impaired dependency.
4. Upstream services time out and start their own retries.
5. Latency and failure propagate through the service graph.

This is a cascading failure. A Circuit Breaker limits the propagation. After observing enough relevant failures, it stops sending calls for a bounded period and fails them immediately. That protects the caller and gives the dependency room to recover.

A Circuit Breaker does not make an operation succeed. It changes an expensive, repeated, remote failure into a cheap, explicit, local failure.

Background

Retry alone can amplify an incident

Retry is appropriate when another attempt has a meaningful chance of succeeding: a connection was reset during a deployment, one replica failed, or a short lock conflict cleared. It is dangerous when failures are correlated across requests.

With two retries, 1,000 logical requests can produce up to 3,000 physical attempts. Exponential backoff spreads those attempts over time; jitter prevents synchronized retry waves. Neither removes the extra load. If the dependency is saturated, retries consume the capacity required for recovery.

Retry can also violate correctness. Retrying a POST after the caller loses the response may execute the operation twice unless the endpoint provides idempotency semantics. Microsoft.Extensions.Http.Resilience therefore lets applications disable retries for unsafe HTTP methods. That is still a coarse rule: some PUT operations are idempotent, while a nominally safe read may be expensive enough that retrying it is harmful.

Use Retry to handle isolated transient faults. Use Circuit Breaker to stop repeating a fault pattern. They solve different problems.

What a Circuit Breaker measures

A breaker classifies each execution as handled or unhandled. In Polly v8, ShouldHandle defines that classification. During a sampling window, the breaker compares the handled-failure count with total sampled throughput.

It opens only when both conditions are true:

• observed throughput reaches MinimumThroughput; and
• the handled failure ratio reaches FailureRatio.

This two-part rule avoids opening on one failure during low traffic. It also creates a low-traffic trap: if minimum throughput is never reached within the sampling duration, the breaker never opens, even when every request fails.

The breaker is shared mutable state. Its scope determines what failure domain it represents. A circuit per request remembers nothing and is useless. One global circuit for unrelated hosts lets one dependency block another. In most HTTP systems, use a long-lived pipeline per logical dependency, endpoint group, tenant partition, or other independently recoverable failure domain.

Problem

Consider an order API calling a payment authorization endpoint. During a database saturation event in the payment service:

• latency climbs before the error rate;
• caller-side attempt timeouts produce TimeoutRejectedException;
• some responses are 503 Service Unavailable;
• invalid cards still return 422 Unprocessable Entity;
• the service may use 429 Too Many Requests to ask callers to reduce load.

Counting every non-success response as breaker failure is wrong. A 422 is a valid business outcome and says nothing about dependency health. Counting only exceptions is also wrong because repeated 503 responses are explicit availability failures.

The breaker needs a dependency-specific outcome model:

Outcome	Retry candidate?	Breaker failure?	Reason
DNS, connection, or transport failure	Usually	Yes	The dependency cannot be reached reliably
Per-attempt timeout	Sometimes	Yes	The dependency exceeded the caller’s latency budget
408, 429, or 5xx	Context-dependent	Usually	The server is unavailable, overloaded, or asking for less traffic
401 or 403	No	Usually no	Configuration or identity failure will not heal through waiting
404	Usually no	No	Often a valid resource outcome
409 or 422	Domain-specific	Usually no	Usually a business or concurrency outcome
Caller cancellation	No	No	It is not evidence that the dependency is unhealthy

“Usually” is deliberate. A breaker predicate is part of the service contract, not a universal HTTP status list.

Core Concepts

Closed

Closed is normal operation. Calls reach the dependency, and the breaker records outcomes selected by ShouldHandle. A handled outcome still returns or throws to the surrounding pipeline; Circuit Breaker observes failures but does not consume them.

When minimum throughput and the failure-ratio threshold are met within the sampling window, the circuit opens.

Open

Open means fail fast. Polly does not invoke the inner HTTP operation. It rejects calls with BrokenCircuitException; the exception can include RetryAfter, which indicates how long the caller should wait before trying again.

Open is not a background health-check loop. In Polly, expiration of BreakDuration makes a later execution eligible to move the circuit to Half-Open. If no call executes the pipeline, no probe occurs.

Half-Open

Half-Open is a controlled recovery test. An eligible execution reaches the dependency. A successful probe closes the circuit and normal sampling resumes. A handled failure reopens it for another break duration.

The probe must represent dependency health. A cached response, local validation failure, or operation routed to a different host cannot prove that the protected endpoint recovered.

State timeline

For a breaker configured with a 20-second sampling duration, 50% failure ratio, minimum throughput 20, and 30-second break duration, a simplified incident can look like this:

12:00:00  CLOSED     Sampling begins
12:00:08  CLOSED     20 executions: 11 handled failures, 9 successes
12:00:08  OPEN       55% >= 50%; later calls fail locally
12:00:38  HALF-OPEN  Next execution becomes the recovery probe
12:00:39  OPEN       Probe receives 503; wait another break duration
12:01:09  HALF-OPEN  Next execution probes again
12:01:09  CLOSED     Probe succeeds; normal traffic resumes

The exact transition time depends on executions, not a scheduler.

Internal Model

Ratio, volume, and time form one decision

Treat the main settings as a system:

• FailureRatio expresses how much relevant failure the caller can tolerate.
• SamplingDuration controls how quickly old evidence ages out and how much traffic is observed.
• MinimumThroughput protects against statistically weak samples.
• BreakDuration controls how long the caller stops applying load.
• ShouldHandle defines what “failure” means.

A 50% ratio does not mean “open after five errors.” At 100 requests per second with a 30-second sample, it may represent hundreds of failures. At one request per minute with minimum throughput 20, it may never trigger.

Start configuration from observed request rate, latency budget, dependency recovery behavior, and error-budget policy. Do not copy thresholds between a high-volume catalog read and a low-volume settlement operation.

A breaker is local, not distributed consensus

Each application instance normally owns its own breaker state. Ten caller replicas can open at different times. During recovery, each replica may probe independently. This is usually desirable: the breaker has no network dependency and isolates the resources of its own process.

It also means dashboards must aggregate without pretending there is one global state. A dependency can receive recovery traffic from several caller replicas. If that traffic is unsafe, combine the breaker with dependency-side admission control and caller-side concurrency limiting; do not turn circuit state into a distributed lock.

Scope is an architecture decision

Choose a scope that matches independent recovery:

• one circuit per logical dependency when all requests share the same capacity and failure behavior;
• separate circuits per host or endpoint pool when destinations fail independently;
• separate circuits for operations with materially different latency or failure semantics;
• partitioned circuits only when tenant or region isolation justifies the state and telemetry cardinality.

Avoid a circuit per URL containing resource IDs, customer IDs, or trace IDs. It fragments evidence, grows state, and creates unbounded metric dimensions.

How It Works

Where the breaker belongs in an HTTP call flow

This is the default ordering used by AddStandardResilienceHandler, from outermost to innermost: rate limiter, total timeout, retry, Circuit Breaker, and attempt timeout.

Ordering changes semantics:

• Retry outside Circuit Breaker: the breaker sees every physical attempt. It can open during one logical request, and a later retry may receive BrokenCircuitException.
• Circuit Breaker outside Retry: the breaker sees one final outcome after retries finish. It opens more slowly and does not measure the load amplification inside Retry.
• Attempt timeout inside Circuit Breaker: the breaker can count TimeoutRejectedException for slow attempts.
• Total timeout outside Retry: one deadline bounds the complete operation, including backoff.
• Fallback outermost: it can translate final failures, including an open circuit, but only when a semantically valid substitute exists.

There is no context-free correct ordering. For outbound HTTP, the standard handler’s ordering is a strong baseline because the breaker observes direct attempts against the dependency. Change it only with an explicit reason and load test.

How the strategies cooperate

Strategy	Primary protection	Interaction with Circuit Breaker
Timeout	Bounds waiting time	Converts pathological latency into a classified outcome the breaker can observe
Retry	Recovers isolated transient faults	Adds attempts; ordering decides whether the breaker sees each attempt or only the final result
Concurrency limiter (Bulkhead in Polly v7 terminology)	Caps in-flight outbound work	Protects caller resources before and while the breaker gathers enough evidence
Rate limiter	Controls request admission over time	Prevents a healthy or recovering dependency from being flooded
Fallback	Supplies an alternate semantic result	Handles final failure; it must not turn stale or fabricated data into apparent success

Polly v8 replaces the old Bulkhead policy with rate-limiting strategies; AddConcurrencyLimiter is the direct counterpart for bounding parallel work.

The breaker cannot replace any of these. Before it opens, slow calls still consume resources, so timeout and concurrency isolation remain essential. When it opens, callers still need an explicit failure contract or a safe fallback.

Design Guidance

Start with the standard HTTP resilience handler

For a new typed client, start with the maintained integration:

using Microsoft.Extensions.Http.Resilience;

builder.Services
    .AddHttpClient<PaymentClient>(client =>
    {
        client.BaseAddress = new Uri(
            builder.Configuration["Payments:BaseUrl"]
            ?? throw new InvalidOperationException(
                "Payments:BaseUrl is required."));
    })
    .AddStandardResilienceHandler(options =>
    {
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(4);
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(1);

        options.Retry.MaxRetryAttempts = 2;
        options.Retry.BackoffType = DelayBackoffType.Exponential;
        options.Retry.UseJitter = true;
        options.Retry.DisableForUnsafeHttpMethods();

        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(20);
        options.CircuitBreaker.MinimumThroughput = 20;
        options.CircuitBreaker.FailureRatio = 0.50;
        options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);
    });

These values illustrate a budget, not a recommendation to copy. The total timeout must include every attempt and backoff delay. Disabling retries for unsafe methods is a safe default for payment commands; an operation with a durable idempotency key may justify a narrower retry rule after contract review.

The standard handler defaults and handled outcomes can change across package versions. Pin the package, review release notes, and test the effective behavior when upgrading.

Use a custom handler when outcome semantics differ

The following pipeline protects a read-only payment-status client. It counts transport failures, Polly attempt timeouts, 408, 429, and 5xx responses. It does not count caller cancellation or business 4xx outcomes.

using System.Net;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using Polly.Timeout;

builder.Services
    .AddHttpClient<PaymentStatusClient>(client =>
    {
        client.BaseAddress = new Uri(
            builder.Configuration["Payments:BaseUrl"]
            ?? throw new InvalidOperationException(
                "Payments:BaseUrl is required."));
    })
    .AddResilienceHandler("payments-status", static pipeline =>
    {
        static bool IsTransient(HttpResponseMessage response) =>
            response.StatusCode is HttpStatusCode.RequestTimeout
                or HttpStatusCode.TooManyRequests
            || (int)response.StatusCode >= 500;

        var transientOutcomes =
            new PredicateBuilder<HttpResponseMessage>()
                .Handle<HttpRequestException>()
                .Handle<TimeoutRejectedException>()
                .HandleResult(IsTransient);

        pipeline.AddConcurrencyLimiter(
            permitLimit: 100,
            queueLimit: 0);

        pipeline.AddTimeout(TimeSpan.FromSeconds(4));

        pipeline.AddRetry(new HttpRetryStrategyOptions
        {
            ShouldHandle = transientOutcomes,
            MaxRetryAttempts = 2,
            Delay = TimeSpan.FromMilliseconds(200),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true
        });

        pipeline.AddCircuitBreaker(
            new HttpCircuitBreakerStrategyOptions
            {
                ShouldHandle = transientOutcomes,
                SamplingDuration = TimeSpan.FromSeconds(20),
                MinimumThroughput = 20,
                FailureRatio = 0.50,
                BreakDuration = TimeSpan.FromSeconds(30)
            });

        pipeline.AddTimeout(TimeSpan.FromSeconds(1));
    });

Strategies execute in registration order from outermost to innermost. The two timeouts have different jobs: four seconds bounds the logical call; one second bounds each physical attempt.

queueLimit: 0 rejects excess work instead of hiding overload in another queue. If queuing is allowed, its wait time must fit inside the end-to-end latency budget.

Do not stack AddStandardResilienceHandler and AddResilienceHandler on the same client unless duplicate pipelines are intentional and their combined semantics are tested. Microsoft recommends one resilience handler per client.

Treat fallback as domain behavior

A fallback is safe only when the substitute preserves the endpoint’s contract:

• a product-description read may return a clearly marked cached snapshot;
• a recommendation panel may be omitted;
• a payment authorization must not be replaced with “approved”;
• an inventory reservation must not be converted into success because the dependency is unavailable.

Prefer implementing domain fallback after the typed client returns a classified failure. This keeps transport resilience separate from business degradation and makes stale-data metadata explicit.

Real-world Application

Configuration procedure

For each dependency and operation class:

1. Define the deadline. Start with the caller’s end-to-end latency objective, then allocate attempt and total timeout budgets.
2. Classify outcomes. Record which exceptions and status codes indicate dependency health, business rejection, bad configuration, or caller cancellation.
3. Measure traffic. Select a sampling duration and minimum throughput that produce meaningful evidence at normal and low load.
4. Bound load. Set concurrency from downstream capacity and caller resource budgets. Keep queues small or zero.
5. Choose recovery behavior. Set break duration from observed recovery time, deployment behavior, and overload characteristics.
6. Validate retry safety. Require idempotency evidence for mutating operations and cap total amplification.
7. Define caller behavior. Map open-circuit failure to a retryable API error, cached response, deferred workflow, or explicit feature unavailability.

Store the rationale with configuration. A number without its traffic assumptions becomes dangerous after scale changes.

Monitoring and alerting

Monitor outcomes, not only state:

• rate and ratio of handled dependency failures;
• circuit transitions by pipeline and dependency;
• time spent Open and repeated Open → Half-Open → Open cycles;
• BrokenCircuitException rejection rate;
• retry attempts per logical request;
• attempt and total latency histograms;
• concurrency rejections and queue time;
• downstream HTTP status and transport-exception rates;
• caller saturation: active requests, connection-pool pressure, memory, and thread-pool queue length.

Alerting on every Open event creates noise during controlled tests and isolated instance failures. Alert on user impact or sustained/repeated transitions, then attach circuit state as diagnostic evidence.

Polly v8 emits logs under the Polly logger and metrics under the Polly meter. Relevant metric events include OnCircuitOpened, OnCircuitHalfOpened, OnCircuitClosed, OnRetry, OnTimeout, and OnRateLimiterRejected. resilience.polly.strategy.events is a counter; Polly also exposes attempt-duration and pipeline-duration histograms.

Keep metric attributes bounded. Pipeline name, strategy name, dependency, region, and operation class are useful. Raw URL, customer ID, order ID, exception message, and trace ID are not metric dimensions.

OpenTelemetry

Export Polly’s Meter through the application’s OpenTelemetry metrics pipeline and keep normal HttpClient instrumentation enabled. The two views answer different questions:

• HTTP client spans show which physical attempts reached the dependency and how they ended.
• Polly metrics and logs show retries, local rejections, and circuit transitions.

An Open rejection sends no HTTP request, so there may be no downstream client span for that execution. Record the logical operation in the caller span and add a bounded event or status indicating resilience rejection. Do not create fake HTTP client spans for calls that never occurred.

Correlate logs with the current Activity trace and span IDs. Use a stable operation key such as payments.get-status, not a resource identifier. Avoid logging every open-circuit rejection at Error during a high-volume incident; aggregate metrics and sampled traces preserve the signal without causing a second telemetry incident.

Failure Cases

Minimum throughput is unreachable

A low-volume settlement client handles three calls per minute, but MinimumThroughput is 100 in a 30-second window. The dependency can fail continuously and the circuit remains Closed.

Verify threshold reachability against the lowest traffic period that still needs protection. For sparse workloads, timeout and concurrency isolation may matter more than a statistical breaker.

The breaker counts business failures

Including all 4xx responses opens the circuit because customers enter expired cards. This hides valid business feedback and converts healthy service behavior into availability failure.

Make the handled-outcome table part of contract review.

Break duration is too short

Hundreds of caller instances probe a recovering dependency every few seconds. The dependency fails each probe and never regains capacity.

Increase break duration based on recovery evidence, reduce recovery concurrency with load controls, and ensure the dependency has its own admission control. A longer fixed delay trades recovery speed for stability.

Break duration is too long

The dependency recovers in seconds, but callers reject traffic for ten minutes. Availability is now limited by the protection mechanism.

Track Open duration and compare it with actual dependency recovery. Consider dynamic break duration only when its behavior remains predictable and testable.

Retry ordering is accidental

An outer Circuit Breaker sees only the final failure after an inner Retry exhausts three attempts. It opens slowly while the dependency receives amplified traffic. Reversing the order makes it observe each attempt and may open during one logical call.

Document whether thresholds apply to logical requests or physical attempts. Load-test that exact order.

Every request owns a breaker

Creating the pipeline inside a request handler resets state on every call. No circuit accumulates enough evidence to open.

Build pipelines once through dependency injection or a pipeline provider. Do not instantiate them in the hot path.

One breaker covers unrelated destinations

A shared pipeline protects several hosts. One host fails and the circuit blocks healthy hosts.

Partition by independent failure domain, but keep the partition key bounded.

Fallback conceals an outage

A stale cache returns HTTP 200 without age or provenance. Availability appears perfect while users make decisions on invalid data.

Expose freshness and degradation, meter fallback use, and fail when stale data violates the business contract.

Trade-offs

A Circuit Breaker adds state, tuning, new failure modes, and local rejection behavior. It deliberately rejects some calls that might have succeeded during the Open interval. Independent instance state makes fleet behavior probabilistic rather than globally synchronized. Half-Open probes add recovery traffic. Aggressive thresholds create false positives; conservative thresholds allow more damage before opening.

Those costs are justified when repeated calls to a failing dependency threaten caller resources or downstream recovery. They are not justified merely because an operation is remote.

When Not to Use Circuit Breaker

Do not add a breaker when:

• the operation is local and cheap;
• failures are validation or domain outcomes rather than dependency-health signals;
• traffic is too sparse for a ratio-based sample to be meaningful;
• the caller already has strict concurrency limits and a short deadline, and fail-fast state adds no material protection;
• every attempt must be made for correctness and the workflow already uses durable asynchronous delivery with controlled consumers;
• the dependency is a replicated client library that already performs endpoint health tracking, and another breaker would create conflicting state;
• there is no defined behavior for BrokenCircuitException.

For durable commands, a queue with bounded consumers may be better. For load shedding, use rate or concurrency limiting. For one slow operation, use Timeout. For isolated transient faults, use a small Retry. Choose the mechanism that addresses the observed failure.

Decision Matrix

Situation	Circuit Breaker	Complementary control	Key condition
High-volume synchronous HTTP dependency can become slow	Yes	Attempt timeout + concurrency limiter	Thresholds derived from physical-attempt traffic
Low-volume administrative API	Maybe	Timeout	Minimum throughput must be reachable
Idempotent read with sporadic connection resets	Maybe	Small jittered Retry	Break only if failures become correlated
Non-idempotent payment command	Often	Timeout + concurrency limiter	Retry only with proven idempotency
Durable event publication through an outbox	Usually no at producer boundary	Bounded background consumer + broker retry/dead-lettering	Preserve durable delivery
Business rejection such as insufficient funds	No	Explicit domain result	Rejection is not dependency failure
Optional enrichment with a valid cache	Yes	Explicit stale-data fallback	Mark freshness and meter fallback
Shared client routes to independent regions	Per region or endpoint pool	Routing + rate limiting	Partition key remains bounded

Verification

Validate the pipeline in layers.

Deterministic strategy tests

Use a controllable time source or Polly’s documented testing support where available. Drive enough handled outcomes through one long-lived pipeline to meet both MinimumThroughput and FailureRatio.

Expected observations:

1. Before the threshold, handled responses or exceptions pass through unchanged.
2. After the threshold, the next calls throw BrokenCircuitException without invoking the fake transport.
3. After break duration, a successful execution closes the circuit.
4. A failed Half-Open probe reopens it.
5. Caller cancellation does not contribute to dependency failure counts.

Pipeline-order test

Make the first two attempts time out and the third succeed. Assert the number of transport invocations, total elapsed budget, retry count, and breaker telemetry. This proves whether the breaker observes attempts or logical calls.

Load and fault test

Against a non-production dependency:

• inject latency above the attempt timeout;
• inject a controlled 503 ratio;
• sustain expected peak caller traffic;
• verify concurrency never exceeds its limit;
• verify the circuit opens within the intended detection interval;
• verify transport traffic drops while Open;
• restore the dependency and verify stable recovery without a probe surge.

Record request rate, failure ratio, latency percentiles, retry amplification, Open duration, and resource saturation. A functional test alone cannot prove protection against cascading failure.

Telemetry test

Capture one trace that retries, one logical call rejected by an open circuit, and one successful recovery probe. Verify:

• HTTP spans exist only for actual network attempts;
• resilience events carry stable pipeline and strategy names;
• logs correlate with the caller trace;
• metrics contain no unbounded identifiers;
• dashboards distinguish original attempts, retries, and local rejections.

Best Practices

• Share one long-lived breaker per independently recoverable dependency scope.
• Define handled outcomes from the dependency contract.
• Set timeout and concurrency protection before relying on breaker state.
• Derive thresholds from measured request volume and recovery behavior.
• Decide explicitly whether the breaker measures physical attempts or logical requests.
• Retry only safe or idempotent operations, with a small bounded attempt count and jitter.
• Keep total timeout outside Retry and attempt timeout inside it.
• Make open-circuit behavior part of the caller’s API contract.
• Use fallbacks only when they preserve business semantics and expose degradation.
• Monitor transitions, rejections, retries, latency, and saturation together.
• Keep telemetry dimensions bounded and correlate detail through traces.
• Revalidate settings after traffic, topology, timeout, or retry changes.
• Test failure and recovery under load before rollout.

Decision checklist

• Which outcomes prove dependency failure rather than business rejection?
• Does one long-lived breaker represent one independently recoverable dependency?
• Can normal and low traffic reach MinimumThroughput inside the sampling window?
• Does the pipeline measure physical attempts or final logical outcomes?
• Do total timeout, attempt timeout, retry, and concurrency limits share one budget?
• What explicit result does the caller receive while the circuit is open?
• Can a load test prove reduced downstream traffic and stable recovery?

Key takeaways

• A breaker converts repeated remote failure into bounded local rejection.
• Timeout and concurrency limiting protect the caller before the breaker opens.
• Retry recovers isolated faults but can amplify correlated failure.
• Thresholds must come from measured attempt traffic and recovery behavior.
• Fallback is safe only when it preserves business semantics and exposes degradation.