Every call over a network fails sometimes. The service restarts, a load balancer drops a connection, a dependency is slow for ten seconds. Code that treats those failures as exceptional ends up with ad-hoc try/catch loops scattered around, each a little different, most of them wrong in some way.

Polly has been .NET’s answer for years. Version 8 rebuilt it around resilience pipelines: a pipeline is a stack of strategies (retry, circuit breaker, timeout, rate limiter, hedging, fallback) that a call passes through. It’s faster than v7, allocates far less, and is integrated with HttpClientFactory, logging and metrics.

To see what each strategy really does, I ran them against a small local ASP.NET Core server with misbehaving endpoints: one that fails a set number of times and then succeeds, one that always returns 503, one that takes five seconds, and one that answers 429 with Retry-After once. Everything below was measured with Polly 8.8 and Microsoft.Extensions.Http.Resilience 10.11 on .NET 10.

Retry: exponential backoff, with jitter

A retry strategy needs three decisions: what to retry, how many times, and how long to wait between attempts.

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
    {
        ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
            .Handle<HttpRequestException>()
            .HandleResult(r => r.StatusCode is HttpStatusCode.ServiceUnavailable
                                            or HttpStatusCode.TooManyRequests),
        MaxRetryAttempts = 4,
        BackoffType = DelayBackoffType.Exponential,
        Delay = TimeSpan.FromMilliseconds(200),
        UseJitter = true,
    })
    .Build();

var response = await pipeline.ExecuteAsync(
    async ct => await http.GetAsync("/flaky", ct), cancellationToken);

Against an endpoint that fails three times and then works:

retry 1 after 196 ms (ServiceUnavailable)
retry 2 after 111 ms (ServiceUnavailable)
retry 3 after 651 ms (ServiceUnavailable)
/flaky → 200 at 1006 ms

What to retry matters most. Only transient failures: connection errors, 503, 429, 408, and 5xx in general if the operation is safe to repeat. Never a 400 or a 404: the same request will fail the same way. And be careful with non-idempotent operations: retrying a POST that timed out may create the order twice, unless the API accepts an idempotency key.

Why jitter: exponential backoff on its own doubles the delay every time, the same way for every client. Here are the delays from three runs each against an endpoint that is down:

jitter=False: 200, 400, 800, 1600 ms
jitter=False: 200, 400, 800, 1600 ms
jitter=False: 200, 400, 800, 1600 ms
jitter=True:  240, 156, 270, 1025 ms
jitter=True:  144, 386, 348, 685 ms
jitter=True:  198, 88, 807, 1037 ms

Without jitter, a thousand clients that failed together retry together, at exactly 200, 400 and 800 ms, hitting a recovering service in synchronised waves. Polly’s jitter uses a decorrelated algorithm: the delays still grow on average, but each client’s sequence is different, so the load spreads out. There’s rarely a reason to turn it off.

Circuit breaker: stop calling a service that is down

Retries help with short blips. When a dependency is down for minutes, retries make things worse: every request waits through its retries before failing, threads and connections pile up, and the struggling service gets hammered. A circuit breaker watches the failure rate and, past a threshold, fails calls immediately for a while:

.AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .HandleResult(r => (int)r.StatusCode >= 500),
    FailureRatio = 0.5,
    MinimumThroughput = 5,
    SamplingDuration = TimeSpan.FromSeconds(10),
    BreakDuration = TimeSpan.FromSeconds(2),
})

Ten calls in a row to an endpoint that always returns 503:

opened at 8 ms for 2 s
10 calls to /down: 503 503 503 503 503 open(0.97 ms) open(0.03 ms) open(0.01 ms) open(0.01 ms) open(0.01 ms)
half-open at 2112 ms
closed at 2117 ms
after 2.1 s, probe → 200

After five failures (the minimum throughput, all failing, above the 50% ratio) the circuit opened. The next five calls never left the process: they threw BrokenCircuitException in hundredths of a millisecond. After the break, the circuit went half-open and let one probe through; the service had recovered, the probe succeeded, and the circuit closed again.

That fast failure is the point. Your code catches BrokenCircuitException and does something sensible straight away: a cached value, a degraded page, a “try again in a minute”. It doesn’t wait through the timeouts of a service that is down.

Advertisement

Timeout: give up, and cancel

A timeout strategy cancels the operation’s CancellationToken when time runs out and throws TimeoutRejectedException:

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddTimeout(TimeSpan.FromSeconds(1))
    .Build();
/slow timed out at 1017 ms
server saw the cancellation: 1 time(s)

The request was actually aborted, and the server’s handler saw its RequestAborted token fire. This works because Polly’s timeout is cooperative: it cancels the token passed into your callback, and HttpClient honours it. If your callback ignores the token (http.GetAsync(url) without ct), the timeout can’t stop anything. Always pass the token through.

The order of strategies matters

A pipeline runs its strategies from the outside in, in the order you add them. Two timeouts in different places mean different things: one outside the retry limits the whole operation, one inside limits each attempt. Same slow endpoint, retrying on timeouts, with and without the outer one:

total timeout outside retry: TimeoutRejectedException at 1004 ms after 2 attempt(s)
only per-attempt timeout:    TimeoutRejectedException at 1405 ms after 3 attempt(s)

With only a per-attempt timeout of 400 ms, three attempts plus the delays took 1.4 seconds. Raise the retries and the time grows with them. The total timeout outside caps the whole thing at one second, however many attempts are left. For calls made while a user waits, you almost always want both: a per-attempt timeout so one stuck connection doesn’t eat the budget, and a total timeout that matches what the caller is willing to wait.

The usual order, outermost first:

  1. Total timeout: the caller’s budget.
  2. Retry: around everything below, so a broken circuit or an attempt timeout can be retried.
  3. Circuit breaker: sees every attempt, so it counts real failures.
  4. Attempt timeout: closest to the call.

The one-liner: the standard resilience handler

For HttpClient, Microsoft.Extensions.Http.Resilience packages exactly that order as a default:

builder.Services.AddHttpClient<CatalogClient>(c => c.BaseAddress = new Uri("https://catalog"))
    .AddStandardResilienceHandler();

I read its defaults from the options it registers:

total timeout 30 s, attempt timeout 10 s,
retries 3, exponential from 2 s, jitter on, honours Retry-After,
breaker: failure ratio 0.1, minimum 100 calls, sampling 30 s, break 5 s,
rate limiter: 1000 concurrent requests, no queue

It also honours Retry-After. An endpoint that answered 429 with Retry-After: 5 was retried after the five seconds the server asked for, not after the backoff’s own two:

/limited (429, Retry-After: 5) → 200 at 5035 ms

The defaults are reasonable for a typical service-to-service call. Tune them per client when they don’t fit: a payment API might deserve no automatic retries on POST at all, and a call made while a user waits needs a total timeout far below 30 seconds:

.AddStandardResilienceHandler(o =>
{
    o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(5);
    o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
    o.Retry.DisableForUnsafeHttpMethods();
});

DisableForUnsafeHttpMethods turns retries off for POST, PATCH, PUT, DELETE and CONNECT, which is the safe default for any API that isn’t idempotent.

Seeing it in production

Polly v8 reports every retry, timeout and circuit transition through ILogger and through metrics (resilience.polly.strategy.events) when the pipeline is built through dependency injection, as AddResiliencePipeline and the HTTP handlers do. Put them on a dashboard. A rising retry count is often the first sign that a dependency is degrading, well before its own alerts fire, and a circuit that opens is something someone should hear about.

A checklist

  • Retry only transient failures, and only idempotent operations, unless the API takes an idempotency key.
  • Keep jitter on. Synchronised retries are a self-inflicted load spike.
  • Use a circuit breaker for dependencies that can be down for minutes, and handle BrokenCircuitException with a fast fallback.
  • Always pass the cancellation token, or timeouts can’t cancel anything.
  • Use two timeouts: total outside the retry, per-attempt inside.
  • Start from AddStandardResilienceHandler for HTTP, then tune it per client.
  • Watch the telemetry. Resilience that nobody sees hides problems until it can’t any more.