Implementing HTTP Resilience by Microsoft

I was working on a project that needed to call multiple webservices, so I wanted to add resilience. Luckily, Microsoft has created a HTTP Resilience package as a wrapper for Polly, adding sensible defaults. In this blog I'll explore how easy it is to add the new package to our services and make it configurable.

Idea

Our solution has 4 components:

  1. We'll use a named HTTP client, as this makes it easier to bind it to our configuration.
  2. We're using the Microsoft.Extensions.Http.Resilience package to implement resilience, configuration and defaults.
  3. We'll create an option class that we'll use to change defaults and bind to the configuration.
  4. We'll create some extension methods to hook it all up and make our setup reusable.

Background

At Wehkamp, we use many micro services; in 99% of the cases the resilience of a dependency is exactly the same for its endpoints. In other words: we can define the resilience per dependent service.

The Microsoft.Extensions.Http.Resilience package has the following defaults:

OrderStrategyDescriptionDefaults
1Rate limiterThe rate limiter pipeline limits the maximum number of concurrent requests being sent to the dependency.Queue: 0
Permit: 1_000
2Total timeoutThe total request timeout pipeline applies an overall timeout to the execution, ensuring that the request, including retry attempts, doesn't exceed the configured limit.Total timeout: 30s
3RetryThe retry pipeline retries the request in case the dependency is slow or returns a transient error.Max retries: 3
Backoff: Exponential
Use jitter: true
Delay:2s
4Circuit breakerThe circuit breaker blocks the execution if too many direct failures or timeouts are detected.Failure ratio: 10%
Min throughput: 100
Sampling duration: 30s
Break duration: 5s
5Attempt timeoutThe attempt timeout pipeline limits each request attempt duration and throws if it's exceeded.Attempt timeout: 10s

These are pretty sensible defaults. The only thing I want to change is setting the Retry.Delay to 100ms.

Example

Let's use the excellent httpstat.us service to get some error responses. The random endpoint will randomly return on of the specified status codes.

namespace Ktt.Resilience.Clients.HttpClients;

public class HttpStatusApiService(HttpClient client)
{
    public async Task<string> Get()
    {
        var path = "/random/200,500-508";
        return await client.GetStringAsync(path);
    }
}

In most cases this request will return an error in the 500 range, causing a retry. To show resilience in action, we'll create an app that will poll our service 4 times:

using ICustomHttpStatusApiClient = Ktt.Resilience.Clients.HttpClients.HttpStatusApiService;
using KiotaHttpStatusClient = Ktt.Resilience.Clients.Kiota.HttpClients.HttpStatus.HttpStatusClient;

namespace Ktt.Resilience.App.Demos;

public class DemoRetry(
    ICustomHttpStatusApiClient customHttpStatusApiClient,
    KiotaHttpStatusClient kiotaHttpStatusClient
)
{
    public async Task RunAsync()
    {
        await Execute("CustomHttpStatusApiClient", customHttpStatusApiClient.Get);
        await Execute("KiotaHttpStatusClient", async () => await kiotaHttpStatusClient.Random.TwoZeroZeroFiveZeroZeroFiveZeroTwoFiveZeroThree.GetAsync() ?? string.Empty);

        Console.WriteLine("");
    }

    private static async Task Execute(string name, Func<Task<string>> execute)
    {
        Console.WriteLine($"Retry Demo for {name}...");

        for (var i = 0; i < 4; i++)
        {
            Console.WriteLine("");
            Console.WriteLine($"Calling {name} iteration {i}");

            try
            {
                var str = await execute();
                Console.WriteLine($"Result: {str}");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error: {ex.Message}");
            }
        }
    }
}

If we don't add the resilience handlers, our program will fail pretty quickly.

Config

Any HttpClient needs to have a URL. I hate hardcoding URLs in my source code, so let's define an options class that stores it. The official way to configure resilience is to use AddStandardResilienceHandler(IConfigurationSection) with a dedicated section containing only resilience properties. That doesn't work for me: the method enables ErrorOnUnknownConfiguration, which rejects BaseUrl since it's not a property of HttpStandardResilienceOptions. You would have to split config across separate sections, which hurts readability when managing multiple services with different SLAs.

I prefer keeping all settings per service together. By inheriting from HttpStandardResilienceOptions, we can create an options class that holds both BaseUrl and the resilience defaults we want to override. Let's set the retry delay to 100ms:

public class HttpClientOptions : HttpStandardResilienceOptions
{
    [Required]
    [Url]
    [RegularExpression(@"^(http|https)://.*$", ErrorMessage = "The URL must start with http:// or https://")]
    public string BaseUrl { get; set; } = string.Empty;

    public HttpClientOptions()
    {
        Retry.Delay = TimeSpan.FromMilliseconds(100);
    }

    public void CopyTo(HttpStandardResilienceOptions other)
    {
        // I hate that this is needed...

        other.AttemptTimeout = AttemptTimeout;
        other.CircuitBreaker = CircuitBreaker;
        other.RateLimiter = RateLimiter;
        other.Retry = Retry;
        other.TotalRequestTimeout = TotalRequestTimeout;
    }
}

Notice the CopyTo method. The standard pipeline reads from a named option {builder-name}-standard, so we need a way to bridge our combined config into that shape. Since .NET's options pattern doesn't map inherited properties back to the base class automatically, we use Configure to inject and copy manually. It's not pretty, but it works, and it keeps every service's config in one place.

Now, we want to override some of the details of the retry specifically for the HttpStatusApi. We want to set the BaseUrl and let's set the number of retries to 7. We don't want to wait too long for every retry, let's set the backoff type to Linear.

{
  "HttpClients": {
    "HttpStatusApi": {
      "BaseUrl": "https://httpstat.us",
      "Retry": {
        "MaxRetryAttempts": 7,
        "BackoffType": "Linear"
      }
    }
  }
}

Dependency injection 1: configuration

Let's create an extension method to bind the config to our HttpClientOptions to a named section. To make things easier, let's name the option also after our section.

public static IServiceCollection AddNamedOptionsForHttpClient(this IServiceCollection services, string sectionName)
{
    return AddNamedOptionsForHttpClient<HttpClientOptions>(services, sectionName);
}

public static IServiceCollection AddNamedOptionsForHttpClient<TOptions>(this IServiceCollection services, string sectionName)
    where TOptions : HttpClientOptions, new()
{
    services
        .AddOptionsWithValidateOnStart<TOptions>(sectionName)
        .BindConfiguration(sectionName)
        .Validate(options =>
        {
            var results = new List<ValidationResult>();
            var context = new ValidationContext(options);
            if (!Validator.TryValidateObject(options, context, results, validateAllProperties: true))
            {
                throw new OptionsValidationException(
                    sectionName,
                    typeof(TOptions),
                    results.Select(r => $"[{sectionName}] {r.ErrorMessage}")
                );
            }
            return true;
        });

    // note: we need to bind {name-standard} to hook up resilience options
    services
        .AddOptions<HttpStandardResilienceOptions>(sectionName + "-standard")
        .Configure((HttpStandardResilienceOptions options, IOptionsMonitor<TOptions> monitor) =>
        {
            // get the other options and copy the values into
            // the resilience options:
            var config = monitor.Get(sectionName);
            config.CopyTo(options);
        });

    return services;
}

We'll need to add a named option to configure our resilience pipeline. When you dive into the code of the package, you'll discover that the AddStandardResilienceHandler will request a named option with the name builder-name-standard. I don't want to register the section twice and I want to reuse the defaults of the HttpClientOptions class. Solution: use a IOptionsMonitor to get the HttpClientOptions and copy the settings to HttpStandardResilienceOptions.

Dependency injection 2: http client with resilience pipeline

Finally, we're ready to setup our named client. We use generics to bind it to the target class and instantiate the options. We reuse the name of the section as name for the client.

public static class HttpClientExtensions
{
    public static IHttpStandardResiliencePipelineBuilder AddHttpClientWithResilienceHandler<TClass>(
        this IServiceCollection services,
        string sectionName
    )
        where TClass : class
    {
        return services.AddHttpClientWithResilienceHandler<TClass, HttpClientOptions>(sectionName);
    }

    public static IHttpStandardResiliencePipelineBuilder AddHttpClientWithResilienceHandler<TClass, TConfig>(
        this IServiceCollection services,
        string sectionName)
        where TClass : class
        where TConfig : HttpClientOptions, new()
    {
        var httpClientBuilder = services
            // bind the configuration section
            .AddNamedOptionsForHttpClient<TConfig>(sectionName)
            // add the HttpClient itself, configure the base URL
            .AddHttpClient<TClass>(sectionName, (serviceProvider, client) =>
            {
                var monitor = serviceProvider.GetRequiredService<IOptionsMonitor<TConfig>>();
                var config = monitor.Get(sectionName);
                if (!string.IsNullOrWhiteSpace(config?.BaseUrl))
                {
                    client.BaseAddress = new Uri(config.BaseUrl);
                }
            });

        // add resilience handler
        return httpClientBuilder.AddStandardResilienceHandler();
    }

Now, we can register it all in our program:

// the order matters!
services.AddTransient<HttpStatusApiService>();
services
    .AddHttpClientWithResilienceHandler<HttpStatusApiService>("HttpClients:HttpStatusApi")
    .Configure(config =>
    {
        config.Retry.OnRetry = async args =>
        {
            Console.WriteLine($"Retry {args.AttemptNumber}: Retrying after {args.RetryDelay} due to {args.Outcome.Result?.StatusCode}");
            await Task.CompletedTask;
        };
    });

When we run our App, we see the following:

Conclusion

The setup is pretty easy and reproducible. I like the fact that Microsoft has bundled the configuration into a single package that we can reuse.

The code is on GitHub, so check it out: code gallery / 12. HTTP and resilience.

Changelog

  • Restructured Config section to explain the official approach first. Fixed poll count, retry delay, and phrasing throughout. Added link to official Polly docs. Added link to .NET options docs. Fixed code include ranges and missing closing braces.
  • Improved the validation exception with the name of the section for better discoverability.
  • Initial article.
expand_less brightness_auto