Jaegerize my .NET Core setup!

Adding observability to micro services is vital if you want to discover bottle necks. In this blog, I'll show how we implemented Jaeger in .NET Core to observe incoming and outgoing requests. We'll also use a Jaeger decorator to observe spans in classes.

Jaeger describes itself as:

As on-the-ground microservice practitioners are quickly realizing, the majority of operational problems that arise when moving to a distributed architecture are ultimately grounded in two areas: networking and observability. It is simply an orders of magnitude larger problem to network and debug a set of intertwined distributed services versus a single monolithic application.

Why Jaeger

1. Intro
2. Run Jaeger locally for testing
3. .NET Core Setup
   1. NuGet packages
   2. Environment variables
   3. Jaeger configuration
   4. Startup: hook it up in DI
4. Jaeger decorator
   1. Jaeger Decorator Implementation Example
5. Comments

Run Jaeger locally for testing

To do some local development, you'll want to run Jaeger locally. The documentation describes how to run the stack locally with a single Docker container:

The UI runs on http://localhost:16686. Super easy.

.NET Core Setup

Before we can start, we'll need to set some things up: NuGet Packages, environment variables and the Jaeger dependency injection (DI) configuration.

NuGet packages

Never go anywhere without a package. We'll be using the following packages:

As always we're using Scrutor to write a decorator.

Environment variables

By default most Jaeger clients use environment variables (although I noticed that some implementation might use slightly different variable names). Locally we'll be using:

• JAEGER_AGENT_HOST=localhost
• JAEGER_AGENT_PORT=6831
• JAEGER_SAMPLER_TYPE=const
• JAEGER_SAMPLER_PARAM=1

This means we're tracing all requests. You don't want to do this in production, cause it might generate too much data.

Jaeger configuration

I like to have my configuration in a separate class to keep my Startup class cleaner. This class will initialize the Jaeger tracer:

using Jaeger;
using Jaeger.Samplers;
using Jaeger.Senders;
using Jaeger.Senders.Thrift;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using OpenTracing;
using OpenTracing.Contrib.NetCore.Configuration;
using OpenTracing.Util;

public static class JaegerConfiguration
{
    public static void AddJaeger(this IServiceCollection services)
    {
        // Use "OpenTracing.Contrib.NetCore" to automatically generate spans for ASP.NET Core, Entity Framework Core, ...
        // See https://github.com/opentracing-contrib/csharp-netcore for details.
        services.AddOpenTracing();

        // Adds the Jaeger Tracer.
        services.AddSingleton<ITracer>(serviceProvider =>
        {
            var serviceName = Program.Name;
            var loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();

            // This is necessary to pick the correct sender, otherwise a NoopSender is used!
            Jaeger.Configuration.SenderConfiguration.DefaultSenderResolver = new SenderResolver(loggerFactory)
                .RegisterSenderFactory<ThriftSenderFactory>();

            // This will log to a default localhost installation of Jaeger.
            var tracer = new Tracer.Builder(serviceName)
                .WithLoggerFactory(loggerFactory)
                .Build();

            // Allows code that can't use DI to also access the tracer.
            if (!GlobalTracer.IsRegistered())
            {
                GlobalTracer.Register(tracer);
            }

            return tracer;
        });


        services.Configure<AspNetCoreDiagnosticOptions>(options =>
        {
            options.Hosting.IgnorePatterns.Add(context => context.Request.Path.Value.StartsWith("/status"));
            options.Hosting.IgnorePatterns.Add(context => context.Request.Path.Value.StartsWith("/metrics"));
            options.Hosting.IgnorePatterns.Add(context => context.Request.Path.Value.StartsWith("/swagger"));
        });
    }

}

Note: we're skipping the health check, metrics and documentation end points.

Startup: hook it up in DI

Add the following to your Startup class DI:

public void ConfigureServices(IServiceCollection services)
{
    services.AddJaeger();
}

Congratulations, Jaeger now works with you .NET Core application. By default it includes spans for your controllers and your outgoing HTTP requests. In the next section I'll be showing how to create a decorator, to log more information.

Jaeger decorator

First, make sure you've copied the Decorator base class from here. The base class takes care of getting the class name from the decorated object. Next, you can copy this JaegerDecorator:

using OpenTracing;
using System;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;

/// <summary>
/// Implements Jaeger tracing as a decorator.
/// </summary>
/// <typeparam name="TDecorated">The decorated type.</typeparam>
public abstract class JaegerDecorator<TDecorated> : Decorator<TDecorated>
{
    private ITracer _tracer;

    protected JaegerDecorator(TDecorated decorated, ITracer tracer) : base(decorated)
    {
        _tracer = tracer ?? throw new ArgumentNullException(nameof(tracer));
    }

    protected virtual IScope StartScope(string methodName)
    {
        return _tracer
            .BuildSpan($"{DecoratedClassName}.{methodName}")
            .WithTag("method", methodName)
            .WithTag("class", DecoratedFullClassName)
            .StartActive(true);
    }

    protected async Task Decorate(
        Func<Task> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            await action();
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }
    protected async Task Decorate(
        Func<IScope, Task> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            await action(scope);
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected async Task<TReturn> Decorate<TReturn>(
        Func<Task<TReturn>> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            return await action();
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected async Task<TReturn> Decorate<TReturn>(
        Func<IScope, Task<TReturn>> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            return await action(scope);
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected TReturn Decorate<TReturn>(
        Func<TReturn> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            return action();
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected TReturn Decorate<TReturn>(
        Func<IScope, TReturn> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            return action(scope);
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected void Decorate(
        Action action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            action();
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected void Decorate(
        Action<IScope> action,
        [CallerMemberName] string methodName = "")
    {
        using var scope = StartScope(methodName);

        try
        {
            action(scope);
        }
        catch (Exception ex)
        {
            HandleException(ex, scope);
            throw;
        }
    }

    protected virtual void HandleException(Exception ex, IScope scope)
    {
        scope.Span.SetTag("error", true);
        scope.Span.Log($"Exception: {ex.Message}");
    }
}

Note: the code uses the simple using syntax - a C# 8.0 feature.

The basis of the class is the ITracer object. It allows us to generate new spans. When an exception is caught, we'll set the tag error=true, which will show up in the Jaeger UI like this:

Jaeger Decorator Implementation Example

The JaegerDecorator exposes an IScope object that can be used to log additional information in the request. It is optional, so you don't have to use it:

public interface IMyInterface
{
    bool InOutMethod(int x);

    void VoidMethod();
}

public class MyInterfaceLatencyDecorator : JaegerDecorator<IMyInterface>, IMyInterface
{
    protected MyInterfaceLatencyDecorator(IMyInterface decorated, ITracer tracer) : base(decorated, tracer)
    {
    }

    public bool InOutMethod(int x) =>
        Decorate((scope) =>
        {
            scope.Span.SetTag("x", x);
            return Decorated.InOutMethod(x);
        });


    public void VoidMethod() => Decorate(Decorated.VoidMethod);
}