Unit Testing complex validations (study)

In the blog Let’s combine Data Annotation Validation with FluentValidation, I've explored how to use FluentValidation with data annotations. Let's explore how a combination of the two can be tested trough unit testing. This is meant as a study, so there it has no other point than to look at code and "see" what it does.

First we'll come up with some rules on the type of validation that we should use. Then we'll introduce a test case. I'll discuss some common ways to make testing easier and finally we'll discuss the various tests we should throw at our case.

Rules for validation: when to use what?

Let's first define some rules for the types of validations we can do on our object:

Favor attributes over FluentValidation, as they can be read by tools like OpenAPI.
Properties that are required should use the [Required] attribute.
Properties that are conditionally required should use FluentValidation.
Properties that need a certain format should use the [RegularExpression] attribute.
Properties that need multiple regular expressions should use FluentValidation (the attribute does not support multiple regular expressions).
If you need to reuse validations, consider attributes for improved readability.
For validations that use multiple attributes, use FluentValidation.
Remember: a validationContext is an implementation of IServiceProvider, so you can use it to resolve services through dependency injections.

Captain Barbossa explaining to Jack Sparrow that the code is more like "guidelines" rather than actual rules. — Well... "rules", it's more like guidelines.

A test case

We need something to work with. I've extracted this example class from of our internal provisioning projects I'm working on. It is the input model for provisioning applications. It has some interesting aspects:

The Type property supports multiple types of applications, based on the type, different validations are required.
We have some generic properties, like: Cpu, Environment, DockerHubRepo, ImageTag and Ram.
Some types may use commands, which require valid Command and Postfix properties.
Some types require cron specific properties: like Schedule.

public class ComplexApplication : IValidatableObject
{
    [Required]
    public string Name { get; set; } = default!;

    [Required, Team]
    public string Team { get; set; } = default!;

    [Required, Environment]
    public string Environment { get; set; } = default!;

    public ComplexApplicationType Type { get; set; }

    [Required, MinLength(3), MaxLength(75)]
    public string DockerHubRepo { get; set; } = default!;

    [Required]
    public string ImageTag { get; set; } = default!;

    [Required, RegularExpression("^\\d+m$")]
    public string Cpu { get; set; } = default!;

    [Required, RegularExpression("^\\d+Mi$")]
    public string Ram { get; set; } = default!;

    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public string? Command { get; set; } = default!;

    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public string? Schedule { get; set; } = default!;

    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    [RegularExpression(@"^$|^(?!.*(cron|site|service))([a-z0-9-]*)$", ErrorMessage = "The value must be lower-kebab-case and may not contain the words cron, site or service.")]
    public string? Postfix { get; set; } = default!;

    public IEnumerable<ValidationResult> Validate(ValidationContext validationContext)
    {
        var dockerHubService = validationContext.GetRequiredService<IDockerHubService>();

        var v = new InlineValidator<ComplexApplication>();

        v.RuleFor(x => x.DockerHubRepo)
            .NotEmpty()
            .MustAsync(dockerHubService.Exists)
            .WithMessage("The DockerHub repository does not exist.");

        var isTypeWithCommand = Type is ComplexApplicationType.ApplicationWithCommand or ComplexApplicationType.CronJobWithCommand;
        if (isTypeWithCommand)
        {
            v.RuleFor(x => x.Command).NotEmpty();
            v.RuleFor(x => x.Command)
                .Must(cmd => string.IsNullOrEmpty(cmd) || !cmd.Contains(".sh") || cmd.Contains("tini"))
                .WithMessage("Script files (.sh) may only be executed when tini is used.");

            v.RuleFor(x => Postfix)
                .NotEmpty();
        }
        else
        {
            v.RuleFor(x => x.Command).Empty();
            v.RuleFor(x => x.Postfix).Empty();
        }

        var isCronJobType = Type is ComplexApplicationType.CronJob or ComplexApplicationType.CronJobWithCommand;
        if (isCronJobType)
        {
            v.RuleFor(x => x.Schedule).NotEmpty();
            v.RuleFor(x => x.Schedule)
                .Must(x => !string.IsNullOrEmpty(x) && CrontabSchedule.TryParse(x) != null)
                .WithMessage("Schedule must be a valid cron expression.");
        }
        else
        {
            v.RuleFor(x => x.Schedule).Empty();
        }

        var result = v.ValidateAsync(this).Result;
        var errors = result.Errors.Select(e => new ValidationResult(e.ErrorMessage, [e.PropertyName]));
        return errors;
    }
}

public enum ComplexApplicationType
{
    Application = 0,
    ApplicationWithCommand = 1,
    CronJob = 2,
    CronJobWithCommand = 3
}

Resolve services

Our validations use a lot of services and it may be quite a burden to set everything up again and again. One thing is clear: we need to override some services, as they should not be allowed to call out to external services. Let's create a TestServiceOverride to bundle these overrides for our application:

public class TestServiceOverrides
{
    public Mock<IDockerHubService> DockerHubService { get; } = new(MockBehavior.Loose);

    public ProvisioningOptions ProvisioningOptions { get; } = new();

    public TestServiceOverrides()
    {
        SetupDockerHubService();
    }

    public void Apply(IServiceCollection services)
    {
        FluentValidationLanguageManager.SetGlobalOptions();

        services
            .AddSingleton(_ => DockerHubService.Object)
            .AddSingleton<IDataAnnotationsValidator, DataAnnotationsValidator>()
            .AddSingleton(_ => ProvisioningOptions)
            .AddTransient(sp => Options.Create(sp.GetRequiredService<ProvisioningOptions>()));
    }

    protected virtual void SetupDockerHubService()
    {
        string[] repos = ["repo-one"];

        DockerHubService
            .Setup(x => x.Exists(
                It.IsAny<string>(),
                It.IsAny<CancellationToken>())
            ).ReturnsAsync((string repo, CancellationToken _) =>
            {
                if (repo.StartsWith("ktt/"))
                {
                    return true;
                }

                return repos.Contains(repo);
            });
    }
}

Notice how we expose the actual Mock<T> instance, so a test can override the central mocking settings. We are using MockBehavior.Loose to configure the behavior explicitly here. This allows the methods to return the default when the parameters of the setup are not matching. This is the default behavior, but it might not be obvious to everyone.

Let's combine our application DI and this overrides class:

public class TestWebApplicationFactory : WebApplicationFactory<Program>
{
    private readonly Action<IServiceCollection>? _overrides;

    public TestServiceOverrides Mocks { get; } = new();

    public TestWebApplicationFactory(Action<IServiceCollection>? overrides = null)
    {
        _overrides = overrides;
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureAppConfiguration((context, config) =>
        {
            config.AddJsonFile("appsettings.json", optional: false);
        });

        builder.ConfigureServices(services =>
        {
            // Remove all background services
            foreach (var serviceDescriptor in services.ToArray())
            {
                if (serviceDescriptor.ServiceType.IsGenericType &&
                    serviceDescriptor.ServiceType.GetGenericTypeDefinition() == typeof(IHostedService))
                {
                    services.Remove(serviceDescriptor);
                }
            }

            // remove console logger
            services.RemoveAll<Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider>();

            // replace all loggers with NullLogger
            services.RemoveAll(typeof(Microsoft.Extensions.Logging.ILogger<>));
            services.AddSingleton(typeof(Microsoft.Extensions.Logging.ILogger<>), typeof(Microsoft.Extensions.Logging.Abstractions.NullLogger<>));

            Mocks.Apply(services);

            // Apply the additional overrides if provided
            if (_overrides != null)
            {
                _overrides(services);
            }
        });
    }
}

We can use the service provider to create the validator in each test class:

private readonly IDataAnnotationsValidator _validator =
    new TestWebApplicationFactory()
        .Services
        .GetRequiredService<IDataAnnotationsValidator>();

Now, the IList<ValidationResults> will be inspected to see if certain validation results are present (or not). These methods will make it easier to make asserts on the ValidationResult collections and produce a readable error when a test fails:

public static class ValidationResultAssertionsExtensions
{
    public static void ShouldBeValid(this IEnumerable<ValidationResult> errors)
    {
        var length = errors.Count();

        var formatted = string.Join("\n", errors.Select(e =>
            $" - {string.Join(", ", e.MemberNames)}: {e.ErrorMessage}"));

        Assert.True(length == 0, "There should be no validation errors, but found:\n" + formatted);
    }

    public static void ShouldContain(
        this IEnumerable<ValidationResult> errors,
        string memberName,
        string? expectedMessage = null)
    {
        var matchFound = Matches(errors, memberName, expectedMessage);
        Assert.True(matchFound, BuildFailureMessage(
            memberName,
            expectedMessage,
            isContainCheck: true,
            errors));
    }

    public static void ShouldNotContain(
        this IEnumerable<ValidationResult> errors,
        string memberName,
        string? expectedMessage = null)
    {
        var matchFound = Matches(errors, memberName, expectedMessage);
        Assert.False(matchFound, BuildFailureMessage(
            memberName,
            expectedMessage,
            isContainCheck: false,
            errors));
    }

    private static bool Matches(
        IEnumerable<ValidationResult> errors,
        string memberName,
        string? expectedMessage) =>
        errors.Any(e =>
            (expectedMessage == null || e.ErrorMessage == expectedMessage) &&
            e.MemberNames.Contains(memberName)
        );

    private static string BuildFailureMessage(
        string memberName,
        string? expectedMessage,
        bool isContainCheck,
        IEnumerable<ValidationResult> errors)
    {
        var header = isContainCheck
            ? $"Expected a validation error for \"{memberName}\""
            : $"Did not expect a validation error for \"{memberName}\"";

        if (expectedMessage != null)
        {
            header += $" with message \"{expectedMessage}\"";
        }

        var formatted = string.Join("\n", errors.Select(e =>
            $" - {string.Join(", ", e.MemberNames)}: {e.ErrorMessage}"));

        return $"{header}, but found:\n{formatted}";
    }
}

Our tests might look something like this:

// a specific validation result for CPU
errors.ShouldContain("Cpu", "The Cpu field is required.");

// The test might fail with a message like this:
//
// Expected a validation error for "Cpu" with message "The Cpu field is required.", but found:
// - Environment: The Environment field is required.
// - DockerHubRepo: The DockerHubRepo field is required.
// - ImageTag: The ImageTag field is required.
// - Ram: The Ram field is required.
//

// no validation result for CPU should be there
errors.ShouldNotContain("Cpu");

// The test might fail with a message like this:
//
// Did not expect a validation error for "Cpu", but found:
// - Environment: The Environment field is required.
// - DockerHubRepo: The DockerHubRepo field is required.
// - ImageTag: The ImageTag field is required.
// - Cpu: The field Cpu must match the regular expression '^\d+m$'.
// - Ram: The Ram field is required.
//

Test by type

You can test in a few ways. Let's first see how you can test per type. You group all the features functionally together.

1. Types: generic properties

First we should test the generic properties, like: Cpu, Environment, DockerHubRepo, ImageTag and Ram. We start by testing if the required properties are working. Next, we will check each individual property, by adding a bogus value and by adding a valid value.

We'll use the [Theory] to inject the various ComplexApplicationType values, so we test all interactions.

[Theory]
[InlineData(ComplexApplicationType.Application)]
[InlineData(ComplexApplicationType.ApplicationWithCommand)]
[InlineData(ComplexApplicationType.CronJob)]
[InlineData(ComplexApplicationType.CronJobWithCommand)]
public void ValidateGenericProperties(ComplexApplicationType type)
{
    var request = new ComplexApplication
    {
        Type = type
    };

    // 1. validate required fields
    _validator.TryValidate(request, out var errors);

    errors.ShouldContain("Name", "The Name field is required.");
    errors.ShouldContain("Team", "The Team field is required.");
    errors.ShouldContain("Cpu", "The Cpu field is required.");
    errors.ShouldContain("Environment", "The Environment field is required.");
    errors.ShouldContain("DockerHubRepo", "The DockerHubRepo field is required.");
    errors.ShouldContain("ImageTag", "The ImageTag field is required.");
    errors.ShouldContain("Ram", "The Ram field is required.");

    // 2. Name
    request.Name = "test";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Name");

    // 3. Team

    // 3.1 invalid team
    request.Team = "kaas";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Team");

    // 3.2 valid team
    request.Team = "Racing Greens";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Team");

    // 2. CPU

    // 2. invalid CPU
    request.Cpu = "blah";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Cpu", "The field Cpu must match the regular expression '^\\d+m$'.");

    // 2. valid CPU
    request.Cpu = "100m";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Cpu");

    // 3. ram

    // 3.1 test invalid ram
    request.Ram = "blah";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Ram", "The field Ram must match the regular expression '^\\d+Mi$'.");

    // 3.2 test valid ram
    request.Ram = "100Mi";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Ram");

    // 4. image tag
    request.ImageTag = "12-abcefe";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Ram");

    // 5. environment

    // 5.1 invalid environment
    request.Environment = "blah";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Environment");

    // 5.2 valid environment
    request.Environment = "server-one";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Environment");

    // 6. dockerhub repo

    // 6.1 invalid dockerhub repo

    request.DockerHubRepo = "blah";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("DockerHubRepo");

    // 6.2 valid dockerhub repo
    request.DockerHubRepo = "repo-one";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("DockerHubRepo");
}

What I like is that the properties are tested in depth. I'm not sure if I like having so many asserts in a single test.

2. Types: with a command

Types with commands should have both a Command and a Postfix property. We'll start by testing if the required properties are present. Then we'll inspect each individual property, by adding a bogus value and by adding a valid value.

[Theory]
[InlineData(ComplexApplicationType.ApplicationWithCommand)]
[InlineData(ComplexApplicationType.CronJobWithCommand)]
public void ValidateTypesWithCommand(ComplexApplicationType type)
{
    var request = new ComplexApplication
    {
        Name = "test",
        Team = "Racing Greens",
        Type = type,
        Cpu = "100m",
        Ram = "100Mi",
        ImageTag = "12-abcefe",
        Environment = "server-one",
        DockerHubRepo = "repo-one"
    };

    // 1. Command and Postfix must not be empty
    request.Command = string.Empty;
    request.Postfix = string.Empty;
    _validator.TryValidate(request, out var errors);
    errors.ShouldContain("Command", "Command must not be empty.");
    errors.ShouldContain("Postfix", "Postfix must not be empty.");

    // 2. Scripts without tini is not allowed
    request.Command = "/app/start.sh service-a";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Command", "Script files (.sh) may only be executed when tini is used.");

    // 3. Script with tini is allowed
    request.Command = "tini /app/start.sh service-a";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Command");

    // 4. Validate invalid postfix

    // 4.1 Postfix invalid due to casing
    request.Postfix = "I'm a program";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Postfix", "The value must be lower-kebab-case and may not contain the words cron, site or service.");

    // 4.2 Postfix invalid due to forbidden words
    request.Postfix = "cron-site-service";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Postfix", "The value must be lower-kebab-case and may not contain the words cron, site or service.");

    // 4.3 Postfix valid
    request.Postfix = "pinger";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Postfix");
}

3. Types: without a command

Some types may not have a Command or Postfix, which is the inverse of the test above.

[Theory]
[InlineData(ComplexApplicationType.Application)]
[InlineData(ComplexApplicationType.CronJob)]
public void ValidateTypesWithoutCommand(ComplexApplicationType type)
{
    var request = new ComplexApplication
    {
        Name = "test",
        Team = "Racing Greens",
        Type = type,
        Cpu = "100m",
        Ram = "100Mi",
        ImageTag = "12-abcefe",
        Environment = "server-one",
        DockerHubRepo = "repo-one"
    };

    // 1. Some fields must be empty
    request.Command = "dotnet run /app/kaas.dll";
    request.Postfix = "kaas";
    _validator.TryValidate(request, out var errors);
    errors.ShouldContain("Command", "Command must be empty.");
    errors.ShouldContain("Postfix", "Postfix must be empty.");

    // 2. Command is not allowed
    request.Command = string.Empty;
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Command");

    // 3. Postfix is not allowed
    request.Postfix = string.Empty;
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Postfix");
}

4. Types: cron

Cron types need to have a Schedule property, which should be a valid cron schedule.

[Theory]
[InlineData(ComplexApplicationType.CronJob)]
[InlineData(ComplexApplicationType.CronJobWithCommand)]
public void ValidateCronJobSpecificProperties(ComplexApplicationType type)
{
    var request = new ComplexApplication
    {
        Name = "test",
        Team = "Racing Greens",
        Type = type,
        Cpu = "100m",
        Ram = "100Mi",
        ImageTag = "12-abcefe",
        Environment = "server-one",
        DockerHubRepo = "repo-one"
    };

    // 1. schedule cannot be empty
    request.Schedule = string.Empty;
    _validator.TryValidate(request, out var errors);
    errors.ShouldContain("Schedule", "Schedule must not be empty.");

    // 2. invalid schedule
    request.Schedule = "maandag de 14e";
    _validator.TryValidate(request, out errors);
    errors.ShouldContain("Schedule", "Schedule must be a valid cron expression.");

    // 2. valid schedule
    request.Schedule = "5 4 * * *";
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Schedule");
}

5. Types: without a schedule

We should also test for the inverse of the test above.

[Theory]
[InlineData(ComplexApplicationType.Application)]
[InlineData(ComplexApplicationType.ApplicationWithCommand)]
public void ValidateApplicationSpecificProperties(ComplexApplicationType type)
{
    var request = new ComplexApplication
    {
        Name = "test",
        Team = "Racing Greens",
        Type = type,
        Cpu = "100m",
        Ram = "100Mi",
        ImageTag = "12-abcefe",
        Environment = "server-one",
        DockerHubRepo = "repo-one"
    };

    // 1. schedule must be empty
    request.Schedule = "blah";
    _validator.TryValidate(request, out var errors);
    errors.ShouldContain("Schedule", "Schedule must be empty.");

    // 2. validate schedule empty
    request.Schedule = string.Empty;
    _validator.TryValidate(request, out errors);
    errors.ShouldNotContain("Schedule");
}

Discussion

While this keeps the number of test cases pretty low, it becomes harder to see which test case if failing. Especially when test a number of generic properties, there are a total of 16 asserts in a single test case. In terms of documentation it works a bit better, as you halve grouped everything together.

Test by trait

Another route is splitting all the tests up into singular tests where we only inspect one property. We could group those tests into: command, postfix, schedule and generic property tests.

1. Trait: Generic properties

Let's split up all of our asserts into separate testcases.

In order to test the format of an individual property, we must first make sure that the required properties of the model are filled. The required properties take precedence over other validations.

public class GenericTests
{
    private readonly IDataAnnotationsValidator _validator =
        new TestWebApplicationFactory()
            .Services
            .GetRequiredService<IDataAnnotationsValidator>();

    private ComplexApplication CreateBaseRequest(ComplexApplicationType type) => new()
    {
        Type = type
    };

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Require_Required_Fields(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Name", "The Name field is required.");
        errors.ShouldContain("Team", "The Team field is required.");
        errors.ShouldContain("Cpu", "The Cpu field is required.");
        errors.ShouldContain("Environment", "The Environment field is required.");
        errors.ShouldContain("DockerHubRepo", "The DockerHubRepo field is required.");
        errors.ShouldContain("ImageTag", "The ImageTag field is required.");
        errors.ShouldContain("Ram", "The Ram field is required.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Reject_Invalid_Cpu_Format(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.DockerHubRepo = "repo-one";
        request.Environment = "server-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";
        request.Cpu = "blah";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Cpu", "The field Cpu must match the regular expression '^\\d+m$'.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Accept_Valid_Cpu_Format(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.DockerHubRepo = "repo-one";
        request.Environment = "server-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";
        request.Cpu = "100m";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Cpu");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Reject_Invalid_DockerHubRepo(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.Cpu = "100m";
        request.Environment = "server-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";
        request.DockerHubRepo = "blah";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("DockerHubRepo", "The DockerHub repository does not exist.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Accept_Valid_DockerHubRepo(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.Cpu = "100m";
        request.Environment = "server-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";
        request.DockerHubRepo = "repo-one";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("DockerHubRepo");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Reject_Invalid_Environment(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.Cpu = "100m";
        request.DockerHubRepo = "repo-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";
        request.Environment = "blah";

        _validator.TryValidate(request, out var errors);
        errors.ShouldContain("Environment", "blah is not valid or allowed. Options are: [server-one, server-two, server-three]");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Accept_Valid_Environment(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.Cpu = "100m";
        request.DockerHubRepo = "repo-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";
        request.Environment = "server-one";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Environment");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Reject_Invalid_Ram_Format(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.Cpu = "100m";
        request.DockerHubRepo = "repo-one";
        request.Environment = "server-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "blah";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Ram", "The field Ram must match the regular expression '^\\d+Mi$'.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Accept_Valid_Ram_Format(ComplexApplicationType type)
    {
        var request = CreateBaseRequest(type);
        request.Name = "test";
        request.Team = "Racing Greens";
        request.Cpu = "100m";
        request.DockerHubRepo = "repo-one";
        request.Environment = "server-one";
        request.ImageTag = "12-abcefe";
        request.Ram = "100Mi";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Ram");
    }
}

2. Trait: Command

Next up are the commands. Here's where the CreateDefaultRequestForType shines, as we are adding all of the required properties first, so we can properly test the validation rules concerning commands.

public class CommandTests
{
    private readonly IDataAnnotationsValidator _validator =
        new TestWebApplicationFactory()
            .Services
            .GetRequiredService<IDataAnnotationsValidator>();

    private ComplexApplication CreateDefaultRequestForType(ComplexApplicationType type)
    {
        return new ComplexApplication
        {
            Name = "test",
            Team = "Racing Greens",
            Type = type,
            Cpu = "100m",
            Ram = "100Mi",
            ImageTag = "12-abcefe",
            Environment = "server-one",
            DockerHubRepo = "repo-one",
            Command = string.Empty,
            Postfix = "kafka-processor"
        };
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Disallow_Script_Command_Without_Tini(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Command = "/app/start.sh";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Command", "Script files (.sh) may only be executed when tini is used.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Allow_Script_Command_With_Tini(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Command = "tini /app/start.sh";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Command");
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Allow_NonScript_Command_For_CommandTypes(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Command = "dotnet run /app/main.dll";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Command");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.CronJob)]
    public void Should_Not_Allow_Command_For_Types_Without_Command(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Command = "dotnet run /app/service.dll";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Command", "Command must be empty.");
    }
}

3. Trait: Postfix

The Postfix property has some relation with the command property, as they are only allowed on types that support a command. I decided to split it into a separate class for added clarity.

public class PostfixTests
{
    private readonly IDataAnnotationsValidator _validator =
        new TestWebApplicationFactory()
            .Services
            .GetRequiredService<IDataAnnotationsValidator>();

    private ComplexApplication CreateDefaultRequestForType(ComplexApplicationType type)
    {
        return new ComplexApplication
        {
            Name = "test",
            Team = "Racing Greens",
            Type = type,
            Cpu = "100m",
            Ram = "100Mi",
            ImageTag = "12-abcefe",
            Environment = "server-one",
            DockerHubRepo = "repo-one",
            Command = "tini /app/start.sh",
            Postfix = string.Empty
        };
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Require_NonEmpty_Postfix(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Postfix", "Postfix must not be empty.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Reject_NonKebabCase_Postfix(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Postfix = "MyService";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Postfix", "The value must be lower-kebab-case and may not contain the words cron, site or service.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Reject_ForbiddenWords_In_Postfix(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Postfix = "cron-site-service";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Postfix", "The value must be lower-kebab-case and may not contain the words cron, site or service.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Allow_Valid_Postfix(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Postfix = "pinger";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Postfix");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.CronJob)]
    public void Should_Require_Empty_Postfix(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Postfix = string.Empty;

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Postfix");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.CronJob)]
    public void Should_Reject_NonEmpty_Postfix(ComplexApplicationType type)
    {
        var request = CreateDefaultRequestForType(type);
        request.Postfix = "pinger";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Postfix", "Postfix must be empty.");
    }
}

4. Trait: Schedule

Last, but not least, we should validate the schedule.

public class ScheduleTests
{
    private readonly IDataAnnotationsValidator _validator =
        new TestWebApplicationFactory()
            .Services
            .GetRequiredService<IDataAnnotationsValidator>();

    private ComplexApplication CreateRequest(ComplexApplicationType type)
    {
        return new ComplexApplication
        {
            Name = "test",
            Team = "Racing Greens",
            Type = type,
            Cpu = "100m",
            Ram = "100Mi",
            ImageTag = "12-abcefe",
            Environment = "server-one",
            DockerHubRepo = "repo-one",
            Command = "tini /app/start.sh",
            Postfix = "job-runner"
        };
    }

    [Theory]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Fail_When_Schedule_Is_Empty(ComplexApplicationType type)
    {
        var request = CreateRequest(type);
        request.Schedule = string.Empty;

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Schedule", "Schedule must not be empty.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Fail_When_Schedule_Is_Invalid(ComplexApplicationType type)
    {
        var request = CreateRequest(type);
        request.Schedule = "this is not a cron";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Schedule", "Schedule must be a valid cron expression.");
    }

    [Theory]
    [InlineData(ComplexApplicationType.CronJob)]
    [InlineData(ComplexApplicationType.CronJobWithCommand)]
    public void Should_Pass_When_Schedule_Is_Valid(ComplexApplicationType type)
    {
        var request = CreateRequest(type);
        request.Schedule = "*/5 * * * *";

        _validator.TryValidate(request, out var errors);

        errors.ShouldNotContain("Schedule");
    }

    [Theory]
    [InlineData(ComplexApplicationType.Application)]
    [InlineData(ComplexApplicationType.ApplicationWithCommand)]
    public void Should_Fail_When_Schedule_Is_Provided(ComplexApplicationType type)
    {
        var request = CreateRequest(type);
        request.Schedule = "*/5 * * * *";

        _validator.TryValidate(request, out var errors);

        errors.ShouldContain("Schedule");
    }
}

Discussion

It took some time to split the tests up. Even though I had help from AI, I kept missing test cases. Having the test cases grouped by trait makes it easier to discover which test cases are missing. With more test cases, it is also easier to see why a test case is failing, as only a single property is involved.

Is it worth it?

This blog was meant as the study of a subject, so what can we conclude? Well... testing the various interactions of the properties of an object can be quite verbose. The test class itself is about 300 lines of code, which is extensive. But does it pay off? I think validation is one of the most important things to get right, as it is the last line of defense before the data enters your controller or business service. These tests might be considered documentation, as they contain the spec via explicit tests.

The code is on GitHub, so check it out: code gallery / 14. validation.

Changelog

2025-06-08 Improved the way dependency injection into the test works.
2025-05-27 Improved the fields of the ComplexApplication to not be rendered in the API. Nullable improvements for the validator.
2025-04-17 Moved the section on localization to another article.
2025-04-14 Added some information on MockBehavior.Loose, as I didn't know about it.
2025-04-14 Refactored 2 regex rules for Postfix into a single [RegularExpression] attribute on the property.
2025-04-11 Initial article.