[API Proposal]: WebApplicationFactory.ConfigureHostApplicationBuilder for early test config override

[Youssef1313]

Background and Motivation

See #37680 for the background and motivation. This is highly requested user feature.

Proposed API

namespace Microsoft.AspNetCore.Mvc.Testing;

public class WebApplicationFactory<TEntryPoint> : IDisposable, IAsyncDisposable where TEntryPoint : class
{
+    protected virtual void ConfigureHostApplicationBuilder(IHostApplicationBuilder hostApplicationBuilder);
}

ConfigureHostApplicationBuilder should be called during the entrypoint's call to WebApplication.CreateBuilder.

#66527 is an implementation of this proposal.

Open questions:

• What instance should be passed to IHostApplicationBuilder? The WebApplicationBuilder or its inner _hostApplicationBuilder? Does it matter that much?
• What are all different ways to implement the entrypoint? Do they all create WebApplicationBuilder? Do they all create HostApplicationBuilder? Do we care about any entrypoint that doesn't create those types?

Usage Examples

public class TestServerFixture : WebApplicationFactory<Program>
{
    internal Dictionary<string, string> OverrideConfiguration = new();

    protected override void ConfigureHostApplicationBuilder(IHostApplicationBuilder hostApplicationBuilder)
    {
        base.ConfigureHostApplicationBuilder(hostApplicationBuilder);

        var configuration = hostApplicationBuilder.Configuration;
        var testDir = Path.GetDirectoryName(GetType().Assembly.Location);
        var configLocation = Path.Combine(testDir!, "testsettings.json");

        configuration.Sources.Clear();
        configuration.AddJsonFile(configLocation);
        configuration.AddInMemoryCollection(OverrideConfiguration);
    }
}

Alternative Designs

Risks

[MihuBot]

I'm a bot. Here are possible related and/or duplicate issues (I may be wrong):

• Add additional configuration sources for the WebHostBuilder in the WebApplicationFactory #4898
• .NET 6 Minimal API - WebApplicationFactory doesn't see custom settings json file values #38435
• Unable to load test appsettings in integration tests #40681

[dotnet-policy-service]

Thank you for submitting this for API review. This will be reviewed by @dotnet/aspnet-api-review at the next meeting of the ASP.NET Core API Review group. Please ensure you take a look at the API review process documentation and ensure that:

• The PR contains changes to the reference-assembly that describe the API change. Or, you have included a snippet of reference-assembly-style code that illustrates the API change.
• The PR describes the impact to users, both positive (useful new APIs) and negative (breaking changes).
• Someone is assigned to "champion" this change in the meeting, and they understand the impact and design of the change.

[halter73]

What happens if you override ConfigureHostApplicationBuilder while testing an old-style HostBuilder/ConfigureWebHostDefaults application? Would we throw some sort of exception during WebApplicationFactory constructor?

Right now, it seems like this code just gets silently skipped which isn't ideal. On problem with virtual methods is that it's hard to detect if they've been overridden making it harder to provide clear errors early for misconfiguration. I do get that overrides are more consistent with the rest of this class compared to callbacks though. I wonder if there's a way we can get the best of both worlds.

At a certain point, it might be worth it to create a new version of WebApplicationFactory designed to be used exclusively with WebApplicationBuilder-based projects.

Also, should we create a separate API proposal or issue for HostFactoryResolver changes in the https://github.com/dotnet/runtime repo? It's not exactly public API, but the source is shared a bunch of places. Even if we think an issue is too much process, it probably wouldn't hurt to open a PR or something sooner rather than later so folks interested in just the runtime changes can start taking a look.

[Youssef1313]

@halter73 That's exactly the open question I have in the proposal, for entry points that don't use the WebApplicationBuilder.

For runtime, I already have a PR: dotnet/runtime#127743. Given the limited scope of the change, I thought it doesn't need a separate API proposal, as it's mostly an internal API shared with ASP.NET Core via an internal source-only package.