Azure DevOps, Scrum, & .NET Software Leadership and Consulting Services

Free course! Predicting the Future, Estimating, and Running Your Projects with Flow Metrics

ASP.NET Core Integration Tests with Selenium & WebApplicationFactory


WebApplicationFactory<T> is one of those awesome features in ASP.NET Core that practically nobody knows about. WebApplicationFactory<T> lets you quickly and easily run integration tests against your ASP.NET Core MVC or Web API application without having to deploy the app to an actual server like Kestrel or IIS. Instead of running in a “real” server, your app gets run in a test server and that lets you do some handy testing tricks that makes testing your app easier.

(TL;DR — here’s the source code on GitHub)

Testing Tricks with WebApplicationFactory<T>

What do I mean by “handy testing tricks”? The handiest of handy testing tricks is simply just turning off security during your tests. You’d never do this in production but when you’re writing automated integration tests against your ASP.NET Core apps, you usually just want to focus on feature functionality in your application. So if you have to worry about coding all the “log into the app” test code in addition to testing the feature functionality — well, if you’re like most devs, you’ll say “that feels like a whole lot of work” and then not write the test. With integration tests that are written with WebApplicationFactory<T>, you can swap our your production security with some fake security and be on your way to Productivity Nirvana with just a few lines of code.

Here’s the basic idea. If you’re using dependency injection to configure your dependencies in Startup.cs for production, what WebApplicationFactory<T> does is it lets you inject some custom code into Startup.cs for the sake of your tests. Want to turn off security? Sure. Change your database connection strings? No problem. Replace some type registrations with mock/fake type registrations that return canned answers? Simple.

Selenium & WebApplicationFactory<T>

What about running Selenium tests using WebApplicationFactory<T>? Uhhhhh…that’s a little bit of a problem. You’ll almost definitely get an error that says something like ERR_CONNECTION_REFUSED.

OpenQA.Selenium.WebDriverException: unknown error: net::ERR_CONNECTION_REFUSED (Session info: headless MicrosoftEdge=91.0.864.70) Stack Trace: at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse) at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters) at OpenQA.Selenium.Remote.RemoteWebDriver.set_Url(String value) at OpenQA.Selenium.Remote.RemoteNavigator.GoToUrl(String url)

I worked for HOURS and HOURS on this error and didn’t make a whole lot of progress. But then I found a couple of blog posts that started to clear things up for me:

What’s Going Wrong?

The core of the problem is related to how integration tests work with WebApplicationFactory<T>. If you’re running a simple, non-selenium integration test, you call WebApplicationFactory<T>.GetClient() and it gives you an instance of HttpClient to work with. You use it and everything’s happy. But then you try to access your app at the same URL but instead using a different instance of HttpClient or a browser or a Selenium test, and then you get ERR_CONNECTION_REFUSED.

WHY!?!?!?!?!?!?!?!

Well, under the surface, that call to WebApplicationFactory<T>.GetClient() is doing some magic tricks. It’s running your app in memory and it’s basically behaving like an HTTP application…but it’s not actually exposing the application to your network. So when you come along with your Selenium tests and try to hit that same URL, it’s not really actually there and listening for HTTP requests and therefore ERR_CONNECTION_REFUSED.

How to Fix It?

This is where Bertrand Thomas’s blog post helped out immensely. He figured out that you needed to make a separate call to spin up an instance of TestServer in order to make your application respond to HTTP requests coming over the network. (High five, Bertrand! 🙌)

Bertrand’s solution is to use a custom version of WebApplicationFactory<T> that adds the missing initialization calls.

Where Stuff Starts Getting Weird

Like I mentioned earlier in this article, when I do integration tests with WebApplicationFactory<T>, I want to do type replacements during startup so that I can make my test cases easier to write and maintain. When I took Bertrand’s sample code and started making changes to support type replacements, I broke everything. Tests didn’t work. Type replacements didn’t work. Nothing behaved like I expected. Basically, everything worked when I was NOT using selenium but then when I tried doing type replacements and testing them with Selenium-based tests, the type replacements disappeared.

Turns out that when you make the calls to create an instance of TestServer, that what you’re doing is starting up a *SECOND* instance of your application! Ahhhhhhh. So all my type replacements were happening in one instance of my app but not the other instance.

The Solution

In the end, I ended up creating my own version of WebApplicationFactory<T>. In order to make type replacements easy, the constructor for CustomWebApplicationFactory<T> takes an optional Action<IWebHostBuilder> parameter. This parameter value gets stored and then later gets called during the initialization of the application code and web servers.

The addDevelopmentConfigurations parameter lets the developer inject custom initialization code.

The CustomWebApplicationFactory<T> is going to be what’s known as the System Under Test (SUT). The SUT is the code that we’re trying to verify using our test code. In our integration test code, we’ll initialize the SUT and optionally pass in an instance of Action<IWebHostBuilder> that contains our runtime adjustments to the application configuration.

InitializeWithTypeReplacements() is a utility method that knows how to configure our System Under Test

Summary

Initialize the System Under Test using our custom WebApplicationFactory<T> and our calls to IWebHostBuilder and now our Selenium tests run happily.

The source code for this is available at Github. There’s a very simple ASP.NET Core MVC web app that I use to test again. Then there are a handful of tests in IntegrationTestFixtures.cs. The logic for loading WebApplicationFactory<T> is in CustomWebApplicationFactory.cs.

I hope this helps.

-Ben

SUBSCRIBE TO THE BLOG


11 responses to “ASP.NET Core Integration Tests with Selenium & WebApplicationFactory”

  1. […] reading some really useful articles Ben Day, Bertrand Thomas, and the old thing Scott Hanselman (thanks guys!), I led the class […]

  2. steinbachio Avatar

    Thank you for this article! Do you know, how to achieve this in net 6 without a StartUp class?

  3. arielmax Avatar
    arielmax

    Congrats for your post and help! The thing is in net6 we only have Program and the server in memory is only that, it only works with its returned client. 🙁
    Do you know how to achieve it in this scenario? Thanks so much!

  4. arielmax Avatar
    arielmax

    @steinbachio could you find something about your question? I’m facing exactly this situation where we only have Program and I need to access my server in an integration test but out of the client returned by CreateClient(). Tks!

  5. steinbachio Avatar

    Hi! Yes after some trial and error I found a solution and wrote it down here https://steinbach.io/asp-net-core-e2e-tests-with-xunit-and-playwright/.

    1. arielmax Avatar
      arielmax

      @steinbachio OMG! haha… i was my whole weekend with trial and error and I was going to user the Host.CreateDefaultBuilder() approach, but you got it! Tks so much for your help! I sure that I’ve implemented this same solution but, evidently, I had errors.. maybe the code inside the override of CreateHost() and I guess i was creating a client by CreateClient() instead of CreateDefaultClient(). Again, tks so much for your help and do you know the difference between CreateClient and CreateDefaultClient? Tks, talk you!

    2. arielmax Avatar
      arielmax

      ohhh i see… great. Now the weird thing is that i wanted to add a middleware and it works, but my original endpoints are not visible anymore… i’ve just added inside method ConfigureWebHost() the line builder.Configure(b => b.UseProviderState()); after the UseUrls(). This UseProviderState is an extensio where it concretely call the UseMiddleWare()

    3. arielmax Avatar
      arielmax

      @steinbachio tks again for you help! I guess I could make it work. I mean using my custom middleware in the WebApplicationFactory.

      Tks!

  6. steinbachio Avatar

    Nice if I could help 🙂 If I remember correctly, CreateClient() will not invoke CreateHost().

  7. Cleber Dias Avatar
    Cleber Dias

    Hello, this help me a lot, congrats you did a great job.
    There is one thing that is bothering me and I have tested in your repo and found the same issue, while testing the app the resources are not being loaded, if I remove the option headless from the driver I can see that all the css and js files give error 404, do you have any idea how to overcome this issue?

  8. Cleber Dias Avatar
    Cleber Dias

    Sorry I did not notice but the solution for my question it was in your repository:
    https://github.com/benday-inc/SeleniumDemo/issues/1

    Thank you.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.