Running Selenuim webdriver tests with VsTest - testAssemblies in Azure Dev ops leads to inconsistent runs - selenium-webdriver

Question

We are running into an issue in our azure devops release pipeline with the selenium webdriver not running consistently. This was the tutorial followed. The problem is the tests will either start and run or they will start and then timeout without any new logs as to what happened. All tests work locally. The code is the same between attempts 1-4. However the results are different for each run.
Attempt 1 passes fine.
Attempt 2 times out
Attempt 3 times out
Attempt 4 failed results.
Here is an example test in C#.
public class HomePageShould : IClassFixture<ChromeDriverFixture>
{
private readonly IWebDriver _driver;
public HomePageShould(ChromeDriverFixture chromeDriverFixture)
{
_driver = chromeDriverFixture.Driver;
_driver.Manage().Cookies.DeleteAllCookies();
_driver.Navigate().GoToUrl("about:blank");
_driver.Manage().Window.Maximize();
}
[Fact]
public void LogInWithoutForm()
{
try
{
//Act
AuthenticatedHomePage? authenticatedHomePage = new(_driver, UserDictionary.UserNameAdmin, UserDictionary.PasswordAdmin);
//Assert
Assert.Equal(authenticatedHomePage.Title, _driver.Title);
Assert.Equal(authenticatedHomePage.Url, _driver.Url);
Assert.Equal(authenticatedHomePage.WelcomeText, AuthenticatedHomePage.ExpectedWelcomeText);
}
catch (Exception ex)
{
Console.WriteLine($"Something went wrong in testing, {ex.Message}");
throw new ApplicationException($"Something went wrong in testing, {ex.Message}");
}
finally
{
_driver.Quit();
}
}
}
AuthenticatedHomePage reaches out and sets sessionStorage for our authentication in the browser. Once authenticated, the user should see authenticated welcome text. From what I have read, this should be the appropriate wait type for selenium
public AuthenticatedPage(IWebDriver driver, string usernameConfig, string passwordConfig) : base(driver)
{
Driver.Navigate().GoToUrl(UrlWithoutRedirect);
try
{
MsalAuthenticator msalAuthenticator = new(usernameConfig, passwordConfig);
msalAuthenticator.AcquireAndSetTokensAsync(this.Driver).Wait();
}
catch
{
throw;
}
WebDriverWait waitForAuth = new(this.Driver, TimeSpan.FromSeconds(WaitTimeConstants.DriveTimeOut));
Driver.Navigate().GoToUrl(BaseUrl);
waitForAuth.Until(ExpectedConditions.ElementIsVisible(AuthenticatedHomePage.WelcomeTextLocation));
}
At first we had Dev and UI tests in one step, but have seperated them out incase the dev instance not being up and running was the issue. We tried to add a wait before this step as well.
Below are the tasks in dev ops and the yaml for VsTest step.
steps:
- task: VSTest#2
displayName: 'VsTest - testAssemblies'
inputs:
testAssemblyVer2: |
**\*UITests.dll
!**\*TestAdapter.dll
!**\obj\**
searchFolder: '$(System.DefaultWorkingDirectory)/Web/drop/CP.UITests'
uiTests: true
vsTestVersion: toolsInstaller
codeCoverageEnabled: false
publishRunAttachments: false
timeoutInMinutes: 15
retryCountOnTaskFailure: 3
This is a screen shot of the runs in Dev ops
This is a log when chromedriver will start but will not stop. We have this stage fail if it takes longer than 15 mins. Like attempts 2 & 3.
2023-02-14T16:43:12.1169573Z A total of 1 test files matched the specified pattern.
2023-02-14T16:43:21.7959394Z [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 6.0.13)
2023-02-14T16:43:27.7561162Z [xUnit.net 00:00:05.98] Discovering: CP.UITests
2023-02-14T16:43:27.9728241Z [xUnit.net 00:00:06.20] Discovered: CP.UITests
2023-02-14T16:43:27.9763969Z [xUnit.net 00:00:06.20] Starting: CP.UITests
2023-02-14T16:43:30.6650810Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53069
2023-02-14T16:43:30.6652977Z Only local connections are allowed.
2023-02-14T16:43:30.6654512Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T16:43:30.6655565Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53070
2023-02-14T16:43:30.6656025Z Only local connections are allowed.
2023-02-14T16:43:30.6656487Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T16:43:31.7961660Z ChromeDriver was started successfully.
2023-02-14T16:43:31.7962394Z ChromeDriver was started successfully.
Here is the first attempt where chromedriver would start, run tests, and pass.
2023-02-14T22:17:37.1799995Z **************** Starting test execution *********************
2023-02-14T22:17:39.9882693Z A total of 1 test files matched the specified pattern.
2023-02-14T22:17:41.7445216Z [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 6.0.13)
2023-02-14T22:17:43.0153630Z [xUnit.net 00:00:01.33] Discovering: CP.UITests
2023-02-14T22:17:43.0848680Z [xUnit.net 00:00:01.40] Discovered: CP.UITests
2023-02-14T22:17:43.0871184Z [xUnit.net 00:00:01.40] Starting: CP.UITests
2023-02-14T22:17:45.3301517Z Only local connections are allowed.
2023-02-14T22:17:45.3303493Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T22:17:45.3309471Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53253
2023-02-14T22:17:45.3310260Z Only local connections are allowed.
2023-02-14T22:17:45.3310889Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T22:17:45.3927940Z ChromeDriver was started successfully.
2023-02-14T22:17:45.3938036Z ChromeDriver was started successfully.
2023-02-14T22:17:52.2781305Z Passed CP.UITests.TestsFixtureClass.HomePageShould.LogInWithoutForm [5 s]
2023-02-14T22:17:58.2665721Z Passed CP.UITests.TestsFixtureClass.CommCenterGradingHeaderPageShould.ButtonNotVisibleForNonAdminUsers(usernameConfig: "usernamereadonly", passwordConfig: "passwordreadonly") [12 s]
2023-02-14T22:17:58.2887099Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53297
2023-02-14T22:17:58.2888078Z Only local connections are allowed.
2023-02-14T22:17:58.2889564Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T22:17:58.3426042Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53300
2023-02-14T22:17:58.3427004Z Only local connections are allowed.
2023-02-14T22:17:58.3427682Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T22:17:58.3528328Z ChromeDriver was started successfully.
2023-02-14T22:17:58.3880372Z ChromeDriver was started successfully.
2023-02-14T22:17:59.8059722Z Passed CP.UITests.TestsFixtureClass.GradingPageShould.ButtonNotVisibleForNonFinalizationUsers(usernameConfig: "usernamereadonly", passwordConfig: "passwordreadonly") [13 s]
2023-02-14T22:18:08.9410340Z Passed CP.UITests.TestsFixtureClass.CommCenterGradingHeaderPageShould.ButtonNotVisibleForNonAdminUsers(usernameConfig: "usernamefinalizationmanager", passwordConfig: "passwordfinalizationmanager") [10 s]
2023-02-14T22:18:08.9664368Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53334
2023-02-14T22:18:08.9665133Z Only local connections are allowed.
2023-02-14T22:18:08.9666070Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T22:18:09.0113746Z ChromeDriver was started successfully.
2023-02-14T22:18:09.1041362Z Starting ChromeDriver 109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414#{#1172}) on port 53337
2023-02-14T22:18:09.1042163Z Only local connections are allowed.
2023-02-14T22:18:09.1042780Z Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-02-14T22:18:09.1534210Z ChromeDriver was started successfully.
2023-02-14T22:18:10.4506270Z Passed CP.UITests.TestsFixtureClass.GradingPageShould.ButtonNotVisibleForNonFinalizationUsers(usernameConfig: "usernameadmin", passwordConfig: "passwordadmin") [10 s]
2023-02-14T22:18:12.8153500Z [xUnit.net 00:00:31.12] Finished: CP.UITests
2023-02-14T22:18:12.8893561Z Passed CP.UITests.TestsFixtureClass.GradingPageShould.ShowFinalizeButtonForFinalizationManager [3 s]
2023-02-14T22:18:12.8894460Z Passed CP.UITests.TestsFixtureClass.CommCenterGradingHeaderPageShould.ShowButtonForAdmin [3 s]
2023-02-14T22:18:12.9413193Z Results File: D:\a\_temp\TestResults\VssAdministrator_WIN-MUNJUIO7Q1G_2023-02-14_22_17_44.trx
2023-02-14T22:18:12.9568402Z Test Run Successful.
2023-02-14T22:18:12.9568984Z Total tests: 12
2023-02-14T22:18:12.9569445Z Passed: 7
2023-02-14T22:18:12.9569770Z Skipped: 5
2023-02-14T22:18:12.9584474Z Total time: 32.9186 Seconds
2023-02-14T22:18:12.9857241Z Vstest.console.exe exited with code 0.
2023-02-14T22:18:12.9857854Z **************** Completed test execution *********************
Some things we tried are:
Using the MS screen resolution tool as
outlined here. This didn't work for us.
Setting --no-sandbox,
but we aren't on Linux and it's unsupported and highly
discouraged. We tried other configs like this as well. We are running with incognito
Tried other drivers besides chrome
Changing the timeout of the webdriver to be a few mins. driver = new ChromeDriver(serviceObj, chromeOptions, TimeSpan.FromMinutes(2)); for example
TLDR: selenium will not run consistently in azure devops pipeline even though code hasn't changed between attempts. Things run fine locally; selenium seems to be set up as MS intended in devops.
We are not sure how to go forward with this pipeline. Any suggestions or information to get this to be idempotent would be helpful.