We are developing some testing infrastructure and I have hit a coders block (lack of sleep?)...this seems like it would be a solved problem but I haven't found what I'm looking for via google.
I would like to automatically push builds from our CI server (TeamCity) to a number of machines (growing, but currently 30). These are several WinForms apps and a number of dlls. Once deployed, I would like to kick off tests (NUnit, for both unit and integration tests) and report all results (back to CI? or somewhere else? Not sure).
The target machines are a number of platforms (Win7,Vista, XP, Server 2k8, Server 2k3, Ubuntu, Fedora, Suse, x64, x86, maybe macs down the line)
This gets me part way there (the actual push). But I can't find existing solutions for 'push starting' the tests and reporting back. So far I am thinking of combining the link (or similar) with custom code running on each client machine that watches the deploy directory, runs the tests and reports the results.
Does anyone know of existing solutions?
Links?
Done something similar and care to share?
Edit
If possible, we prefer .net based solutions, but it isn't strictly necessary. I would have tagged the question as such, but ran out of tags :)
You could use KwateeSDCM to both push and start on all the platforms you mention, including mac. However, you'll have to do some coding to get reports out. I'm not familiar with TeamCity but maybe you could push a script along with your application which could then transfer the test results via ftp to a server accessible by TeamCity.
Have a look at: STAF (Software test Automation Framework)
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of reusable components, called services (such as process invocation, resource management, logging, and monitoring).
Which includes STAX:
STAX is an execution engine which can help you thoroughly automate the distribution, execution, and results analysis of your testcases.
And there's an article here:
http://agiletesting.blogspot.com/2004/12/stafstax-tutorial.html
Assuming you have the push part done already, and you don't mind using a TeamCity license, you can create a TeamCity Command Line Runner build configuration or NUnit test configuration that kicks off the tests on a properly configured agent. The build trigger for this test config would be successful completion of the application build.
So far I have ended up using a seperate build step in TeamCity that executes a bat script that in turn fires of tasks to the list of machines using PsExec. So far my trial runs it is working ok, though I now need to parallelize the copying of build output...
Thanks for the input to those who have provided it.
Related
I have 2 simple GUI-test on a WPF application. They work alright from VisualStudio.
Now I am trying to also do that in my Azure pipeline. Ik keep on bumping against the same error right at the start:
"OpenQA.Selenium.WebDriverException: An element could not be located on the page using the given search parameters"
Basically I am working along this line: WinAppDriver in CI with Azure Pipelines. One difference with the example is that I have put my tests in my build pipeline instead of my release pipeline. As far as I can determine that should not be the cause of my problem, using the same Windows-2019 agent.
This is how the pipeline looks like, up to the tests:
In the past I have had problems from VisualStudio similar to those decribed here, but I have left those behind. FindElementByName - Element couldn't be located
Nevertheless I have tried to improve on these aspects by using DefaultWait. It works fine from VisualStudio but has not helped me on Azure.
I read a couple of times about an 'interactive mode' for the agent. But I don't see anything about that in the guideline, and it seems to apply to self hosted agents. I also don't see any configuration option on the standard agent. So I am confused about that.
I am lost here.
Could someone tell me what I am missing?
It would be much appreciated, I have been spending quite some time on this already.
This took me a while to find the necessary information and piece together the solution, which turned out to work.
The key thing is using an interactive agent, which has to be self hosted. Key instruction for me was following Self-hosted Windows agents. This resulted in installing an environment on my own hardware, and to be pretty simple and straight forward.
This implies one has to startup this environment with Powershell, and probably guard the graphical test as it executes. For an one man team as mine that is no problem, for a larger team this may be an issue.
An additional advantage is that the entire building and execution environment is now under one's own control for overview and inspection.
It's funny to push a commit and see the pipeline progress on Azure, plus starting up the graphical test on my own hardware.
I am using selenium to automate my application.. My applicaton works on IE only.. I dont have much scope for other browser..
but when my scripts are running if i do other activities like reading mails, or update QC in another window.. Those are breaking my scripts (no such element or no such window..) once in a while (mostly out of 10, 2 to 3 times i am facing this issue). But can not make system idle till my scripts are run as i do have other activities after started the script. How to stabilize this?? Any one facing these kind of issues??
Running Selenium tests locally using your one and only IE browser does come at the cost of having to not touch your mouse or keyboard during test executions. A way to get around this is to create one or more virtual machines. VirtualBox from Oracle is a popular choice but there are others. You need to install an operating system on your new virtual machine and odds are that your existing Windows license is single use. You can request an additional Microsoft Windows OS license from your IT department or simply buy one yourself for, what? $190? I have done that when the paper work at my client was unmanageable. My time is worth more than that.
Another alternative is to take advantage of 90 day free licenses from Microsoft.
Start your test on the virtual machine then change focus back to your desktop to do other work. I have even added code to the end of test logic to beep when the test completes so that I know when to expand my virtual machine.
Update
You should add an antivirus to that virtual machine. Safety first. :-)
Selenium does not support and non web-based applications, it only supports web based applications.
So if you are doing activities like reading mails and update QC in other window, you will not able to find the elements through selenium.
If you are facing some challenges with IE browser. Please refer this link:https://code.google.com/p/selenium/wiki/InternetExplorerDriver
There are following limitations are given below:
Some limitations of Selenium Automation tool are as follows:
It does not support and non web-based applications, it only supports web based applications.
Its and open source tool so in case of any technical issues you need to rely on the selenium community forums to get your issue resolved.
You need to know at least one of the supported language very well in order to automate your application successfully.
No inbuilt reporting capability so you need plugins like JUnit and TestNG for test reports.
Lot of challenges with IE browser.
I'm using MSpec to drive some automated UI tests using Selenium WebDriver. Much like the examples I found online. I'm having problems getting it to take screenshot when the test fails.
I saw a comment on another issue where it works because they have a ResultSupplementer in the sample web specs. However, ResultSupplementer does not seem to exist in the latest version of Mspec (0.9.1).
Is there a different way to do this in the latest version of mspec? Ultimately, I'm going to generate HTML reports as TeamCity artifacts and include the screenshot on any failing specs.
I've updated the samples for the latest version of MSpec (in short, you need to implement ISupplementSpecificationResults yourself).
I've also merged the solutions and converted the MVC project to Nancy. You'll find that there's a bit more infrastructure-related code that grew over the last couple of years and works around various things, like
status codes 4xx and 5xx logged by IIS Express
IIS and Chrome Driver ports bound by other processes
page objects access the web driver with a high-level API
I use Paket for dependency management because it's far more powerful than plain NuGet
All that said, you need to run msbuild.exe mspec-samples.sln and then All-Specs.cmd. I've also checked that a TeamCity build creates screenshots.
I'm trying to implement automated integration tests for my application. It's a very complex monster. You could say that its database and part of the filesystem are part of its state, because it saves image files in the hard drive, and references to those in the DB. The software needs all those, in a coherent state, to work properly.
Back to writing tests: To run any relevant test, I need some image files in the filesystem, and certain records filled in the database. I thought of putting all of these in a separate folder called TestEnvironmentData in the repository, and retrieving them from the Continuous Integration Server (Team City), but a colleague said the repo is quite full as it is, and that I should set up a special directory, and databases, only in the Continuous Integration server. I don't like that because the tests success depend on me manually mantaining stuff in the server, and restoring initial state before every test becomes cumbersome.
What do you guys do when you need to write integration tests for an app like this? The main goal is having an automated test harness to approach a large scale refactoring. There's lots of spaghetti code and the app's current architecture is hardly unit testable, that's why I decided on integration tests first.
Any alternative approach is welcome.
Developer Repeatability is key when setting up a Continous Integrations Server. I have set one up for my last three employers and I have found the key to success is the developers being able to run the same tests from their dev system in order to get the same results as the CI Server.
The easiest way to do this would be to check in the test artifacts into source control but you could also use dropbox or a Network Share that you copy them from in one of the build steps.
For a .Net solution I have always used MsBuild as you can most easily replicate the build process of Visual Studio and get the same binaries/deployables. As for keeping your database in sync so that tests can be repeatable in the past I used the MbUnit test framework and the [Rollback] attribute as it would roll back any changes to Sql Server that happened in the test. I believe that Nunit now has this attribute as well.
The CI server is great for finding code that breaks existing functionality but unless developers can reproduce the error on their machine they won't trust the CI server for some time.
First of all, we use Maven to build our code. It's like ant, but it relies on convention instead of configuration for many things, like Ruby On Rails does. One of those conventions is a standardized directory structure:
(project)----src----main----(language)
| | \--resources
| \--test----(language)
| \--resources
\--target---...
Using a directory structure like this makes it easy to keep your application resources and testing resources near each other, yet still be able to build for test or build for production, or just build both but just package up the application parts after running the tests.
As far as resetting the database between tests, how you do that is greatly dependent on the DBMS you're using. For instance, if you're using MySQL it's very easy to get the test data the way you want and do a mysqldump to a file you then load before the test. With other DBMSs you may have to drop and recreate the tables and reload the data, or make separate tables for the starting point and use a CREATE/SELECT sql statement to duplicate it each time.
There really is no reliable way around the "reset the database between tests" step.
When creating an auto updating feature for a .NET WinForms application, how does it update the DLLs and not affect the currently running application?
Since the application is running during the update process, won't there be a lock on the DLLs (because those DLLs will have to be overwritten during the update).
Usually you would download the new files into a separate area. Then shutdown and restart and at startup you look for and use the new files if found. Always keeping a last known working version on the side so that the user can revert to something that definitely works if the download causes problems.
ClickOnce is a good technology from Microsoft that does this for you and you can use it directly from Visual Studio 2008.
You'll have to shutdown your application and restart it, as other people have already commented.
I wrote an open-source code to do just that in a transparent mode - including an external update application to do the actual cold update. See http://www.code972.com/blog/2010/08/nappupdate-application-auto-update-framework-for-dotnet/
The code is at http://github.com/synhershko/NAppUpdate (Licensed under the Apache 2.0 license)
I have a seperate 'launcher' application that checks for updates via a web service. If there are updates, it downloads them and then executes my application, which is in a seperate assembly.
The other alternatives are using things like ClickOnce, or downloading the files to a seperate area and restarting the app, as someone else mentioned.
Be warned about ClickOnce, though - it's not as flexible as it sounds. And if you deploy to a system that requires elevating your program to a higer security level to run, you might run into problems if you don't have a certificate for your app installed. I found it very difficult to get straight answers on the Internet to things like certificate management when it comes to ClickOnce. If you have a complex app, you may want to just roll your own updater, which is what I ended up having to do.
If you publish via ClickOnce, all of that tends to be handled for you. It has it's own pro's and con's but usually easier than trying to code it all yourself.
Both Wikipedia and 15seconds have decent info on using ClickOnce, how it works, etc.
As others have stated, ClickOnce isn't as flexible as rolling your own solution but it is a LOT less complicated. It has a small learning curve at first, but with pretty much everything bundled into Visual Studio and the use of Wizards, it usually doesn't take long to stumble onto a working solution.
As deployments get more complex (i.e. beyond than just having prerequisites or application code that needs updating) and you need to do a lot of post-install or pre-install tasks, there are things like WiX which give you somewhat of a hybrid solution between Windows Installer and ClickOnce, with the cost of flexibility being a much steeper learning curve.
The only reason I try to avoid custom installers is that you end up spending way too much time trying to get it just right to handle a bunch of different "What If" scenarios...
These days Windows can do such updates automatically for you with AppInstaller if your app is packaged in the MSIX package.
It downloads the new version of the app in another folder inside ProgramFiles\WindowsApps, then when a user runs the app via the start menu, the system knows what folder it should use. The previous version gets deleted when not in use.
If you want to know how to package your app this way I collected my findings in this answer.