I have configured my test scripts to run from Jenkins using TestNG XML. This xml triggers a suite of testcases from my available test suite. This xml is triggered using a batch file and configured as a build step in Jenkins.
Issue:
Though some of my test case is failing, jenkins job is reporting as Pass since the batch script was completed successfully.
Is there a way to pass the Failure report back to batch file and then to Jenkins to report fail if any test case in the suite failed.
Use TestNG Results Plugin, https://plugins.jenkins.io/testng-plugin.
Then publish the testng result by providing appropriate path to the testng-result.xml. The job will turn yellow on failures.
Related
Setup:
VS CI build configured (pass)
Deploy TestAgent on VM (pass)
AzureVM File Copy (pass)
Run Functional Tests - Selenium (broken)
Here are the logs/errors that I see when this build step executes:
Preparing task execution handler.
Executing the powershell script:
d:\a_tasks\RunVisualStudioTestsusingTestAgent_d353d6a2-e361-4a8f-8d8c-123bebb71028\1.0.52\RunDistributedTests.ps1
DistributedTests: Environment WinRm Protocol HTTPS.
DistributedTests: Task 'CheckTACompat' on machine
'AzureVM.canadacentral.cloudapp.azure.com:5986' is being run
DistributedTests: Task 'CheckTACompat' on machine
'AzureVM.canadacentral.cloudapp.azure.com:5986' completed.
DistributedTests: Creating run for selected test plan with following
parameters
DistributedTests: Test plan ID: 15
DistributedTests: Test suite ID: 16,18
DistributedTests: Test configuration ID: 3
DistributedTests: Run title: Selenium Test Run
DistributedTests: is automated: True
DistributedTests: test settings id : 38
DistributedTests: build location:
C:\seleniumwd\seleniumtestproject\bin\debug\
DistributedTests: build id: 70
DistributedTests: test configuration mapping:
DistributedTests: Test Run with Id 41 Queued
DistributedTests: Please use this link to analyze the test run :
<>
DistributedTests: Test run '41' is in 'InProgress' state.
DistributedTests: Total Tests : 2, Passed Tests : 0
DistributedTests: Test run '41' is in 'Aborted' state.
DistributedTests: Total Tests : 2, Passed Tests : 0
DistributedTests: Test run is aborted. Logging details of the run
logs.
DistributedTests: New test run created.
Test Run queued for Project Collection Build Service (App
Service).
DistributedTests: Test execution started. Test run id : 41
DistributedTests: UnExpected error occured during test execution. Try
again.
DistributedTests: Error : One or more errors occurred.--->No test
assemblies found on the test machine matching the source filter
criteria or no tests discovered matching test filter criteria. Verify
that test assemblies are present on the machine and test filter
criteria is correct.
DistributedTests: Test run aborted. Test run id: 41 System.Exception:
The test run was aborted, failing the task. PowerShell script
completed with 1 errors.
In my Azure File copy task I have this:
Source: $/baseproject/SeleniumTestProject1
Destination Folder: C:\seleniumwd\seleniumtestproject\
If I go to the Azure VM and look at that destination folder, I see the following files/folders inside
bin
obj
Properties
Packages
3 C# selenium tests (.cs)
1 test project (seleniumTestProject)
bin/debug/ on the test agent contains the test project dll (SeleniumTestProject.dll) and all the dlls and webdriver files.
obj/debug/ on the test agent contains the test project dll (SeleniumTestProject.dll) and some text and cache files
When I point the Run Functional Tests task at either of the above folders, I get the error mentioned earlier.
If I switch the test task to use Assemblies instead of test plan I am still not successful.
Any ideas how to properly point to the assemblies?
The error is exactly what it says that there were some missing test assemblies found in the test drop location. I will post bullet points of things that I troubleshooted and confirmed while working on this issue.
ISSUES
MTM tests were linked to some selenium tests and some unit tests
If there are test cases that are linked to unit tests or selenium
tests that don't exist, you will get this error. All tests that are
part of the plan must have an associated test that is in the test
drop location
There are some issues with the Azure copy as some of the .cs files
are not being included in the copy (part of the original problem)
TIPS
While initially setting up this test task you might set the timeout
to 0. I've seen times where there are selenium tests still being run
and the task times out
Select a test plan that has a few manual tests and linked selenium tests while initially setting up this task. This will ensure you get to see one complete
1 failed selenium test in the task will fail the task
You don't need to set up the agent for "Interactive Mode"
Does anybody work with Tosca and Jenkins?
I have problems to run my batch script from Jenkins. It does not open browser, but job is finished successfully. Batch script runs normally when I try to run in on Windows, without Jenkins.
Anybody has an experience using these two tools together?
In order to use Tosca CI with Jenkins you need to:
Configure your Jenkins build project
Add an additional build step (Execute Windows batch command)
Call ToscaCIClient.exe/ToscaCIClient.jar with parameters
e.g. "PathToTosca\ToscaCI\Client\ToscaCIClient.exe" -t junit -r PathToResults\result.xml
Further information can be found here: https://support.tricentis.com/community/article.do?number=KB0012411
The issue your facing is because your running Jenkins as windows service, in this case headless execution happens technically your script is executed for other user account.
Solution
Instead of installing Jenkins host it on Apache Tomcat server
Download Jenkins war file
Copy it inside webapps folder of Tomcat
Run startup.bat
Refer-
https://www.tutorialspoint.com/jenkins/jenkins_tomcat_setup.htm
I had faced the same issue.
I am guessing your slave machine running the test has jenkins services installed as services.
Uninstall the service. (.jenkins/jenkins-slave.exe uninstall)
Use the .jnlp or slave.jar to connect the machine as a slave.
Your test should run fine after that.
If it does just create a batch file to connect the machine as slave whenever it is logged in.
I tried that. But in any case, Jenkins does not open the browser, so Tests from Execution List and I do not know why.
We use Tosca 10.
Step 1. I have written selenium tests and bundled it into executable jar.
Step 2. When I execute this jar tests runs normally. Invokes New browser and execute all steps.
Step 3. I created bat file to execute this jar. In this case also test execution happens normally.
Step 4. TFS Build integration. (TFS 2015)
- I created new build definition in TFS
- Added build step "Batch Script"
- Entered path of bat file created in step 3 in build step
- When I Queue this build. bat file gets called and test execution starts. But It does not open new browser..console says Started IE driver...it starts driver internal to build agent. and execution fails on setting value in textbox.
Click to see TFS Build Cosole
I can't use Visual studio tests. Requirement is to selenium tests should be triggered from jar created on Step 1.
Make sure the build agent is running in interactive mode.
First you could try to RDP to the remote build server(the build agent located) and manually run the batch file to see if it works.
If it's also not work then the issue may related the IE browser on the server.
If it works then maybe the TFS build service account don't have enough permission to run the IE driver. Please make sure the build service account has local Admin permission on the build server.
I have a Jenkins job which is building /packaging/ deploying .NET project. I use "Clean Before checkout" git property for each build.
Finally, I'm executing a batch file. This batch file scanning sonar Qube MSBuild, c# analysis and update the code quality results. But It is taking a long time and I don't want to increment Jenkins job's total execution time from 1 minute to 5 minutes. Developers are thinking so the deployment is taking a long time and not finished yet.
I want to run this batch file independent from the Jenkins job and don't want to see the console output in Jenkins. How could I do this without creating a new Jenkins job for each Jenkins project (upstream/downstream projects)
you need to use the start cmd command,for example:
start YourBatch.bat
This will open a new CMD window and Jenkins won't wait for result.
Bare in mind that if you get an exit code different than 0, it won't fail the build.
What is the best way to run silverlight unit tests automatically using team city?
I have found StatLight which we had working well when we used cc.net, and it says that it has support for teamcity. Does this just mean the test results output file is compatible with teamcity? Do I need to create a command line runner to run the tests? If so how do I get the test results into team city?
Thanks
TeamCity has an extensibility feature where you can output special commands to the console and the TeamCity agent will capture the commands and publish the results within TeamCity.
If you get StatLight running on your desktop - do a regular console-run. Then do another run by giving it the "--teamcity" parameter. Notice the difference in the output?
In TeamCity you can setup a Command Line Build Runner
Command executable: "<Path to statlight.exe>"
Command parameters: "-x=%system.teamcity.build.checkoutDir%\PathToXap\SilverlightClient.Tests.xap --teamcity"
Hope this helps.
There's also a StatLight TeamCity plugin that adds a test runner.
You can try using Lighthouse Silverlight Unit Test Runner, it works with every Build Server including TeamCity and CCNet because it by default produces NUnit compatible xml results file:
http://lighthouse.codeplex.com/