I am using the Jenkins Allure plugin to generate reports for PyTest runs.
I've noticed that if I delete a failing test from my repository, or rename a failing test, the Allure reports generated by Jenkins continue to show failures for the old tests, even though they no longer exist and did not run in the most recent job.
How do I ensure that Allure reports only contain results for tests that actually ran in the latest job?
You should generate the results in allure-results directory in your root project.
Every time you run your job, those new allure results files will be generated in the Jenkins workspace. You should clean your workspace before the build starts to ensure that you are taking the last execution:
Frank Escobar's answer is correct.
I want to add that if you're using a pipeline the option in his screenshot is not available.
In that case, use the Jenkins Clear Workspace plugin https://jenkins.io/doc/pipeline/steps/ws-cleanup/ and create a pipeline step to clear your workspace before starting the test run.
Related
I have a project to test UI on Java Selenide. I want to show it to one company as a portfolio. I do not want them to bother installing allure command-line tools and add its bin to the PATH variable. Is it possible to generate an allure report without it?
I am talking about the Windows environment.
Assuming you have Allure specified in your pom (and you're using Maven), after your test run you can run this:
mvn allure:report
This will generate a report in the target/site folder (click on index.html to view it)
I'm using the jenkins test-results-analyzer plugin, which uses the jenkins Junit plugin. They get the test results from the jenkins archive and import them into the plugins. I then delete the archived test results, and the test results are still visible through the test-results-analyzer plugin. This suggests that the test-results-analyzer/junit plugins must be storing the test results somewhere....but where?
I've run "du -h" in the jenkins script console and have looked at the directories containing both plugins but nothing jumps out as storing the test results, either in terms of naming convention or directory size...
We're using Jest to perform our React.js unit tests (on the frontend) of our Node.js app which runs in a docker container.
We have set up a Pipeline in Jenkins but I'm unsure of the best way (or best practice) to include the tests as part of the pipeline.
The steps we have are the following:
Check out the code from source control
NPM install and npm run build (front-end)
Docker build + publish
Deploy app
Bump version
Git push
Docker cleanup
I have 3 main queries:
A. I'm assuming it's best to include npm run test between Step 1 and Step 2 and if all tests pass successfully to move further?
B. But how are the snapshots handled? For example, if there's some change which occurred that generates a difference in a snapshot this will not be "checked" back into the source control.
C. I read that people use Cobertura, jest-junit, etc to have unit tests and coverage within Jenkins - what is the best?
Thanks in advance.
Good questions!
A. You can run tests after npm install. And if all tests pass you move further. Another common thing to do is to run linting or code style check.
B. A bad snapshot will fail tests. Which is why it's important to update snapshots before committing. If your jenkins is hooked up to a code review system, you may disable merges that fail builds, to make sure bad snapshots don't get on your master branch.
C. I have seen people use jest-junit, but that's only because there was a requirement to have the coverage report combined with a junit coverage report. If you don't have any particular requirements around the structure of the report, then the default report jest produces should be fine, and you don't need anything extra.
We are executing our Selenium WebDriver GUI tests as a Nunit 3.0 build step in TeamCity 10 and after going through the documentation, we were expecting to see the test execution log in a separate "Tests" tab in the build log. However, there is no "Tests" tab at all in our log and the nunit.xml file with the results is saved only as an artifact in the tmp folder.
Have you come into the same issue? Could you please assist us?
Thank you in advance.
The tests results should be somehow reported into TeamCity. Test can reported by build runner (see the list of the supported frameworks), using XML Report processing build feature or via service messages. In your case you can use XML report processing feature to parse nunit.xml and report test in TeamCity.
I am relatively new to Selenium WebDriver and a self learner. I have created a webDriver project which consists of different packages and have made use of Page Factory concepts extensively. I use TestNg framework to run the suite to generate my report. So my test cases are also following the TestNG framework concepts.
Now my team wants to use my script to be run at every build to test the sanity. The build team just wants to run my whole script by just a click. It can be shell script command or .exe or jar. Build team uses linux m/c and they dont have Ecplise , TestNg installed in their machine. Their intention is whenever build is given they want to run my script by just a click or a command in Command Prompt.( it has to be that simple for them) and a report should be generated in some location in hdfs
My script runs on FF version 32 and Selenium webDriver 2.44.0
Would really appreciate if someone could give me a solution that actually works for my requirement.
I found a similar query from someone but i am not sure if the answer still suits.
How to make Java executable Jar file of WebDriver project
Could somebody please give me a solution. Or the solution mentioned in the above link stil is the best?
Regards
There are few ways to do it:
Use CI tool (Jenkins): You can set up your project in Jenkins and it will allow you to run the project in single click. Jenkins also provides you an option to run the tests periodically so you can configure it in such a way that it will always get executed after certain time period.Jenkins is also capable to trigger the execution if there is any changes in source code of tests. I'd suggest to use this tool. https://jenkins-ci.org/
Use .bat or .sh file: I am not sure if you are using any build tool like ANT or Maven, if you are using any one of them then you can write a .bat or .sh file to run the tests. ANT
If you are not using any build tool then start using, it will help to run the tests in simplest way.For me to run entire suite, I just type below command in terminal from root directory of project
ant run -Dsuite=all
If you are more concerned about system configuration for e.g. the system which will be used by build team does not have a specific version of browser or specific required library. Then I'd suggest to keep all essentials like browser installation files, libraries in your project directory. This is not a good practice but yes it will help others to run the tests smoothly.
I would suggest just exporting a jar file for your project. All the libraries will be packaged together in the jar files(including TestNG) and you can simply double click on the jar file to start your tests.
External resources (if any), will have to be available for the jar file though. So, you can provide the jar file and the external resources together. The external resources might include your test data (if any) or Portable Firefox(if you are using the portable version).
Steps:
Right click on Project -> Export -> Runnable jar file
Give a name and file path for the jar file
Select option - Extract required libraries into generated JAR
And Finish
Troubleshooting:
Check the java version for the machines that you will be running your jar file on. Programs compiled with java 7 will mostly not run if the machine has java 6. Either compile with java 6 or update the jre on the target machines.
If the jar file does not launch, try using Jarfix.