I already have a SQL Server DB and Source control "Redgate" beside that I already connected my project to repo Github and this project also connected to Jenkins.
What I need is to do some kind of unit test on a the tables, like I already got the query and the test will pass values to this query , in away or another it is Unit test.
But I need it to done by Jenkins. Is there any possible way to do it !? in case of YES has that something to with POWERSHELL ?
I'm kind of confused because I know what I need to do, and I have the tools but the structure is not clear for me .
Where and how to do the Unit test and how to push it to everyone that they develop on the DB.
To do unit testing in databases, I'd recommend the tSQLt framework. Have a look at the quick start guide to see how to use it.
If you've already got a license for Redgate's SQL Source Control then you might also have a license for some of their other products - they have one called SQL Test which makes it much easier to use tSQLt within SQL Server Management Studio.
As for running the unit tests within Jenkins, check to see if you also have a license for Redgate's DLM Automation product - it's a set of Powershell cmdlets that you can use for automating database continuous integration and delivery, including running any tSQLt unit tests that are present. Luckily there is a Jenkins plugin available for it - https://plugins.jenkins.io/redgate-sql-ci
There are also some good videos here on how to use it - http://www.red-gate.com/products/dlm/dlm-automation/resources/
You've got a handful of questions over there, let me try and address most of them.
I'd use Pester Powershell module to create a test (that's using Invoke-SqlCmd) that would output result to xml, so the jenkins can parse them.
You can do db tests in any language you like, actually, like Jenkins can run whatever you feed into it, c# tests, python tests, powershell tests and so on, its just a matter of personal preference which language\stack are you going to use.
I'm not really sure what you mean by: "how to push it to everyone that they develop on the DB".
This is what I had to run the tests with powershell and have jenkins consume them:
$tests = Invoke-Pester -OutputFormat NUnitXml -OutputFile tests.xml -PassThru
if ($tests.FailedCount -eq 0) { Remove-AzureRmResourceGroup -ResourceGroupName $rgName -Force }
Related
Is it possible to have a kiwi-tcms test case launch an executable on a remote server in order to execute the test case and if so how could that be done ?
Short answer - NO!
Long answer:
what you are looking for is some kind of test runner or CI system - that will connect to a remote computer (or use an API directly) and launch automated test cases based on some parameters.
This brings up so many questions I can't even list all of them here but some of the most important ones:
who/where we store authentication credentials
how are progress and results monitored and reported
when are tests scheduled ? What kind of triggers are supported/desired ?
Kiwi TCMS takes a different approach when dealing with automated tests. You can schedule your tests in any way you like and then report the execution results back to Kiwi TCMS.
We are working on plugins for popular test runners, like JUnit, Python Nose, etc, that will automatically discover the result and name of your automated test cases and report them back to Kiwi TCMS.
If you do need a specific plugin/framework please open a request on GitHub and our team will take it into consideration.
Edit: upvote, comment & follow this feature request at https://github.com/kiwitcms/Kiwi/issues/914
I needed to do something similar: run automation on remote systems and report results back to Kiwi. I put together several components to get the entire system working. Here's what worked for me:
Jenkins to initiate test runs and manage remote machines
A Python script to create test runs against a test plan and write out a custom test run manifest, which is...
Passed to the automation system (.NET/C#) via more scripts to make sure the remote machine is configured correctly
Automation output is directly consumed by Jenkins to report test results for the build/job as well as consumed by another Python script that pushes results back to Kiwi
The automation system knows how to interpret the test run manifest and map test cases to test methods implementing the test. It's important to include the Kiwi caserunid throughout the pipeline so the result is associated with the correct Kiwi entry.
please open feature requests on GitHub if you'd like to get test runner adapters (essentially plugins) for Kiwi TCMS. GitHub is the only place where we can track who needs what and prioritize!
I'm looking at implementing Team City and Octopus Deploy for CI and Deployment on demand. However, database deployment is going to be tricky as many are old .net applications with messy databases.
Redgate seems to have a nice plug-in for Team City, but the price will probably be stumbling block
What do you use? I'm happy to execute scripts, but it's the comparison aspect (i.e. what has changed) I'm struggling with.
We utilize a free tool called RoundhousE for handling database changes with our project, and it was rather easy to use it with Octopus Deploy.
We created a new project in our solution called DatabaseMigration, included the RoundhousE exe in the project, a folder where we keep the db change scripts for RoundhousE, and then took advantage of how Octopus can call powershell scripts before, during, and after deployment (PreDeploy.ps1, Deploy.ps1, and PostDeploy.ps1 respectively) and added a Deploy.ps1 to the project as well with the following in it:
$roundhouse_exe_path = ".\rh.exe"
$scripts_dir = ".\Databases\DatabaseName"
$roundhouse_output_dir = ".\output"
if ($OctopusParameters) {
$env = $OctopusParameters["RoundhousE.ENV"]
$db_server = $OctopusParameters["SqlServerInstance"]
$db_name = $OctopusParameters["DatabaseName"]
} else {
$env="LOCAL"
$db_server = ".\SqlExpress"
$db_name = "DatabaseName"
}
&$roundhouse_exe_path -s $db_server -d $db_name -f $scripts_dir --env $env --silent -o > $roundhouse_output_dir
In there you can see where we check for any octopus variables (parameters) that are passed in when Octopus runs the deploy script, otherwise we have some default values we use, and then we simply call the RoundhousE executable.
Then you just need to have that project as part of what gets packaged for Octopus, and then add a step in Octopus to deploy that package and it will execute that as part of each deployment.
We've looked at the RedGate solution and pretty much reached the same conclusion you have, unfortunately it's the cost that is putting us off that route.
The only things I can think of are to generate version controlled DB migration scripts based upon your existing database, and then execute these as part of your build process. If you're looking at .NET projects in future (that don't use a CMS), could potentially consider using entity framework code first migrations.
I remember looking into this a while back, and for me it seems that there's a whole lot of trust you'd have to get put into this sort of process, as auto-deploying to a Development or Testing server isn't so bad, as the data is probably replaceable... But the idea of auto-updating a UAT or Production server might send the willies up the backs of an Operations team, who might be responsible for the database, or at least restoring it if it wasn't quite right.
Having said that, I do think its the way to go, though, as its far too easy to be scared of database deployment scripts, and that's when things get forgotten or missed.
I seem to remember looking at using Red Gate's SQL Compare and SQL Data Compare tools, as (I think) there was a command-line way into it, which would work well with scripted deployment processes, like Team City, CruiseControl.Net, etc.
The risk and complexity comes in more when using relational databases. In a NoSQL database where everything is "document" I guess continuous deployment is not such a concern. Some objects will have the "old" data structure till they are updated via the newly released code. In this situation your code would need to be able to support different data structures potentially. Missing properties or those with a different type should probably be covered in a well written, defensively coded application anyway.
I can see the risk in running scripts against the production database, however the point of CI and Continuous Delivery is that these scripts will be run and tested in other environments first to iron out any "gotchas" :-)
This doesn't reduce the amount of finger crossing and wincing when you actually push the button to deploy though!
Having database deploy automation is a real challenge especially when trying to perform the build once deploy many approach as being done to native application code.
In the build once deploy many, you compile the code and creates binaries and then copy them within the environments. From the database point of view, is the equivalent to generate the scripts once and execute them in all environments. This approach doesn't handle merges from different branches, out-of-process changes (critical fix in production) etc…
What I know works for database deployment automation (disclaimer - I'm working at DBmaestro) as I hear this from my customers is using the build and deploy on demand approach. With this method you build the database delta script as part of the deploy (execute) process. Using base-line aware analysis the solution knows if to generate the deploy script for the change or protect the target and not revert it or pause and allow you to merge changes and resolve the conflict.
Consider a simple solution we have tried successfully at this thread - How to continuously delivery SQL-based app?
Disclaimer - I work at CloudMunch
We using Octopus Deploy and database projects in visual studio solution.
Build agent creates a nuget packages using octopack with a dacpac file and publish profiles inside and pushes it onto NuGet server.
Then release process utilizes the SqlPackage.exe utility to generate the update script for the release environment and adds it as an artifact to the release.
Previously created script executed in the next step with SQLCMD.exe utility.
This separation of create and execute steps gives us a possibility to have a manual step in between, so that someone verifies before the script is executed on Live environment, not to mention, that script saved as an artifact in the release can always be referred to, at any later point.
Would there be a demand I would provide more details and step scripts.
I'm trying to implement automated integration tests for my application. It's a very complex monster. You could say that its database and part of the filesystem are part of its state, because it saves image files in the hard drive, and references to those in the DB. The software needs all those, in a coherent state, to work properly.
Back to writing tests: To run any relevant test, I need some image files in the filesystem, and certain records filled in the database. I thought of putting all of these in a separate folder called TestEnvironmentData in the repository, and retrieving them from the Continuous Integration Server (Team City), but a colleague said the repo is quite full as it is, and that I should set up a special directory, and databases, only in the Continuous Integration server. I don't like that because the tests success depend on me manually mantaining stuff in the server, and restoring initial state before every test becomes cumbersome.
What do you guys do when you need to write integration tests for an app like this? The main goal is having an automated test harness to approach a large scale refactoring. There's lots of spaghetti code and the app's current architecture is hardly unit testable, that's why I decided on integration tests first.
Any alternative approach is welcome.
Developer Repeatability is key when setting up a Continous Integrations Server. I have set one up for my last three employers and I have found the key to success is the developers being able to run the same tests from their dev system in order to get the same results as the CI Server.
The easiest way to do this would be to check in the test artifacts into source control but you could also use dropbox or a Network Share that you copy them from in one of the build steps.
For a .Net solution I have always used MsBuild as you can most easily replicate the build process of Visual Studio and get the same binaries/deployables. As for keeping your database in sync so that tests can be repeatable in the past I used the MbUnit test framework and the [Rollback] attribute as it would roll back any changes to Sql Server that happened in the test. I believe that Nunit now has this attribute as well.
The CI server is great for finding code that breaks existing functionality but unless developers can reproduce the error on their machine they won't trust the CI server for some time.
First of all, we use Maven to build our code. It's like ant, but it relies on convention instead of configuration for many things, like Ruby On Rails does. One of those conventions is a standardized directory structure:
(project)----src----main----(language)
| | \--resources
| \--test----(language)
| \--resources
\--target---...
Using a directory structure like this makes it easy to keep your application resources and testing resources near each other, yet still be able to build for test or build for production, or just build both but just package up the application parts after running the tests.
As far as resetting the database between tests, how you do that is greatly dependent on the DBMS you're using. For instance, if you're using MySQL it's very easy to get the test data the way you want and do a mysqldump to a file you then load before the test. With other DBMSs you may have to drop and recreate the tables and reload the data, or make separate tables for the starting point and use a CREATE/SELECT sql statement to duplicate it each time.
There really is no reliable way around the "reset the database between tests" step.
We are working on a project where database requirement is not clear. So we are building a database agnostic application.
See my previous question here: Database Agnostic Application
Now I want to test my Spring application DAO with multiple database. I've written number of test cases using TestNG and DBUnit.
When I run these test in a CI environment, I want them to test the application against all the configured databases. I've installed the databases on the 'test server'.
e.g. I want something like this:
for ( each database configured ) {
run each dao test
}
Not sure what is the best way of doing this? And help is welcome.
Thanks,
Adi
If you want to be database independent, you have to test against every single database system you want to support. There are very fine differences which leak through Hibernate.
What I did in the past was to make the test retrieve their database configuration through some System Property. Typically by using hibernate_.property instead of the default hibernate.property. Then setup CI Jobs, which set the property to different values and provide one hibernate_xxx.property for every database to test against. I did this using JUnit Rules, to have the logic in one place. Don't know the apropriate tool for TestNG
I'm not to fond of the loop construct you are hinting at, because it might make it difficult to run a test suit against a single specific database.
I'm also not to fond of dbunit, because it seems to make maintaining testdata rather painful. I prefer in most cases a handcrafted DSL. Have a look at some articles I wrote about it:
http://blog.schauderhaft.de/2011/03/13/testing-databases-with-junit-and-hibernate-part-1-one-to-rule-them/
http://blog.schauderhaft.de/2011/03/20/testing-databases-with-junit-and-hibernate-part-2-the-mother-of-all-things/
http://blog.schauderhaft.de/2011/03/27/testing-databases-with-junit-and-hibernate-part-3-cleaning-up-and-further-ideas/
If you're building a database agnostic application and not using any of the inherent features of a specific database vendor, then the scope of your test cases should be to test the setup, manipulation, and accessing of the data through the DAO objects and less with the testing of the actual database backend. Hibernate 3.5 has dialects available for both Oracle 11g and DB2, so if you were writing test cases that tested the integration of the database agnostic application with a specific database vendor, then really what you are doing is testing that the hibernate dialects do as they say they do (which I'm sure has been covered by test cases in the hibernate project).
In other words, in your case I would think that the testing should focus more on the DAO retrieving the data that you think that it will retrieve after you've set that data up, and in-memory databases are fine for that.
Now all that said, both DB2 and Oracle have very good documentation related to setup. Indeed, both of them have "wizards" to do that. If you still think that it's prudent to test adding data to the database and retrieving it from the physical, non-in-memory database, then I would recommend setting up a "test database" environment and pointing your datasource to that during your continuous integration tests. If you're using Hudson or Jenkins for CI, you can set it up to run a script after the build completes that will truncate the database tables so that the next round of tests work from a blank slate.
EDIT:
I just saw the updates that you posted to your question, so let me address them. Since you already have the databases setup and configured then what you really want to do is dynamically select what the database should be. One way to do this would be to setup your datasource using System properties that can be inherited from a properties file, and running your tests in a "DB2-test" environment and an "Oracle-test" environment. Using this method, you'll have to setup the datasource programmatically and have it read system environment variables to determine which database it connects to. This would essentially require you to change your CI script to run the DB2-test environment first, then the Oracle-test environment following that -- your test suites will run twice.
Hope this helps!
Unit 4.9 has a new Feature: TestRule
You should be able to write a rule, that repeat a test for different databases.
There is this stack overflow question: How to Re-run failed JUnit tests immediately?
It is a slightly different question, but the solution should be the same technique.
We are developing some testing infrastructure and I have hit a coders block (lack of sleep?)...this seems like it would be a solved problem but I haven't found what I'm looking for via google.
I would like to automatically push builds from our CI server (TeamCity) to a number of machines (growing, but currently 30). These are several WinForms apps and a number of dlls. Once deployed, I would like to kick off tests (NUnit, for both unit and integration tests) and report all results (back to CI? or somewhere else? Not sure).
The target machines are a number of platforms (Win7,Vista, XP, Server 2k8, Server 2k3, Ubuntu, Fedora, Suse, x64, x86, maybe macs down the line)
This gets me part way there (the actual push). But I can't find existing solutions for 'push starting' the tests and reporting back. So far I am thinking of combining the link (or similar) with custom code running on each client machine that watches the deploy directory, runs the tests and reports the results.
Does anyone know of existing solutions?
Links?
Done something similar and care to share?
Edit
If possible, we prefer .net based solutions, but it isn't strictly necessary. I would have tagged the question as such, but ran out of tags :)
You could use KwateeSDCM to both push and start on all the platforms you mention, including mac. However, you'll have to do some coding to get reports out. I'm not familiar with TeamCity but maybe you could push a script along with your application which could then transfer the test results via ftp to a server accessible by TeamCity.
Have a look at: STAF (Software test Automation Framework)
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of reusable components, called services (such as process invocation, resource management, logging, and monitoring).
Which includes STAX:
STAX is an execution engine which can help you thoroughly automate the distribution, execution, and results analysis of your testcases.
And there's an article here:
http://agiletesting.blogspot.com/2004/12/stafstax-tutorial.html
Assuming you have the push part done already, and you don't mind using a TeamCity license, you can create a TeamCity Command Line Runner build configuration or NUnit test configuration that kicks off the tests on a properly configured agent. The build trigger for this test config would be successful completion of the application build.
So far I have ended up using a seperate build step in TeamCity that executes a bat script that in turn fires of tasks to the list of machines using PsExec. So far my trial runs it is working ok, though I now need to parallelize the copying of build output...
Thanks for the input to those who have provided it.