I have a new project that needs SQL Server unit test, and CI/CD with VSTS.
Below is the features that are required
SQL server unit test against stored procedure, initial target tables setup for each test and clean up
Unit test in sql
CI/CD with VSTS and Git
Easy setup and easy to use
I looked into SSDT 2017, which seems good. But it seems it lacks a feature where common setup script can be shared easily between each test in Pre-Test step. It might lack other features that should be available for daily usage. But I might be wrong.
Which tool fits better for general sql server unit testing in 2017?
SQL Server Data Tools for Visual Studio
TSQLT
One of the reasons why there aren't more unit testing solutions out there for SQL development is because proper unit testing is inherently harder with databases so people don't do it. This is because databases maintain state and also referential integrity. Imagine writing a unit test for a stored procedure (order_detail_update_status) that updates a status flag on an order_detail table. The order_detail table has a dependency on the order_header and product tables, order_header in turn has foreign keys to customer and employee whilst the product table may depend on product_category, product_type and supplier. That is at least seven tables (probably more) that need to be populated with valid data just to write one test and all but one of those tables have nothing to do with the code under test.
So what you should be looking for in a unit testing solution is exactly that - the ability to test discrete units of code - with the minimum of set-up. So ideally, you would be able to just set up the required test data in order_detail and ignore the rest of the tables - I am aware of only one testing framework that allows you to do that.
Additionally, unit tests should have minimal reasons to fail, in the above example, order_detail_update_status just updates a single row on the order_detail table. If a new not null column is added to the customer table, which is not handled by the test set-up then you have a scenario where our test could fail for a totally unrelated reason. This makes for very brittle tests and, under the pressure of tight delivery deadlines, developers will quickly give up writing and maintaining tests.
A suite of unit tests should be runnable in any order, with no interdependencies and a good test framework should support this along with set-up, tear down and support for mocking objects (which may or may not be part of the same framework). In the above scenario, the ability to mock the order_detail table to test a module that only touches the order_detail table is one of the most important features if you don't want to spend huge amounts of time fixing tests that are failing for no "good" reason.
So in terms of your requirements, and the above points, there is only one framework that I am aware of that does all of this - tSQLt. This is based on real-world experience - we had over 6,000 tSQLt unit tests on my last project. It includes the following feautures:
Unit test stored procedures, functions and views
Mock tables, views and functions
Mock (or spy) stored procedures - either for isolation,
replay or pre-defined outcomes
Suite set-up
Automatic tear-down (as every test runs in it's own translation)
Unit tests are completely isolated and can be executed in any order
It works very well with VSTS in a CI/CD and, as all the unit tests are written in T-SQL, it is very easy to use.
The best way to use tSQLt in Visual Studio is to make use of composite projects - where application database objects and modules are maintained in one project whilst the tSQLt framework and all unit tests are part of a second project. There is a good aticle to get you started on this here.
I wrote a more detailed article on the benefits of tSQLt for Simple-Talk a few year back which might also be helpful
Note that Microsoft is promoting slacker, see e.g. Channel 9: SQL Server Database Unit Testing in your DevOps pipeline. We found it to work reasonably well in an Azure SQL setup. For tSQLt on Azure SQL I remember some issues around enabling CLR and TRUSTWORTHY options but also saw that it should still work, e.g. here:
Nikolai Thomassen: Azure SQL unit testing with tSQLt using Azure DevOps
SQLShack: SQL unit testing with the tSQLt framework for beginners
You can re-use scripts, you can do a lot of things. Quick answer to your question is just use tSQLt. There is no other unit testing frameworks for SQL Server to be so powerful/flexible and easy to use than tSQLt so far. Just start using and that's it. It is quite easy and quick to setup in SSDT. #datacentricity wrote you enough about that framework and if you are want to know more, then read the article he provided.
I'll just add few things to make your life a bit easier if you'll go tSQLt direction:
Use synonyms for all cross database or linked objects;
Create all the tSQLt objects using the standard script in SSMS and then import the objects into the SSDT
Create separate project for tSQLt objects and mark it as "the same database" as your database you are willing to test
Create pre-script in the tSQLt project and run the pre-script of your original database project from there
Create post-script in tSQLt project and run the post-script of your original database project from there
In the tSQLt post-script as the last statement write "EXEC tSQLt.RunAll"
Create publish script in tSQLt project and in the settings be sure that it will deploy "extended properties"
Make sure that all test classes (schemas) have extended properties statements
There might some other nuances but just start with something and I am pretty sure that you'll start loving tSQLt very soon.
I have a request to perform a "smoke test" for database deployments that I am doing within TFS. I would like to be able to Execute a database job after deployment in TFS that would be used to populate data. After that, the possibility to run some SQL statements with results of how many records were inserted, things like that.
I have looked into Unit Testing with SSDT, but was wondering if there are any other options (plus it seems you can only kick off SPROCs with that method).
Any advice would be much appreciated.
Assuming you already have a SSDT project and you're using the DACPAC to deploy as part of you build, and you are just wondering how to include the population of data as part of the deploy. What I usually do is just use a post-deploy script with a whole bunch of INSERT's (I typically split it up into separate files with INSERT's for each table, then include these in the main post-deploy script using the :r syntax). To generate the insert scripts I'll usually use the functionality in SSMS to script out a database, and in the advanced options just tell it to script data-only, then copy paste that into my post-deploy script.
Personally, I would use either sqlcmd EXEC tasks, but probably (more likely) this sqlcmd-msbuild task.
http://www.msbuildextensionpack.com/help/4.0.5.0/html/3b72c130-7fc9-8b8a-132c-62999e5b1183.htm
You can have INSERT scripts.
You could also incorporate some 'unit testing' methods that Andy Leonard discusses.
http://www.dotnetrocks.com/default.aspx?showNum=312
I would recommend that you do not do this as part of a build.
You want your build to be fast and that likely means that you do not have not should you need an instance of your application. You should use a release management tool to deploy your application to an environment and them run instance tests there.
You can use RM for Visual Studio 2013 to create a release pipeline that does this. These tools contain drag an drop elements for creating and updating databases.
http://nakedalm.com/building-release-pipeline-release-management-visual-studio-2013/
Have you considered tSQLt? tSQLt is open source and probably the most-used SQL Server unit test framework available. To use it correctly you'd insert the data within the test itself (INSERTs). This is a particularly cunning way of dealing with test set up, as tSQLt rolls back the transaction after the test has run, bringing your test environment back to a known state.
I want each of my unit (integration) test methods to use a clean and consistent database and specific test data for each test.
Could you please provide me some code samples/snippets and tell me what are best practices for the following questions for both scenarios EF 5 database first and model first.
How to create the database for each test method?
How to set up the test data for each test method?
How to delete the database for each test method?
SSDT project is used to handle database schema, how to use the current SSDT schema for each test run? So that the tests are always executed against the current development version of the database.
Please consider the following assumptions for above questions:
The unit tests shall be executed locally on dev machine and on server CI builds.
Each test may have different test data.
Manually defined .mdf test files should be avoided because several developers are working on the product and there is a potential risk that one deleoper overwrites the changes of the .mdf file which another developer may have checked in previously -> development process should be as simpel as possible.
SSDT is used, so maybe this an option to create the database (probably not a good one because I want the database to be created for each test) and I have no deep knowledge yet about SSDT possiblities.
A good performance of test execution time would be nice to have.
VS/TFS 2012 is used.
SQL Server 2012 is used.
Application is a C# desktop application.
Mocking EF context etc. is not an option.
I hope you can guide me into the right direction how to solve the 4 questions from above. I don´t know if EF provides some functionality (I think only for code first) for my challenges or if this all must be solved by executing SQL scripts or something like that.
Thanks!
I am searching for days now on several forums, blogs, MSDN etc. but I was not able to find any guidance on this topic so far. I will try to explain this post in a more detailed manner because I think information and documentation of SSDT development is not well documented and there exists no best practice document like VS 2010 database projects (http://vsdatabaseguide.codeplex.com/).
I am a C# developer (no DBA) and we are in the beginning of the development phase of a green field project (10 – 15 developers) and we are currently defining our development process including handling of database development.
The technology and tool chain we want to use:
EF 5 (model first, maybe we change this to database first because issues like views, indexes etc. are much easier to handle)
SSDT (SQL Server Data Tools)
VS 2012 / TFS 2012
MS Test for automated unit / integration tests
The development process is based on test driven development and looks like this:
Each feature is developed by one developer on a separate feature branch
Design and implement unit tests (=feature implementation)
If a feature requires database access then the developer has to
a) create / update the EF model
b) create the localDB database via EF´s „Generate database from model“
c) create / update the SSDT project via schema compare
d) create the unit tests with a test initialize method that creates a fresh database and according test data for each test
Merge the feature branch back into integration branch
After checking in the merge the CI build executes the unit / integration tests
So there are some points I am not 100% sure about how to solve them (especially database handling with unit tests) and I would appreciate if you can put me in the right direction:
How to solve the database creation for the automated unit tests:
a) Execute the SQL database generation script (which can be manually created previously via SSDT publish feature) for each executed test method? This is the option I would prefer because each test has a clean and consistent database state for each test. Is there a performance problem creating the localdb database for each test?
b) Or use msbuild task „SQLPublish“ or „sqlPackage.exe“? I think this is not the option to go for because this would be a one time thing and I want to create a new test database for each unit test.
c) Or create the test database manually and save the *.mdf file to the root of the source control folder and create a copy for each test? But this I would not prefer because a developer A could override the file which could have changes from another developer B who checked in his changes before. And this means that the developer
How to solve test data creation for the automated unit tests:
a) Execute test specific SQL script that inserts the appropriate test data for each test. I think this also means to create a new database as mentioned in point 1. Again this is my preferred option.
b) Or using EF to create test data seems not to be a clean way, because this depends on the EF model implementation which should actually be tested implicitly through the feature unit tests.
c) Or use manually created test database files. But this would make the development process more complicate for the developer. And this could also be overriden by other developers check ins.
Maybe its good to mention what we expect from our unit tests to be.The goal of our unit tests is not to test the database schema like stored procedures and so on. We want to test parts of our application features using „code“ unit tests that can also be seen as integration tests.
So does anyone of you have a similar development process and what are your experiences?
Any recommendations to improve our development process?
Are there any resources out there or best practices documents on SSDT development?
And the most important question for me, how did you solve the automated unit testing including proper database handling and integration tests?
When you need a database, it isn't a unit test. For unit testing in combination with entity framework you should use a faked dbcontext.
I wonder how you guys manage deployment of a database between 2 SQL Servers, specifically SQL Server 2005.
Now, there is a development and a live one. As this should be part of a buildscript (standard windows batch, even do with current complexity of those scripts, i might switch to PowerShell or so later), Enterprise Manager/Management Studio Express do not count.
Would you just copy the .mdf File and attach it? I am always a bit careful when working with binary data, as this seems to be a compatiblity issue (even though development and live should run the same version of the server at all time).
Or - given the lack of "EXPLAIN CREATE TABLE" in T-SQL - do you do something that exports an existing database into SQL-Scripts which you can run on the target server? If yes, is there a tool that can automatically dump a given Database into SQL Queries and that runs off the command line? (Again, Enterprise Manager/Management Studio Express do not count).
And lastly - given the fact that the live database already contains data, the deployment may not involve creating all tables but rather checking the difference in structure and ALTER TABLE the live ones instead, which may also need data verification/conversion when existing fields change.
Now, i hear a lot of great stuff about the Red Gate products, but for hobby projects, the price is a bit steep.
So, what are you using to automatically deploy SQL Server Databases from Test to Live?
I've taken to hand-coding all of my DDL (creates/alter/delete) statements, adding them to my .sln as text files, and using normal versioning (using subversion, but any revision control should work). This way, I not only get the benefit of versioning, but updating live from dev/stage is the same process for code and database - tags, branches and so on work all the same.
Otherwise, I agree redgate is expensive if you don't have a company buying it for you. If you can get a company to buy it for you though, it really is worth it!
For my projects I alternate between SQL Compare from REd Gate and the Database Publishing Wizard from Microsoft which you can download free
here.
The Wizard isn't as slick as SQL Compare or SQL Data Compare but it does the trick. One issue is that the scripts it generates may need some rearranging and/or editing to flow in one shot.
On the up side, it can move your schema and data which isn't bad for a free tool.
Don't forget Microsoft's solution to the problem: Visual Studio 2008 Database Edition. Includes tools for deploying changes to databases, producing a diff between databases for schema and/or data changes, unit tests, test data generation.
It's pretty expensive but I used the trial edition for a while and thought it was brilliant. It makes the database as easy to work with as any other piece of code.
Like Rob Allen, I use SQL Compare / Data Compare by Redgate. I also use the Database publishing wizard by Microsoft. I also have a console app I wrote in C# that takes a sql script and runs it on a server. This way you can run large scripts with 'GO' commands in it from a command line or in a batch script.
I use Microsoft.SqlServer.BatchParser.dll and Microsoft.SqlServer.ConnectionInfo.dll libraries in the console application.
I work the same way Karl does, by keeping all of my SQL scripts for creating and altering tables in a text file that I keep in source control. In fact, to avoid the problem of having to have a script examine the live database to determine what ALTERs to run, I usually work like this:
On the first version, I place everything during testing into one SQL script, and treat all tables as a CREATE. This means I end up dropping and readding tables a lot during testing, but that's not a big deal early into the project (since I'm usually hacking the data I'm using at that point anyway).
On all subsequent versions, I do two things: I make a new text file to hold the upgrade SQL scripts, that contain just the ALTERs for that version. And I make the changes to the original, create a fresh database script as well. This way an upgrade just runs the upgrade script, but if we have to recreate the DB we don't need to run 100 scripts to get there.
Depending on how I'm deploying the DB changes, I'll also usually put a version table in the DB that holds the version of the DB. Then, rather than make any human decisions about which scripts to run, whatever code I have running the create/upgrade scripts uses the version to determine what to run.
The one thing this will not do is help if part of what you're moving from test to production is data, but if you want to manage structure and not pay for a nice, but expensive DB management package, is really not very difficult. I've also found it's a pretty good way of keeping mental track of your DB.
If you have a company buying it, Toad from Quest Software has this kind of management functionality built in. It's basically a two-click operation to compare two schemas and generate a sync script from one to the other.
They have editions for most of the popular databases, including of course Sql Server.
I agree that scripting everything is the best way to go and is what I advocate at work. You should script everything from DB and object creation to populating your lookup tables.
Anything you do in UI only won't translate (especially for changes... not so much for first deployments) and will end up requiring a tools like what Redgate offers.
Using SMO/DMO, it isn't too difficult to generate a script of your schema. Data is a little more fun, but still doable.
In general, I take "Script It" approach, but you might want to consider something along these lines:
Distinguish between Development and Staging, such that you can Develop with a subset of data ... this I would create a tool to simply pull down some production data, or generate fake data where security is concerned.
For team development, each change to the database will have to be coordinated amongst your team members. Schema and data changes can be intermingled, but a single script should enable a given feature. Once all your features are ready, you bundle these up in a single SQL file and run that against a restore of production.
Once your staging has cleared acceptance, you run the single SQL file again on the production machine.
I have used the Red Gate tools and they are great tools, but if you can't afford it, building the tools and working this way isn't too far from the ideal.
I'm using Subsonic's migrations mechanism so I just have a dll with classes in squential order that have 2 methods, up and down. There is a continuous integration/build script hook into nant, so that I can automate the upgrading of my database.
Its not the best thign in the world, but it beats writing DDL.
RedGate SqlCompare is a way to go in my opinion. We do DB deployment on a regular basis and since I started using that tool I have never looked back.
Very intuitive interface and saves a lot of time in the end.
The Pro version will take care of scripting for the source control integration as well.
I also maintain scripts for all my objects and data. For deploying I wrote this free utility - http://www.sqldart.com. It'll let you reorder your script files and will run the whole lot within a transaction.
I agree with keeping everything in source control and manually scripting all changes. Changes to the schema for a single release go into a script file created specifically for that release. All stored procs, views, etc should go into individual files and treated just like .cs or .aspx as far as source control goes. I use a powershell script to generate one big .sql file for updating the programmability stuff.
I don't like automating the application of schema changes, like new tables, new columns, etc. When doing a production release, I like to go through the change script command by command to make sure each one works as expected. There's nothing worse than running a big change script on production and getting errors because you forgot some little detail that didn't present itself in development.
I have also learned that indexes need to be treated just like code files and put into source control.
And you should definitely have more than 2 databases - dev and live. You should have a dev database that everybody uses for daily dev tasks. Then a staging database that mimics production and is used to do your integration testing. Then maybe a complete recent copy of production (restored from a full backup), if that is feasible, so your last round of installation testing goes against something that is as close to the real thing as possible.
I do all my database creation as DDL and then wrap that DDL into a schema maintainence class. I may do various things to create the DDL in the first place but fundamentally I do all the schema maint in code. This also means that if one needs to do non DDL things that don't map well to SQL you can write procedural logic and run it between lumps of DDL/DML.
My dbs then have a table which defines the current version so one can code a relatively straightforward set of tests:
Does the DB exist? If not create it.
Is the DB the current version? If not then run the methods, in sequence, that bring the schema up to date (you may want to prompt the user to confirm and - ideally - do backups at this point).
For a single user app I just run this in place, for a web app we currently to lock the user out if the versions don't match and have a stand alone schema maint app we run. For multi-user it will depend on the particular environment.
The advantage? Well I have a very high level of confidence that the schema for the apps that use this methodology is consistent across all instances of those applications. Its not perfect, there are issues, but it works...
There are some issues when developing in a team environment but that's more or less a given anyway!
Murph
I'm currently working the same thing to you. Not only deploying SQL Server databases from test to live but also include the whole process from Local -> Integration -> Test -> Production. So what can make me easily everyday is I do NAnt task with Red-Gate SQL Compare. I'm not working for RedGate but I have to say it is good choice.