I am wondering if anyone knows the proper way to unit test a restful webservice. I have a set of webservices built using recess, and I would like to write test code for them. Unfortunately, since my webservices are tied to a database, my tests end up populating the database which seems like a problem.
I am mostly asking about the proper approach to dealing with this from a unit test standpoint. Do I clear the database of the values I have inserted after testing? Do I have a special test database with a whole set of special test routes? I am at a bit of a loss for the best way to approach this.
Obviously in other cases of similar database wrapper classes you would just pass in a dummy database that you set up at the beginning of the tests. This seems like it would be much more challenging though when it comes to working with a restful framework like recess.
I'd appreciate any thought you all might have on the right way to deal with tests saving information to the database.
Thanks in advance.
Generally when testing a web service you are testing the full stack, from the outside in. This means you request a resource and check if the results conform to your expectations.
In nearly all cases populating the database right before every request is a good approach. It might seem like overkill, but in reality with a web service you can't guarantee proper test coverage by mocking/stubbing various elements.
Coming from the Ruby world, Cucumber is the ideal approach as it lets you test from a high level. When you combine this with Rspec to do actual unit testing (lower level tests that query your objects directly) you get the best of both worlds. These libraries even come with something called 'database cleaner' which will manage populating and depopulating the database for you.
You might find the following blog post by Rspec's author very helpful, as it explains brilliantly why you should avoid too much mocking and stubbing. http://blog.davidchelimsky.net/2011/09/22/avoid-stubbing-methods-invoked-by-a-framework/
Generally speaking you have two options:
1) Use a dedicated test database with known data on which you can set your expectations - replace the DB with a "pristine DB" before starting testing. This would be considered integration testing since you are in fact dependent on the database.
2.) Make your code independent of the actual data store and pass in the dependency to the persistence layer. For unit testing you can write (or mock out) a custom persistence layer/object that allows you to observe the state changes that your are unit testing.
A healthy mix of both depending on the scenario usually provides good coverage.
Also instead of testing your Restful web service consider just delegating to a POCO within each service endpoint and then just test these POCOs directly - much easier testable and all you have left to do is verifying the mapping between service endpoint and POCO.
My understanding is that if you do your tests in this order you can test all the verbs but there will be no additional data in the DB at the end.
POST ( add a new record)
GET ( fetch the newly added record)
PUT/PATCH ( modify the newly added record)
DELETE (delete the newly added record)
Of course somebody else using the database at the same time might see transient values during the duration of the test.
Related
Probably there are already around some answers to this question but I haven't find the one I was looking for my specific scenario. So, here is my situation: I'm working on a web app made in Angular where all the unit tests are using mock data. Then we have some end to end tests written in Protractor. I'm not very excited about them because we are testing the user interface with the data we get from a live api. I think we're using this approach because we have no control on the back-end but the side effect of this is that the database could change a mess up our tests. Also, the api we're using for the e2e is runnung on an internal network meaning that we cannot run tests outside of the office. I was thinking about mocking the http responses in order to mock the database and being able to run all the tests from anywhere. The problem is that the backend logic could act differently from the one we simulate in our tests meaning that as soon as we deploy the application, it will work in an unexpected way.
What is the best practice and workflow to follow in a similar situation?
Best practice is subjective but there are known solutions each with pros/cons.
Using a shared environment
If you have manual testing on the same environment as your automated tests you will risk someone screwing with your tests. Copying data from production to this environment will also halt your tests and is not good. There is extra effort in making your test idempotent by ensuring the setup is in the correct state that your test expects as well as ensuring the data set up is not conflicting with manual tests. It is recommended when you create an entity during test setup to have it created with some unique token related to the test so that it is unique for that test. This is just hard and costly.
Using separate e2e environment
This is clearly easier on your test idempotence as you have more control of the data and no manual intervention. You can empty the database or reseed it using several solutions (see below) before every test or group of tests. Still you must be careful ensuring tests do not depend on each other or interfere with other tests.
Mock the APIs
You can mock APIs however it is not a true e2e test. Consumer-driven contracts will work if you know that the APIs are testing against specific output and you can then use those outputs as mocks for your inputs of the e2e. These tests are blazing fast. If you don't have control over your environment and its data, or it is a 3rd party system it is recommended to mock the api. You risk not testing the real integration which can cause a lot of failures.
Use APIs to set up test data
This is a pretty good solution as not only does it catch issues with APIs but it keeps your e2e tests focused only on the area being tested and you do not have to set up data using the GUI. Test setup and clean up can be managed this way. It may be quicker than using the GUI to set up and certainly not quicker than mocking the API responses.
Use the GUI to set up the test data
This can work but you must be smart about it. Since you are sharing the environment with manual testing you must ensure the data is in the correct state. It is wise to create separate entities related to your tests and not share any test cases that someone would touch manually testing around. This is slower. This complicates your tests as you spend a majority of your time navigating around and setting up things in the GUI.
Use scripts to load the data directly to the database
Avoid this because there is probably business logic that you are missing and will lead to incorrect states. It is better to go through the API to load data as it can validate the input and run any business logic.
Here are some relevant resources to follow up on:
Martin Fowler's write-up on testing microservices
https://medium.com/how-we-build-fedora/e2e-testing-with-angular-protractor-and-rails-725fbefb8149#.9rziv2gtp
How about getting a test version of the backend deployed that has a limited amout of data in?
That way, after each round of testing has completed the database can then be reset with the original datasets loaded in.
This would ensure consistency in your result across tests, and means if the backend guys make changes to their master branch, it wont affect your tests.
I am trying specflow for the first time for a WPF project with MVVM pattern. So, I have repository, controller, view and view model. Repository makes calls to webservice wich hits the database and gives back the data. View model has methods to validate the input from the user and make calls to repository methods.
In my specflow, should I be making a complete call including the repository methods or should I mock those methods using Moq? Does it make sense?
The short answer:
You could make you calls hit the database, but I would recommend mocking them out, unless you have logic in your DB
The long answer:
There's two competing ways of looking at this question. Firstly what kind of testing are you expecting to use SpecFlow for, and secondly how easy is it going to be to access your database.
If you are using SpecFlow to test your low level technical requirements, then you really are doing unit testing in a Specification by example style. So this really ought to dictate that you use mocking to isolate your units. (Personally I'd stick to using NUnit for these tests.)
If however you are using SpecFlow to test your business scenarios (i.e. Acceptance testing) then your scenarios are reliant on more than one unit to provide the functionality. This is more like Integration or System testing and so many components will need to be involved and in theory since we are testing the whole system, we should include a DB in that, particularly if you have even one stored procedure or view that you might want to regression test later.
However, the second way of looking at this is how much pain is required to have a database available for your testing.
If there is more than one person in your development team, what strategies would you need to avoid failing if you run your tests at the same time?
For example you could generate a new unique customer in each run and only access updates and queries against that customer, then you won't end up with the wrong number of orders.
How will you reset your database after your tests run?
E.g. If you use the right kind of database you can check an empty .mdb into your source control and just rollback/revert to that.
I've personally found that for a simple DB without stored procs and views, mocking is better, but as soon as you've added business logic into the DB, well, you need to test that logic somehow.
Let me describe the scenario:
Our VB.NET web application talks to a third party data provider via webservices. It also saves data in a HUGE SQLServer database which has extensive business logic implemented in stored procs and triggers. The webservice provider also employs convoluted business logic which is quite dynamic.
The dilemma is that both the webservice provider and the sqlserver are essentially live systems and our company has no access to them outside of normal operations (web and SQL calls). Neither offer helpful support on their end.
Additionally, the webservice has no 'test mode' and all calls are treated as live transactions. It is not possible to mimic the logic in either of these systems, nor is it possible getting a copy of the sqlserver DB.
So, my question is:
How do you do any kind of testing (manual or automated) of our VB.NET application against these live systems?
Your suggestions would be appreciated.
Let's start by challenging your assumption "It is not possible to mimic the logic in either of these systems".
Of course you can mimic the behavior, if you use the right tools. Your system interacts with the database by way of a database object; your system should interact with the web service through a web service object. Either or both of these objects can be mocked through the use of an appropriate mocking framework. For any unit test call to the database object, you create a mock for that object, set its expectations and result for the call, and then hand the mock to your Code Under Test (CUT) instead of the 'real' database object. Your code calls the mock, which then compares its arguments against the pre-set expectations, and hands back the expected result (instead of actually communicating with the database). Your code then operates on the result. If the method arguments don't match the expectations, the mock object will throw an exception.
You can read articles about mock objects and unit testing for .Net here:
Mock Objects to the Rescue! Test Your .NET Code with NMock
How Typemock Isolator Simplifies Unit Testing
Mind you, tools like NMock and Typemock make the job easier, but it's still hard -- you need to design your code to be tested, not just write the code first and pray that you can test it later.
You might want to talk to your webservice provider -- every third-party webservice that I've ever interacted with beyond simple queries has had a test mode (you use test credentials and a test server, instead of the live server). Any transactions to the test server get cleaned up at the end of the day. If they don't offer a test service AND their service involves more than simple queries, then I'd strongly recommend finding another service provider.
There's one other strategy that you can take for working with a database, under certain circumstances: use transactions. When you open your database connection during a unit test, open a transaction. At the end of each unit test, rollback the transaction. It's a simple idea, but the devil is in the details, and there will be chaos if you screw up and accidentally commit the transaction. I don't recommend it, but I worked like this for 2 years on one project.
It sounds like you're building a line-of-business application: software that will be used in one place, by one company. In that scenario testing is not really that important. The best testers find around 20% of the issues in your code, so whether you test or not, you still have to be prepared to deal with issues in production.
So, I'd skip the testing, and instead focus on powerful scaffolding that allows you to develop the application against the live servers in a controlled way:
Create real good logging. Up to half the code can deal with exception handling and logging. If you can, log to a database.
Create lots and lots of consistency checks. The more assumptions you check, the better. When you do a write operation (like placing a web service order) double the amount of checks. If a check fails, stop the application entirely. Don't limp on: stop and analyze what went wrong.
Develop the application one part at a time. Test each part in close cooperation with the end users. First, test a few items manually; then 10; then 100; and if you and the end users are happy, turn it on.
I know from experience this can work very well. The close cooperation with the end users allows you to understand their needs better, and generally add features that they really appreciate.
A word of warning: although users will think of you as someone who gets thinks done, Enterprise Architects will think of you as a hacker. Depending on the organization you're in, this can be either very healthy or very unhealthy :)
I have an application that is database intensive. Most of the applications methods are updating data in a database. Some calls are wrappers to stored procedures while others perform database updates in-code using 3rd party APIs.
What should I be testing in my unit tests? Should I...
Test that each method completes without throwing an exception -or-
Validate the data in the database after each test to make sure the state of data is as expected
My initial thought is #2 but my concern is that I would be writing a bunch of framework code to go along with my unit tests. I read that you shouldn't write a bunch of framework code for unit testing.
Thoughts?
EDIT: What I mean by framework is writing a ton of other code that serves as a library to the unit testing code...not a third party framework.
I do number 2, i.e., test the update by updating a record, and then reading it back out and verifying that the values are the same as the ones you put in. Do both the update and the read in a transaction, and then roll it back, to avoid permanent effect on the database. I don't think of this as testing Framework code, any more than I think of it as testing OS code or networking code... The framework (if you mean a non-application specific Database access layer component) should be tested and validated independently.
There's a third option, which is to use a mock database-access object that knows how to respond to an update as if it had been connected to a live database, but it doesn't really execute the query against a database.
This technique can be used to supplement testing against a live database. This is not the same as testing against a live database, and shouldn't substitute for that kind of testing. But it can be used at least to test that the invocation of the database update by your class was done with proper inputs. It also typically runs a lot faster than running tests against a real database.
You must test the actual effect of the code on the data, and its compliance with the validation rules etc., not just that no exceptions are raised - that would be a bit like just checking a procedure compiles!
It is difficult testing database code that performs inserts, updates or deletes (DML), because the test changes the environment it runs in, i.e. the database. Running the same procedure several times in a row could (and probably should) have different results each time. This is very different to unit testing "pure code", which you can run thousands of times and always get the same result - i.e. "pure code" is deterministic, database code that performs DML is not.
For this reason, you do often need to build a "framework"to support database unit tests - i.e. scripts to set up some test data in the right state, and to clean up after the test has been run.
If you are not writing to the database manually and using a framework instead (jvm, .net framework, ...), you can safely assume that the framework writes to database correctly. What you must test is if you are using the framework correctly.
Just mock the database methods and objects. Test if you are calling them and retrieving data back correctly. Doing this will give you the opportunity to write your tests easier, run them much more faster and make them parallel with no problems at all.
They shouldn't be unit tested at all! The whole point of those methods is to integrate with the outside world (i.e. the database). So, make sure your integration tests beat the you-know-what out of those methods and just forget about the unit tests.
They should be so simple that they are "obviously bug-free", anyway – and if they aren't, you should break them up in one part which has the logic and a dumb part which just takes a value and sticks it in the database.
Remember: the goal is 100% test coverage, not 100% unit test coverage; that includes all of your tests: unit, integration, functional, system, acceptance and manual.
If the update logic is complex then you should do #2.
In practice the only way to really unit test a complex calculation and update
like say, calculating the banking charges on a set of customer transactions,
is to intialise a set of tables to known values at the start of your
unit test and test for the expected values at the end.
I use DBUnit to load the database with data, execute the update-logic and finally read the updated data from the database and verify it. Basically #2.
This past summer I was developing a basic ASP.NET/SQL Server CRUD app, and unit testing was one of the requirements. I ran into some trouble when I tried to test against the database. To my understanding, unit tests should be:
stateless
independent from each other
repeatable with the same results i.e. no persisting changes
These requirements seem to be at odds with each other when developing for a database. For example, I can't test Insert() without making sure the rows to be inserted aren't there yet, thus I need to call the Delete() first. But, what if they aren't already there? Then I would need to call the Exists() function first.
My eventual solution involved very large setup functions (yuck!) and an empty test case which would run first and indicate that the setup ran without problems. This is sacrificing on the independence of the tests while maintaining their statelessness.
Another solution I found is to wrap the function calls in a transaction which can be easily rolled back, like Roy Osherove's XtUnit. This work, but it involves another library, another dependency, and it seems a little too heavy of a solution for the problem at hand.
So, what has the SO community done when confronted with this situation?
tgmdbm said:
You typically use your favourite
automated unit testing framework to
perform integration tests, which is
why some people get confused, but they
don't follow the same rules. You are
allowed to involve the concrete
implementation of many of your classes
(because they've been unit tested).
You are testing how your concrete
classes interact with each other and
with the database.
So if I read this correctly, there is really no way to effectively unit-test a Data Access Layer. Or, would a "unit test" of a Data Access Layer involve testing, say, the SQL/commands generated by the classes, independent of actual interaction with the database?
There's no real way to unit test a database other than asserting that the tables exist, contain the expected columns, and have the appropriate constraints. But that's usually not really worth doing.
You don't typically unit test the database. You usually involve the database in integration tests.
You typically use your favourite automated unit testing framework to perform integration tests, which is why some people get confused, but they don't follow the same rules. You are allowed to involve the concrete implementation of many of your classes (because they've been unit tested). You are testing how your concrete classes interact with each other and with the database.
DBunit
You can use this tool to export the state of a database at a given time, and then when you're unit testing, it can be rolled back to its previous state automatically at the beginning of the tests. We use it quite often where I work.
The usual solution to external dependencies in unit tests is to use mock objects - which is to say, libraries that mimic the behavior of the real ones against which you are testing. This is not always straightforward, and sometimes requires some ingenuity, but there are several good (freeware) mock libraries out there for .Net if you don't want to "roll your own". Two come to mind immediately:
Rhino Mocks is one that has a pretty good reputation.
NMock is another.
There are plenty of commercial mock libraries available, too. Part of writing good unit tests is actually desinging your code for them - for example, by using interfaces where it makes sense, so that you can "mock" a dependent object by implmenting a "fake" version of its interface that nonetheless behaves in a predictable way, for testing purposes.
In database mocks, this means "mocking" your own DB access layer with objects that return made up table, row, or dataset objects for your unit tests to deal with.
Where I work, we typically make our own mock libs from scratch, but that doesn't mean you have to.
Yeah, you should refactor your code to access Repositories and Services which access the database and you can then mock or stub those objects so that the object under test never touches the database. This is much faster than storing the state of the database and resetting it after every test!
I highly recommend Moq as your mocking framework. I've used Rhino Mocks and NMock. Moq was so simple and solved all the problems I had with the other frameworks.
I've had the same question and have come to the same basic conclusions as the other answerers here: Don't bother unit testing the actual db communication layer, but if you want to unit test your Model functions (to ensure they're pulling data properly, formatting it properly, etc.), use some kind of dummy data source and setup tests to verify the data being retrieved.
I too find the bare-bones definition of unit testing to be a poor fit for a lot of web development activities. But this page describes some more 'advanced' unit testing models and may help to inspire some ideas for applying unit testing in various situations:
Unit Test Patterns
I explained a technique that I have been using for this very situation here.
The basic idea is to exercise each method in your DAL - assert your results - and when each test is complete, rollback so your database is clean (no junk/test data).
The only issue that you might not find "great" is that i typically do an entire CRUD test (not pure from the unit testing perspective) but this integration test allows you to see your CRUD + mapping code in action. This way if it breaks you will know before you fire up the application (saves me a ton of work when I'm trying to go fast)
What you should do is run your tests from a blank copy of the database that you generate from a script. You can run your tests and then analyze the data to make sure it has exactly what it should after your tests run. Then you just delete the database, since it's a throwaway. This can all be automated, and can be considered an atomic action.
Testing the data layer and the database together leaves few surprises for later in the
project. But testing against the database has its problems, the main one being that
you’re testing against state shared by many tests. If you insert a line into the database
in one test, the next test can see that line as well.
What you need is a way to roll back the changes you make to the database.
The TransactionScope class is smart enough to handle very complicated transactions,
as well as nested transactions where your code under test calls commits on its own
local transaction.
Here’s a simple piece of code that shows how easy it is to add rollback ability to
your tests:
[TestFixture]
public class TrannsactionScopeTests
{
private TransactionScope trans = null;
[SetUp]
public void SetUp()
{
trans = new TransactionScope(TransactionScopeOption.Required);
}
[TearDown]
public void TearDown()
{
trans.Dispose();
}
[Test]
public void TestServicedSameTransaction()
{
MySimpleClass c = new MySimpleClass();
long id = c.InsertCategoryStandard("whatever");
long id2 = c.InsertCategoryStandard("whatever");
Console.WriteLine("Got id of " + id);
Console.WriteLine("Got id of " + id2);
Assert.AreNotEqual(id, id2);
}
}
If you're using LINQ to SQL as the ORM then you can generate the database on-the-fly (provided that you have enough access from the account used for the unit testing). See http://www.aaron-powell.com/blog.aspx?id=1125