Related
We have an integration test suite that contains numerous tests executing repository classes.
The objective is to have a [OneTimeSetup] method in the BaseTestFixture, that will create/populate each target database (Postgres/SQL Server) only once before all tests and teardown after all tests.
Got this error:
nunit OneTimeSetUp: SetUp and TearDown methods must not have parameters
How can we run the entire test suite against Postgres, SQL server and both without duplicating tests?
Thanks.
Interesting question. I can't really think of an 'out-the-box' solution myself
One simple workaround would be to do two separate console runs and use the --params flag. That way, you could run a different setup for each database type, dependent on the TestParameters value passed in.
A nicer alternative may be to implement a custom attribute, which would allow you to parameterize SetUpFixtures. (There's an existing discussion on adding this feature here- although it hasn't garnered much interest since 2016.) I think it would be reasonably possible to do this as a custom attribute without modifying NUnit however.
Take a look at how SetUpFixtureAttribute is implemented. I would think you'd want to create your own IFixtureBuilder attribute which works in a similar way, except can be parameterised, and return two suites, with a different setup for each database. I think that would work, although it's not functionality I'm totally familiar with myself.
I have inherited a Web API 2 project written in C#/.NET that uses ADO.NET to access an SQL Server database.
The data access layer of the project contains many methods which look similar to this:
public class DataAccessLayer
{
private SqlConnection _DBConn;
public DataAccessLayer()
{
_DBConn = new SqlConnection(ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString);
}
public string getAllProductsAsJSON()
{
DataTable dt = new DataTable();
using (SqlConnection con = _DBConn)
{
using (SqlCommand cmd = new SqlCommand("SELECT productId, productName FROM product ORDER BY addedOn DESC", con))
{
cmd.CommandType = CommandType.Text;
// add parameters to the command here, if required.
con.Open();
SqlDataAdapter da = new SqlDataAdapter(cmd);
da.Fill(dt);
return JsonConvert.SerializeObject(dt);
}
}
}
// ... more methods here, but all basically following the above style of
// opening a new connection each time a method is called.
}
Now, I want to write some unit tests for this project. I have studied the idea of using SQL transactions to allow for insertion of mock data into the database, testing against the mock data, and then rolling back the transaction in order to allow for testing against a "live" (development) database, so you can have access to the SQL Server functionality without mocking it out completely (e.g. you can make sure your views/functions are returning valid data AND that the API is properly processing the data all at once). Some of the methods in the data access layer add data to the database, so the idea is that I would want to start a transaction, call a set of DAL methods to insert mock data, call other methods to test the results with assertions, and then roll back the entire test so that no mock data gets committed.
The problem I am having is that, as you can see, this class has been designed to create a new database connection every single time that a query is made. If I try to think like the original developer probably thought, I could see how it could make at least some sense to do this, considering the fact that these classes are used by a web API, so a persistent database connection would be impractical especially if a web API call involves transactions, because you then do need a separate connection per request to maintain separation.
However, because this is happening I don't think I can use the transaction idea to write tests as I described, because uncommitted data would not be accessible across database connections. So if I wrote a test which calls DAL methods (and also business-logic layer methods which in turn call DAL methods), each method will open its own connection to the database, and thus I have no way to wrap all of the method calls in a transaction to begin with.
I could rewrite each method to accept an SQLConnection as one of its parameters, but if I do this, I not only have to refactor over 60 methods, but I also have to rework every single place that such methods are called in the Web API controllers. I then have to move the burden of creating and managing DB connections to the Web API (and away from the DAL, which is where it philosophically should be).
Short of literally rewriting/refactoring 60+ methods and the entire Web API, is there a different approach I can take to writing unit tests for this project?
EDIT: My new idea is to simply remove all calls to con.Open(). Then, in the constructor, not just create the connection but also open it. Finally, I'll add beginTransaction, commitTransaction and rollbackTransaction methods that operate directly upon the connection object. The core API never needs to call these functions, but the unit tests can call them. This means the unit test code can simply create an instance, which will create a connection which persists across the entire lifetime of the class. Then it can use beginTransaction, then do whatever tests it wants, and finally rollbackTransaction. Having a commitTransaction is good for completeness and also exposing this functionality to the business-logic layer has potential use.
There are multiple possible answers to this question, depending on what exactly you are trying to accomplish:
Are you primarily interested in unit testing your application logic (e.g., controller methods), rather than the data access layer itself?
Are you looking to unit test the logic inside your data access layer?
Or are you trying to test everything together (i.e., integration or end-to-end testing)?
I am assuming you are interested in the first scenario, testing your application logic. In that case, I would advise against connecting to the database at all (even a development database) in your unit tests. Generally, unit tests should not be interacting with any outside system (e.g., database, filesystem, or network).
I know you mentioned you were interested in testing multiple parts of the functionality all at once:
I have studied the idea of using SQL transactions [...] so you can have access to the SQL Server functionality without mocking it out completely (e.g. you can make sure your views/functions are returning valid data AND that the API is properly processing the data all at once).
However, that rather goes against the philosophy of unit testing. The whole point of a unit test is to test a single unit in isolation. Typically, this unit ("System Under Test", or SUT, in more technical terms) is a single method inside some class (for instance, an action method in one of your controllers). Anything other than the SUT should be stubbed or mocked out.
To accomplish this, broadly speaking, you will need to refactor your code to use dependency injection, and also use a mocking framework in your tests:
Dependency Injection: If you are not using a dependency injection framework already, chances are your controller classes are instantiating your DataAccessLayer class directly. This approach will not work for unit tests - instead, you will want to refactor the controller class to accept its dependencies via the constructor, and then use a dependency injection framework to inject the real DataAccessLayer in your application code, and inject a mock/stub implementation in your tests. Some popular dependency injection frameworks include Autofac, Ninject, and Microsoft Unity. Depending on which framework you choose, this may also require that you refactor DataAccessLayer a bit so it implements an interface (e.g., IDataAccessLayer).
Mocking Framework: In your tests, rather than using the real DataAccessLayer class directly, you will instead create a mock, and set up expectations on that mock. Some popular mocking frameworks for .NET include Moq, RhinoMocks, and NSubstitute).
Granted, if the code was not initially written with unit testing in mind (i.e., no dependency injection), this may involve a fair amount of refactoring. This is where alltej's suggestion comes in with creating a wrapper for interacting with the legacy (i.e., untested) code.
I strongly recommend you read the book The Art of Unit Testing: With Examples in C# (by Roy Osherove). That will help you understand the ideology behind unit testing a bit better.
If you are actually interested in testing multiple parts of your functionality at once, then what you are describing (as others have pointed out) is integration, or end-to-end testing. The setup for this would be entirely different (and often more challenging), but even then, the recommended approach would be to connect to a separate database (specifically for integration testing, separate even from your development database), rather than rolling back transactions.
When working with legacy system, what I would do is create a wrapper for this DLLs/projects to isolate communication with legacy and to protect integrity of your new subsystem/domain or bounded context. This isolation layer is known as anticorruption layer in DDD terminology. This layer contains interfaces written in terms of your new bounded context. The interface adapts and interacts with your API layer or to other services in the domain. You can then write unit/mock tests with this interfaces. You can also create an integration tests from your anticorruption layer which will eventually call the database via the legacy dlls.
Actually, from what I see in the code, the DAL creates only one connection in the constructor and then it keeps using it to fire commands, one command per method in the DAL. It will only create new connections if you create another instance of the DAL class.
Now, what you are describing is multiple test into, integration, end to end and I am not convinced that the transaction idea, while original, is actually doable.
When writing integration tests, I prefer to actually create all the data required by the test and then simply remove it at the end, that way nothing is left behind and you know for sure if your system works or not.
So imagine you're testing retrieving account data for a user, I would create a user, activate them, attach an account and then test against that real data.
The UI does not need to go all the way through, unless you really want to do end to end tests. If you don't, then you can just mock the data for each scenario you want to test and see how the UI behaves under each scenario.
What I would suggest is that you test your api separately, test each endpoint and make sure it works as expected with integration tests covering all scenarios needed.
If you have time, then write some end to end tests, possibly using a tool like Selenium or whatever else you fancy.
I would also extract an interface from that DAL in preparation of mocking the entire layer when needed. That should give you a good start in what you want to do.
CodeIgniter comes with a Unit Testing class built in, and I would very much like to use it. However, almost all functions I would want to test interact with the database by adding records, deleting records, etc. How would I, for example, write tests for the 'create user' function without actually creating users every time I run the test?
Upon some further research, it seems I need to be using Mock objects for external services like the database, etc. I haven't been able to find much in the way of docs on how to do that besides this one forum thread:
http://codeigniter.com/forums/viewthread/106737
Is there any actual documentation?
If your database driver allows transactions, use them. Do whatever needs to be tested, then rollback (on success or failure).
I've found that it's hard to run unit tests with controller actions. If you find a good way of doing that, let us know!
I'm near the beginning of a new project and (gasp!) for the first time ever I'm trying to include unit tests in a project of mine.
I'm having trouble devising some of the unit tests themselves. I have a few methods which have been easy enough to test (pass in two values and check for an expected output). I've got other parts of the code which are doing more complex things like running queries against the database and I'm not sure how to test them.
public DataTable ExecuteQuery(SqlConnection ActiveConnection, string Query, SqlParameterCollection Parameters)
{
DataTable resultSet = new DataTable();
SqlCommand queryCommand = new SqlCommand();
try
{
queryCommand.Connection = ActiveConnection;
queryCommand.CommandText = Query;
if (Parameters != null)
{
foreach (SqlParameter param in Parameters)
{
queryCommand.Parameters.Add(param);
}
}
SqlDataAdapter queryDA = new SqlDataAdapter(queryCommand);
queryDA.Fill(resultSet);
}
catch (Exception ex)
{
//TODO: Improve error handling
Console.WriteLine(ex.Message);
}
return resultSet;
}
This method essentially takes in all the necessary bits and pieces to extract some data from the database, and returns the data in a DataTable object.
The first question is probably the most complex: What should I even test in a situation like this?
Once that's settled comes the question of whether or not to mock out the database components or try to test against the actual DB.
What are you testing?
There are three possibilities, off the top of my head:
A. You're testing the DAO (data access object) class, making sure it's correctly marshaling the values/parameters being passed to the database,, and correctly marshaling/transforming/packaging results gotten frm the database.
In this case, you don't need to connect to the database at all; you just need a unit test that replaces the database (or intermediate layer, eg., JDBC, (N)Hibernate, iBatis) with a mock.
B. You're testing the syntactic correctness of (generated) SQL.
In this case, because SQL dialects differ, you want to run the (possibly generated) SQL against the correct version of your RDBMS, rather than attempting to mock all quirks of your RDBMS (and so that any RDBMS upgrades that change functionality are caught by your tests).
C. You're testing the semantic correctness of your SQL, i.e, that for a given baseline dataset, your operations (accesses/selects and mutations/inserts and updates) produce the expected new dataset.
For that, you want to use something like dbunit (which allows you to set up a baseline and compare a result set to an expected result set), or possibly do your testing wholly in the database, using the technique I outline here: Best way to test SQL queries.
This is why (IMHO) unit tests can sometimes create a false sense of security on the part of developers. In my experience with applications that talk to a database, errors are commonly the result of data being in an unexpected state (unusual or missing values etc.). If you routinely mock up data access in your unit tests, you will think your code is working great when it is in fact still vulnerable to this kind of error.
I think your best approach is to have a test database handy, filled with gobs of crappy data, and run your database component tests against that. All the while remembering that your users will be much much better than you are at screwing up your data.
The whole point of a unit test is to test a unit (duh) in isolation. The whole point of a database call is to integrate with another unit (the database). Ergo: it doesn't make sense to unit test database calls.
You should, however, integration test database calls (and you can use the same tools you use for unit testing if you want).
For the love of God, don't test against a live, already-populated database. But you knew that.
In general you already have an idea of what sort of data each query is going to retrieve, whether you're authenticating users, looking up phonebook/org chart entries, or whatever. You know what fields you're interested in, and you know what constraints exist on them (e.g., UNIQUE, NOT NULL, and so on). You're unit testing your code that interacts with the database, not the database itself, so think in terms of how to test those functions. If it's possible for a field to be NULL, you should have a test that makes sure that your code handles NULL values correctly. If one of your fields is a string (CHAR, VARCHAR, TEXT, &c), test to be sure you're handling escaped characters correctly.
Assume that users will attempt to put anything* into the database, and generate test cases accordingly. You'll want to use mock objects for this.
* Including undesirable, malicious or invalid input.
Strictly speaking, a test that writes/reads from a database or a file system is not a unit test. (Although it may be an integration test and it may be written using NUnit or JUnit). Unit-tests are supposed to test operations of a single class, isolating its dependencies. So, when you write unit-test for the interface and business-logic layers, you shouldn't need a database at all.
OK, but how do you unit-test the database access layer? I like the advice from this book: xUnit Test Patterns (the link points to the book's "Testing w/ DB" chapter. The keys are:
use round-trip tests
don't write too many tests in your data access test fixture, because they will run much slower than your "real" unit tests
if you can avoid testing with a real database, test without a database
You can unit test everything except: queryDA.Fill(resultSet);
As soon as you execute queryDA.Fill(resultSet), you either have to mock/fake the database, or you are doing integration testing.
I for one, don't see integration testing as being bad, it's just that it'll catch a different sort of bug, has different odds of false negatives and false positives, isn't likely to be done very often because it is so slow.
If I was unit testing this code, I'd be validating that the parameters are build correctly, does the command builder create the right number of parameters? Do they all have a value? Do nulls, empty strings and DbNull get handled correctly?
Actually filling the dataset is testing your database, which is a flaky component out of the scope of your DAL.
For unit tests I usually mock or fake the database. Then use your mock or fake implementation via dependency injection to test your method. You'd also probably have some integration tests that will test constraints, foreign key relationships, etc. in your database.
As to what you would test, you'd make sure that the method is using the connection from the parameters, that the query string is assigned to the command, and that your result set returned is the same as that you are providing via an expectation on the Fill method. Note -- it's probably easier to test a Get method that returns a value than a Fill method the modifies a parameter.
The first question is probably the most complex: What should I even test in a situation like this?
Since your code code is basically a DAO/repository without any
business logic you need an integration test, not a unit test.
Unit test should test classes without external dependencies (like DB
or calls to other remote services).
You should always try to separate the business logic (your Domain
Model) code from infrastructure code then it will be easy to use unit
tests.
Be careful with Mocks, it can be a signal of bad design. It means
you business logic is mixed with infrastructure.
Check these patterns: "Domain Model", "Hexagonal Architecture", "Functional Core, Imperative Shell"
In order to do this properly though you would should use some dependency injection (DI), and for .NET there are several. I am currently using the Unity Framework but there are others that are easier.
Here is one link from this site on this subject, but there are others:
Dependency Injection in .NET with examples?
This would enable you to more easily mock out other parts of your application, by just having a mock class implement the interface, so you can control how it will respond. But, this also means designing to an interface.
Since you asked about best practices this would be one, IMO.
Then, not going to the db unless you need to, as suggested is another.
If you need to test certain behaviors, such as foreign key relationships with cascade delete then you may want to write database tests for that, but generally not going to a real database is best, esp since more than one person may run a unit test at a time and if they are going to the same database tests may fail as the expected data may change.
Edit: By database unit test I mean this, as it is designed to just use t-sql to do some setup, test and teardown.
http://msdn.microsoft.com/en-us/library/aa833233%28VS.80%29.aspx
On JDBC based project, JDBC connection can be mocked, so that tests can be executed without live RDBMS, with each test case isolated (no data conflict).
It allow to verify, persistence code passes proper queries/parameters (e.g. https://github.com/playframework/playframework/blob/master/framework/src/anorm/src/test/scala/anorm/ParameterSpec.scala) and handle JDBC results (parsing/mapping) as expected ("takes in all the necessary bits and pieces to extract some data from the database, and returns the data in a DataTable object").
Framework like jOOQ or my framework Acolyte can be used for: https://github.com/cchantep/acolyte .
Perhaps, a good approach would be to test the behaviour of your domain logic that communicates with the DB under the hood.
Without diving into DDD and CQRS, you could check out DbSample on GitHub, a sample EF Core based project with fully automated tests against services that work with MS SQL Server. It also has a GitHub Actions pipeline to run the tests in cloud builds.
An example of the test would be
[Fact]
public async Task Update_Client_Works()
{
// GIVEN a DB with a client
var existingClient = await DataContext.AddAsync(new Client { Name = "Name" });
await DataContext.SaveChangesAsync();
var clientId = existingClient.Id;
// WHEN update name of the client (this domain command executes an SQL query)
await _clientCommandService.Update(clientId, new CreateUpdateClientRequest("XYZ"));
// THEN the name is updated
var client = await DataContext.Clients.FindAsync(clientId);
Assert.Equal("XYZ", client!.Name);
}
For a deeper dive into orchestrating tests, see "Pain & Gain of automated tests against SQL (MS SQL, PostgreSQL)" article. It goes into the woods of "how?" and "why?". Spoiler alert – it relies on Docker a lot.
Does anyone have some good hints for writing test code for database-backend development where there is a heavy dependency on state?
Specifically, I want to write tests for code that retrieve records from the database, but the answers will depend on the data in the database (which may change over time).
Do people usually make a separate development system with a 'frozen' database so that any given function should always return the exact same result set?
I am quite sure this is not a new issue, so I would be very interested to learn from other people's experience.
Are there good articles out there that discuss this issue of web-based development in general?
I usually write PHP code, but I would expect all of these issues are largely language and framework agnostic.
You should look into DBUnit, or try to find a PHP equivalent (there must be one out there). You can use it to prepare the database with a specific set of data which represents your test data, and thus each test will no longer depend on the database and some existing state. This way, each test is self contained and will not break during further database usage.
Update: A quick google search showed a DB unit extension for PHPUnit.
If you're mostly concerned with data layer testing, you might want to check out this book: xUnit Test Patterns: Refactoring Test Code. I was always unsure about it myself, but this book does a great job to help enumerate the concerns like performance, reproducibility, etc.
I guess it depends what database you're using, but Red Gate (www.red-gate.com) make a tool called SQL Data Generator. This can be configured to fill your database with sensible looking test data. You can also tell it to always use the same seed in its random number generator so your 'random' data is the same every time.
You can then write your unit tests to make use of this reliable, repeatable data.
As for testing the web side of things, I'm currently looking into Selenium (selenium.openqa.org). This appears to be a cross-browser capable test suite which will help you test functionality. However, as with all of these web site test tools, there's no real way to test how well these things look in all of the browsers without casting a human eye over them!
We use an in-memory database (hsql : http://hsqldb.org/). Hibernate (http://www.hibernate.org/) makes it easy for us to point our unit tests at the testing db, with the added bonus that they run as quick as lightning..
I have the exact same problem with my work and I find that the best idea is to have a PHP script to re-create the database and then a separate script where I throw crazy data at it to see if it breaks it.
I have not ever used any Unit testing or suchlike so cannot say if it works or not sorry.
If you can setup the database with a known quantity prior to running the tests and tear down at the end, then you'll know what data you are working with.
Then you can use something like Selenium to easily test from your UI (assuming web-based here, but there are a lot of UI testing tools out there for other UI-flavours) and detect the presence of certain records pulled back from the database.
It's definitely worth setting up either a test version of the database - or make your test scripts populate the database with known data as part of the tests.
You could try http://selenium.openqa.org/ it is more for GUI testing rather than a data layer testing application but does record your actions which then can be played back to automate tests across different platforms.
Here's my strategy (I use JUnit, but I'm sure there's a way to do the equivalent in PHP):
I have a method that runs before all of the Unit Tests for a specific DAO class. It puts the dev database into a known state (adds all test data, etc.). As I run tests, I keep track of any data added to the known state. This data is cleaned up at the end of each test. After all the tests for the class have run, another method removes all the test data in the dev database, leaving it in the state it was in before the tests were run. It's a bit of work to do all this, but I usually write the methods in a DBTestCommon class where all of my DAO test classes can get to them.
I would propose to use three databases. One production database, one development database (filled with some meaningful data for each developer) and one testing database (with empty tables and maybe a few rows that are always needed).
A way to test database code is:
Insert a few rows (using SQL) to initialize state
Run the function that you want to test
Compare expected with actual results. Here you could use your normal unit testing framework
Clean up the rows that were changed (so the next run won't see the previous run)
The cleanup could be done in a standard way (of course, only in the testing database) with DELETE * FROM table.
In general I agree with Peter but for creating and deleting of test data I wouldn't use SQL directly. I prefer to use some CRUD API that is used in product to create data as similar to production as possible...