I am considering to write smoke tests for our WPF application. The question that I am faced is: should we use UI automation( or some other technology that creates a UI script), or is it good enough to use ViewModels directly (after all the viewmodels were created to make unit testing easier in a first place).
There's no reason why you can't write unit tests for your ViewModels if they're properly separated from your Views.
A smoke test is a test that actually fires up your application and checks that it works and (for an application with a UI) UI automation is the way to go for that.
We used to write our integration tests (smoke tests if you like) using the ViewModels directly. It worked, but we did have to deal with some interesting threading issues: what happens, for example if your ViewModel causes a Message Box to be shown - how does your test close the Message box? We had to make sure that our application was running on one thread and our tests on another.
We've now moved over to UIAutomation and those kind of problems go away because your tests and the application are explicitly running in two separate processes. There is a bit of a learning curve involved, but UIAutomation isn't as scary as it first appears: I've written a tutorial that might help you get started - follow the link, and you'll also find a few helper methods that tame the UI Automation API somewhat.
Related
I am currently developing for Silverlight 4.0 and after mostly creating class libraries with TDD in usual C# (before SL) I can say that my current process is way slower than I am used to. (I think this can be said about any UI code compared to library classes, but here I think its really serious issue for me.)
I am wondering what techniques can be recommended to increase SL development performance.
I am mainly concerned about hard to test code (from my POV) - MVVM & UI - what can be done to improve performance here, I am thinking maybe theres a way to use a smaller sandbox somehow and test/debug control behaviour outside of scope of whole application, its pretty clear to me that me running whole application to test whether a new dialog box works correctly isnt fastest way and I could improve performance if I had a way to test this dialog alone for example, and there are probably other ways I cannot think of that can be a solution too.
EDIT: 1)here is something that I found useful , for TDD there is now a project that allows console runner to run tests so you dont have to run silverlight tests in browser & can integrate in your build process LightHouse
2) found following page, it provides some idea about a possible approach one could use to test view:
http://fohjin.blogspot.com/2008/09/how-to-test-your-xaml-behavior-using.html
there is no magic beautiful way and this one can be utilized but having to name all controls for example is a must to get this to work which isnt very good often
Statlight for the build server.
AgUnit to allow resharper to run silverlight tests.
WebAii for automation testing.
I'm not a fan of SLUT, as to run an individual test you have to cut and paste its name, and it doesnt remember it until you let it run all the way through, which I rarely do if I'm debugging.
Have you tried to use slut?
http://archive.msdn.microsoft.com/silverlightut
she will do what you want and pretend she enjoys it
I am developing quite complex .NET custom control (40K lines of code) but have some trouble testing it.
I have done several sample projects demonstrating main features of the control, but one can test only small subset of control states and operations.
Unit tests are also useless, because of these problems:
tremendous number of use cases (e.g. describing "item selections" can take some 4 pages of specs)
many ways of doing the same thing (and also from user-code or GUI)
the control have many states a sub-states, and anything may not possibly work in every state
how to test design-time support?
I know this is a common problem of GUI testing, so I would like to ask you if there are any well-estabilished practices of testing custom visual components?
Thanks for any advice.
The main thing is to limit the GUI testing to a minimum, because it's the most expensive way to test. In 40K lines of code I bet 90% of code doesn't really work with the WinForm GUI elements at all. So most of it could be covered by unit tests, but this depends on the way you structured your code.
Things to consider:
SOLID principles, especially the Single responsibility principle (SRP) - instead of a couple of huge "God" classes, you should rely on a large number of small "service" interfaces & classes which only do one thing and do it well. You can write isolated tests for these and then assemble them into a larger picture once you know they work OK.
MVP pattern (Passive View actually): the WinForm GUI code should be only a thin layer (View), everything else should be in Presenters and the Model.
Presenter First approach.
I work on a relatively complex WinForms application and these patterns have never failed me.
As for GUI testing, I use UI Automation, but only for some standard cases. Everything else is covered by non-GUI unit testing.
I'm currently using the Silverlight Unit Test Framework, but I'd prefer to run tests directly in VS2010. I'm curious to know what approaches and tools everyone else uses.
I'm using Silverlight 4 with Prism and the MVVM pattern, and I'm specifically interested in integrated Silverlight unit test support in VS 2010 that I can use with my ViewModel unit tests. I'm using dependency injection with Unity, and I am writing unit tests by mocking the calls in my WCF layer using Moq for Silverlight. I am not even looking at integration tests at the moment, but even in a simple unit test which tests a single ViewModel command, the service request to my mocked service layer can take around 50 milliseconds. Therefore support for asynchronous tests is important to me.
The issue I'm raising here is not related to View testing, which I have handled with some success in the past using System.Windows.Automation.Peers, and - although I have not used it yet - could possibly now handle more easily with the support in VS 2010 Feature Pack 2 (which appears to be targeted at automation / playback of UI tests from what I gather).
I should mention that my findings from the products I've looked at and used so far are as follows:
Silverlight Unit Test Framework - I currently use this, and it's great as far as it goes, but its limitations are (a) it is not integrated with Visual Studio; and (b) if you don't want to run all tests, you are limited to the crude tag expression filter.
StatLight - very nice. I currently use this, and have used it since v0.9 when targeting Silverlight 3 on a previous project. Being a command-line tool, it can be integrated with a continuous integration server - which certainly handles another required scenario. But it is of no use directly in terms of Visual Studio integration during the development process.
Unit Test Result Viewer for Silverlight (Visual Studio extension on the Visual Studio gallery) - looks promising, but its limitations are (a) currently fails to find projects which are located in solution folders, rather than under the solution root; and (b) runs all tests in a given assembly (via StatLight), with no apparent ability to run a specific test, or a selection of tests.
Einar Ingebrigtsen's Silverlight Unit Test Runner for ReSharper, which later became Odin - ahead of the game (it first appeared in 2008), but the limitation is that it appears that this project is no longer maintained (most recent update is Apr 2009).
AgUnit ReSharper plugin ( http://agunit.codeplex.com/ ) - looks excellent initially. After downloading the source code for it and building the latest (bug-fixed) version to work against ReSharper 5.1, I was very encouraged. But unfortunately it does not handle asynchronous tests. This is a design limitation with the threading, so it does not matter whether you try to use the asynchronous support that is built into Silverlight Unit Test Framework (Microsoft.Silverliht.Testing.SilverlightTest base class), or whether you're using AutoResetEvent or anything else. This has been noted by the coordinator on the project's discussion forum on CodePlex. This is a massive limitation.
TestDriven.NET 3.0 - appears to have support for Silverlight 4.0 tests at first glance, but the limitation (I suspect) is Silverlight 4 "assembly portability" (i.e. the 5 dependent assemblies that are portable between SL4 and .NET 4). Certainly, when I tried using it with a simple POC, it crashed my instance of VS 2010.
Perhaps I've missed something here - I wonder if anyone in the community has any better ideas for Silverlight unit testing?
I use the Silverlight Unit test framework, AgUnit, and RX with a mock IScheduler provider to make my unit tests single threaded :)
UPDATE:
I'd already settled on using StatLight for my continuous integration server, but I was looking for a solution to allow me to run asynchronous Silverlight unit tests directly in VS2010 during development.
Inspired by Rob Fonseca-Ensor's suggestion to use Rx (see his separate answer on this page), I took another look at the issue. That led me to find a non-Rx solution to the problem of using AgUnit to run async Silverlight unit tests.
The solution I think I will use - at least for now - is the combination of:
Silverlight Unit Test Framework
ReSharper 5.1
AgUnit ReSharper plugin
[Mock async pattern]
My ViewModels have bespoke service classes injected into them which provide abstractions of my (auto-generated) WCF service reference classes. To help provide access to the WCF service methods, my bespoke service classes also rely on another common class which wraps the async pattern. For my unit tests, I was already mocking my WCF service reference classes and my bespoke service classes with Moq - but I hadn't looked at mocking my async pattern wrapper class.
So I decided to mock my async pattern wrapper class as well. The attraction of doing this was that I thought I might then be able to use Rx with a mock IScheduler (as Rob suggested in his answer) in all my mocks, while leaving my real classes free of any references to Rx (which is a requirement because for this project at the place I'm working, I need to keep Rx out of any code which gets deployed to a production environment). However, once I'd mocked the wrapper class, I realised that I didn't even need Rx, and that there was an even simpler solution - which I should have really looked at before. It's pretty trivial - all I really needed to do was mock the wrapper class and ensure that it called Invoke on the callback operations, rather than calling BeginInvoke. This ultimately prevents the callbacks which work fine when running under the Silverlight Unit Test Framework in a browser session from going into a black hole when the unit tests are run by AgUnit within VS2010. (Allowing the callbacks to go into black hole when using AgUnit would of course have prevented the individual tests from completing properly, and could have led to either timeouts or - worse still - false positives in the test results.)
In the absence of better advice from any subsequent answers to this question, this appears to be the easiest way for me to handle this scenario - allowing my async unit tests to run in AgUnit without requiring Rx in my production code.
For anyone else not facing company-imposed restrictions on deploying code that uses Rx (which is still a DevLabs project) to production, I think Rob's solution would definitely be worth looking at.
I'm the author of the AgUnit plugin.
AgUnit blocks asynchronous unit testing to speed up the test run. Under the covers it uses the same code as the Silverlight Unit Test Framework test runner, but this runner is very slow if you have to handle a large number of tests. I've seen differences between almost half an hour and a few minutes for a couple thousand tests.
That said, if you do want asynchronous testing, it's a very isolated part of AgUnit that does this. I'll try to create a build with it disabled. In the future this will be a configuration option or an attribute, I've not decided yet.
Feel free to contact me with any questions or requests.
I usually create a normal unit test project in VS.NET, add my Silverlight assembly in references and write unit test classes.
So everything works out of the box. What's a problem with this solution?
Have you read about the recently released Visual Studio 2010 Feature Pack 2. Currently it only accessible to MSDN subscribers, and it only works with certain editions of VS2010 (Premium, Ultimate and Test Pro).
Here is a quick overview: Link
Hi I've searched some other posts, but most of them assumed that people knew what they were doing in their unit testing, and frankly I don't. I see the idea behind unit testing, and I'm coding an silverlight application much in the blind right now, and I'd like to write some unit tests to kind of be sure I'm on the right path. I'd like to be able to use the SL4 vs 2010 silverlight unit test project template, to keep it simple and not use external tools. So what I need an answer for are questions like:
what are the methods of unit testing?
what are the differences between unit tests, and automated unit tests?
How do I meaningfully unit test in silverlight?
What should I be aware of while unit testing (in silverlight) ?
Also should I implement some kind of IRepository pattern in my silverlight app to make unit testing easier?
EDIT:
I will be posting useful links here as I explore this along the way:
Implementing MVVM in silverlight - http://community.infragistics.com/pixel8/media/p/91949.aspx
Mix10 MVVM talk - http://live.visitmix.com/MIX10/Sessions/EX14
Unit testing Ria apps - Link
PK's great resource: http://dotenetscribbles.blogspot.com/2009/12/unit-testing-dependency-injection-and.html
I have never used silverlight unit test project template. I have only used nunit to do unit testing in silverlight. So, I will leave that for someone else to answer.
As far as your other questions are concerned:
Unit testing should drive your design. Your unit tests are first users of your code. Your code is based on some expectation and unit tests verify that expectations are being met. Using MVVM pattern (as you are using silverlight), facilitates unit testing. The most important thing to remember is that you have to write testable code. And to write testable code, the most important thing to remember is to inject dependencies. For example, if your code makes a call to the db. You, can't have a unit test making a call to the DB. Instead, you will mock your data acccess layer. This is where concepts like mocks and stubs come into picture. I use moq for mocking in my sivlerlight unit tests. Another, important thing I follow which I think facilitates unit testing is single responsibility principle. Finally, treat your test code as production code, else your tests might give you a false notion, that expectations are being met. Your unit tests are code and hence can have bugs.
what are the differences between unit
tests, and automated unit tests?
I am not very sure what do you mean by this. Unit tests are an automated way of white box testing. You can have scripts which run all unit tests, every time you checkin any code in the repository. This can be part of continuous integration.
How do I meaningfully unit test in
silverlight
In silverlight, to facilitate testing you should use commands, instead of writing code in code-behind files. This allows you to simulate button click and other GUI events while unit testing. Using MVVM pattern along with commands, you can test all of C# code (not xaml), right up to UI (Converter, VMs, etc).
It is very difficult to mention everything in this one answer. I would suggest, you google for MVVM, commands in Silverlight, Martin fowler - mock are not stubs, mocking frameworks for silverlight, dependency injection
I'm starting a new WPF/Silverlight custom control project and wanted to do unit testing on this one. However I'm a little confused about how to approach this.
This control would be based on the same codebase for both WPF and Silverlight with minor forking using #ifs and partial classes to tame the differences. I guess I could write unit tests for WPF part with NUnit, MSTest, xUnit, etc. and for the Silverlight part with Silverlight Unit Test Framework but this doesn't sound very elegant to me. I'd have to either ignore testing identical code on one of the platforms and test only differing parts (which is not very trustworthy) or rewrite tests for 2 frameworks (which is annoying). Is this the right way to go?
I'm wondering if there's some guidance, articles, tutorials out there on how to approach this task. Any pointers?
I am hardly an expert in WPF and Silverlight, but wouldn't it be possible to write the tests using the same techniques as the production code (#ifs and partial classes as you said)?
I tried to use xUnit first but it was kind of complicated to make same tests work in xUnit and SLUT (different attributes, syntax, etc.)
Then I did some basic experimenting with MSTest and from very simple test it looks like you can successfully use MSTest for the WPF part and same code with some #ifs, etc. and SLUT for the Silverlight part. So I'll try to go this route and see how it works in real-world situations.