I'm currently using the Silverlight Unit Test Framework, but I'd prefer to run tests directly in VS2010. I'm curious to know what approaches and tools everyone else uses.
I'm using Silverlight 4 with Prism and the MVVM pattern, and I'm specifically interested in integrated Silverlight unit test support in VS 2010 that I can use with my ViewModel unit tests. I'm using dependency injection with Unity, and I am writing unit tests by mocking the calls in my WCF layer using Moq for Silverlight. I am not even looking at integration tests at the moment, but even in a simple unit test which tests a single ViewModel command, the service request to my mocked service layer can take around 50 milliseconds. Therefore support for asynchronous tests is important to me.
The issue I'm raising here is not related to View testing, which I have handled with some success in the past using System.Windows.Automation.Peers, and - although I have not used it yet - could possibly now handle more easily with the support in VS 2010 Feature Pack 2 (which appears to be targeted at automation / playback of UI tests from what I gather).
I should mention that my findings from the products I've looked at and used so far are as follows:
Silverlight Unit Test Framework - I currently use this, and it's great as far as it goes, but its limitations are (a) it is not integrated with Visual Studio; and (b) if you don't want to run all tests, you are limited to the crude tag expression filter.
StatLight - very nice. I currently use this, and have used it since v0.9 when targeting Silverlight 3 on a previous project. Being a command-line tool, it can be integrated with a continuous integration server - which certainly handles another required scenario. But it is of no use directly in terms of Visual Studio integration during the development process.
Unit Test Result Viewer for Silverlight (Visual Studio extension on the Visual Studio gallery) - looks promising, but its limitations are (a) currently fails to find projects which are located in solution folders, rather than under the solution root; and (b) runs all tests in a given assembly (via StatLight), with no apparent ability to run a specific test, or a selection of tests.
Einar Ingebrigtsen's Silverlight Unit Test Runner for ReSharper, which later became Odin - ahead of the game (it first appeared in 2008), but the limitation is that it appears that this project is no longer maintained (most recent update is Apr 2009).
AgUnit ReSharper plugin ( http://agunit.codeplex.com/ ) - looks excellent initially. After downloading the source code for it and building the latest (bug-fixed) version to work against ReSharper 5.1, I was very encouraged. But unfortunately it does not handle asynchronous tests. This is a design limitation with the threading, so it does not matter whether you try to use the asynchronous support that is built into Silverlight Unit Test Framework (Microsoft.Silverliht.Testing.SilverlightTest base class), or whether you're using AutoResetEvent or anything else. This has been noted by the coordinator on the project's discussion forum on CodePlex. This is a massive limitation.
TestDriven.NET 3.0 - appears to have support for Silverlight 4.0 tests at first glance, but the limitation (I suspect) is Silverlight 4 "assembly portability" (i.e. the 5 dependent assemblies that are portable between SL4 and .NET 4). Certainly, when I tried using it with a simple POC, it crashed my instance of VS 2010.
Perhaps I've missed something here - I wonder if anyone in the community has any better ideas for Silverlight unit testing?
I use the Silverlight Unit test framework, AgUnit, and RX with a mock IScheduler provider to make my unit tests single threaded :)
UPDATE:
I'd already settled on using StatLight for my continuous integration server, but I was looking for a solution to allow me to run asynchronous Silverlight unit tests directly in VS2010 during development.
Inspired by Rob Fonseca-Ensor's suggestion to use Rx (see his separate answer on this page), I took another look at the issue. That led me to find a non-Rx solution to the problem of using AgUnit to run async Silverlight unit tests.
The solution I think I will use - at least for now - is the combination of:
Silverlight Unit Test Framework
ReSharper 5.1
AgUnit ReSharper plugin
[Mock async pattern]
My ViewModels have bespoke service classes injected into them which provide abstractions of my (auto-generated) WCF service reference classes. To help provide access to the WCF service methods, my bespoke service classes also rely on another common class which wraps the async pattern. For my unit tests, I was already mocking my WCF service reference classes and my bespoke service classes with Moq - but I hadn't looked at mocking my async pattern wrapper class.
So I decided to mock my async pattern wrapper class as well. The attraction of doing this was that I thought I might then be able to use Rx with a mock IScheduler (as Rob suggested in his answer) in all my mocks, while leaving my real classes free of any references to Rx (which is a requirement because for this project at the place I'm working, I need to keep Rx out of any code which gets deployed to a production environment). However, once I'd mocked the wrapper class, I realised that I didn't even need Rx, and that there was an even simpler solution - which I should have really looked at before. It's pretty trivial - all I really needed to do was mock the wrapper class and ensure that it called Invoke on the callback operations, rather than calling BeginInvoke. This ultimately prevents the callbacks which work fine when running under the Silverlight Unit Test Framework in a browser session from going into a black hole when the unit tests are run by AgUnit within VS2010. (Allowing the callbacks to go into black hole when using AgUnit would of course have prevented the individual tests from completing properly, and could have led to either timeouts or - worse still - false positives in the test results.)
In the absence of better advice from any subsequent answers to this question, this appears to be the easiest way for me to handle this scenario - allowing my async unit tests to run in AgUnit without requiring Rx in my production code.
For anyone else not facing company-imposed restrictions on deploying code that uses Rx (which is still a DevLabs project) to production, I think Rob's solution would definitely be worth looking at.
I'm the author of the AgUnit plugin.
AgUnit blocks asynchronous unit testing to speed up the test run. Under the covers it uses the same code as the Silverlight Unit Test Framework test runner, but this runner is very slow if you have to handle a large number of tests. I've seen differences between almost half an hour and a few minutes for a couple thousand tests.
That said, if you do want asynchronous testing, it's a very isolated part of AgUnit that does this. I'll try to create a build with it disabled. In the future this will be a configuration option or an attribute, I've not decided yet.
Feel free to contact me with any questions or requests.
I usually create a normal unit test project in VS.NET, add my Silverlight assembly in references and write unit test classes.
So everything works out of the box. What's a problem with this solution?
Have you read about the recently released Visual Studio 2010 Feature Pack 2. Currently it only accessible to MSDN subscribers, and it only works with certain editions of VS2010 (Premium, Ultimate and Test Pro).
Here is a quick overview: Link
Related
I was a .net based windows application developer and being a web developer now.
For the last couple of years, I tried to follow principles of Specification by example in a project.
On the last project I was involved, our team used WPF with MVVM(Model-View-ViewModel) pattern and I tried SpecFlow for testing. (Well.. the coverage by SpecFlow was very low though cause it was not officially adopted from our team and I couldn't spend enough time on it.)
Now our team started to migrate the existing windows application to a web based application and decided to use cucumber.js as a tool for implementing 'Specification by example'.
When I was trying SpecFlow before, I made all the step definitions to test ViewModel layer instead of View and it was the only way actually. What I'm confused with cucumber.js is that most of examples demonstrate how to test features via web browsers(headless or real ones).
If the features are specifying specific ui behaviors then accessing a DOM element and evaluating its value make sense. However, as far as I learned about Specification by examples, testing under the UI layer is recommended instead of UI elements directly.
It seems like most of modern web applications have layers such as MVC, MVVM(knouckout.js) nowadays. Our web application will use Angularjs and this is my theory.
I guess it is possible to use feature files - which do not state any specific UI controls and behaviors - to test under the skin of UI layer, without relying browser support. That layer would be Model of angularjs, not ui elements.
Is this approach missing something or wrong with cucumber.js?
Please give me some insights and correct me if I'm wrong.
Thanks in advance!
Specification by example specifies requirements by giving examples from the point of view of the user. What's more representative of the user: clicking items on a web page, or directly calling code? A user won't be calling methods on your view model, so in my opinion a feature test is a much better way to test expected behaviour.
I need to test an angular application and I'm thinking which testing framework to use.
Can someone please list the differences between those 2 testing framework, what is good / bad in each. when should each of them should be used? can the one replace the second?
Edit 1:
I'll try to focus my question. what I'm really interested in is the technical abilities of the 2 frameworks.
for example, some of the differences I have found:
selenium webdriver has the ability to run on grid in parallel on multiple machines, on the other side, angular scenarios runs much faster than selenium.
with selenium I can use keyboard keys such as Enter, Backspace and more and also do drag and drop actions.
selenium is not dependent on the app technology, angular is for angular apps only
Those are the type of differences I'm looking for - what can be done with one and can't be done with the other
thanks
Both should be used for testing an angular application. However, in my experience, they serve different purposes.
Angular e2e tests are your best friends when integrating new features into your application, or changing already existing features. They are there to make sure your application still behaves as expected if you are making changes to your code. These tests are much faster than selenium, do not need to be as in-depth, and, in my experience, are best executed before you push a new feature to your version control server.
Selenium tests should be used for regression testing. These tests should be far more in-depth than Angular e2e tests, and should be performed before pushing code into production.
UPDATE: This question may be less relevant now. As of January 2014, Angular plans to move from their own E2E suite to the protractor library for function tests, which use Selenium Webdriver.
If you're starting a new Angular project, you may want to look into using Protractor, as it is going to replace the current method of E2E Testing in the near future.
(Source: http://docs.angularjs.org/guide/dev_guide.e2e-testing)
I am struggling with this question as well.
Here is my thinking today:
1) Use Angular e2e testing for integration/regression testing the UI with mocks of your external dependencies (like the app server).
2) Use Selenium for integration/regression testing the complete system in a test/qa environment.
It seems Angular e2e testing helps fill in the gaps that unit testing leaves in testing view and controller logic that is tied to UI events. All of the Angular tests should mock external dependencies.
Selenium seems like the best fit for testing the entire system as the user will experience it.
This is such a subjective question, and therefore I'd first tell you to do some research. There is not much difference.
It is a very new project, whereas Selenium is a lot more mature. For instance, the JSON Wire Protocol that WebDriver is built on top of, is an actual W3C spec. This shows the project is very mature and very stable.
Due to the above, Selenium has a much much wider set of documentation, and a much wider community.
With Selenium you have a variety of different languages to use to code the tests in, as well as use the IDE. Angular limits use to pure JavaScript. For a lot of manual testers, this is a skill most do not have.
Both Selenium and Angular are open source, which (believe me) helps tremendously in debugging problems and understanding design decisions in their respective API's.
I am currently developing for Silverlight 4.0 and after mostly creating class libraries with TDD in usual C# (before SL) I can say that my current process is way slower than I am used to. (I think this can be said about any UI code compared to library classes, but here I think its really serious issue for me.)
I am wondering what techniques can be recommended to increase SL development performance.
I am mainly concerned about hard to test code (from my POV) - MVVM & UI - what can be done to improve performance here, I am thinking maybe theres a way to use a smaller sandbox somehow and test/debug control behaviour outside of scope of whole application, its pretty clear to me that me running whole application to test whether a new dialog box works correctly isnt fastest way and I could improve performance if I had a way to test this dialog alone for example, and there are probably other ways I cannot think of that can be a solution too.
EDIT: 1)here is something that I found useful , for TDD there is now a project that allows console runner to run tests so you dont have to run silverlight tests in browser & can integrate in your build process LightHouse
2) found following page, it provides some idea about a possible approach one could use to test view:
http://fohjin.blogspot.com/2008/09/how-to-test-your-xaml-behavior-using.html
there is no magic beautiful way and this one can be utilized but having to name all controls for example is a must to get this to work which isnt very good often
Statlight for the build server.
AgUnit to allow resharper to run silverlight tests.
WebAii for automation testing.
I'm not a fan of SLUT, as to run an individual test you have to cut and paste its name, and it doesnt remember it until you let it run all the way through, which I rarely do if I'm debugging.
Have you tried to use slut?
http://archive.msdn.microsoft.com/silverlightut
she will do what you want and pretend she enjoys it
Hi I've searched some other posts, but most of them assumed that people knew what they were doing in their unit testing, and frankly I don't. I see the idea behind unit testing, and I'm coding an silverlight application much in the blind right now, and I'd like to write some unit tests to kind of be sure I'm on the right path. I'd like to be able to use the SL4 vs 2010 silverlight unit test project template, to keep it simple and not use external tools. So what I need an answer for are questions like:
what are the methods of unit testing?
what are the differences between unit tests, and automated unit tests?
How do I meaningfully unit test in silverlight?
What should I be aware of while unit testing (in silverlight) ?
Also should I implement some kind of IRepository pattern in my silverlight app to make unit testing easier?
EDIT:
I will be posting useful links here as I explore this along the way:
Implementing MVVM in silverlight - http://community.infragistics.com/pixel8/media/p/91949.aspx
Mix10 MVVM talk - http://live.visitmix.com/MIX10/Sessions/EX14
Unit testing Ria apps - Link
PK's great resource: http://dotenetscribbles.blogspot.com/2009/12/unit-testing-dependency-injection-and.html
I have never used silverlight unit test project template. I have only used nunit to do unit testing in silverlight. So, I will leave that for someone else to answer.
As far as your other questions are concerned:
Unit testing should drive your design. Your unit tests are first users of your code. Your code is based on some expectation and unit tests verify that expectations are being met. Using MVVM pattern (as you are using silverlight), facilitates unit testing. The most important thing to remember is that you have to write testable code. And to write testable code, the most important thing to remember is to inject dependencies. For example, if your code makes a call to the db. You, can't have a unit test making a call to the DB. Instead, you will mock your data acccess layer. This is where concepts like mocks and stubs come into picture. I use moq for mocking in my sivlerlight unit tests. Another, important thing I follow which I think facilitates unit testing is single responsibility principle. Finally, treat your test code as production code, else your tests might give you a false notion, that expectations are being met. Your unit tests are code and hence can have bugs.
what are the differences between unit
tests, and automated unit tests?
I am not very sure what do you mean by this. Unit tests are an automated way of white box testing. You can have scripts which run all unit tests, every time you checkin any code in the repository. This can be part of continuous integration.
How do I meaningfully unit test in
silverlight
In silverlight, to facilitate testing you should use commands, instead of writing code in code-behind files. This allows you to simulate button click and other GUI events while unit testing. Using MVVM pattern along with commands, you can test all of C# code (not xaml), right up to UI (Converter, VMs, etc).
It is very difficult to mention everything in this one answer. I would suggest, you google for MVVM, commands in Silverlight, Martin fowler - mock are not stubs, mocking frameworks for silverlight, dependency injection
I download command behavior from web and have implemented in my silverlight project. Now I am trying to figure out how to unit test ICommand properties. I know lots of people are working on this, so if you have a good simple example of unit testing ICommand, please let me know.
Thanks
Dev
You downloaded an ICommand implementation from the web? And you want to test it?
I recommend downloading prism and taking a look at their implementation of ICommand. They also have plenty of unit tests in the Reference Implementation (RI).
However, usually we don't test the ICommand implementations very heavily. They either work or they don't. The idea is that Commanding allows us to have thin UI layers and then testable ViewModels/Presenters. The View Models and Presenters we do test heavily. It's like in ASP.Net MVC, we do MVC so we don't have to test the views*.
*- To be sure, views can still be tested, but I think most people are moving away from the button pushing/angle bracket testing of MVC apps.
There is an open source framework called Silverunit CThru, that sits on top of Typemock, that can help you we when unit testing Silverlight. It's a real unit testing framework (the original is more of an integration test system). Cthru contains components that let you test Silverlight controls like real unit tests.