I am facing intermittent protractor test failures across a range of test suites without any real pattern in the fail cases to indicate what could be going on, for example it's not the same tests that are failing. Sometimes I will get many failures and on other occasions just a single fail case.
I should point out that this tends to only happen when performing test runs on a Jenkins CI server we have configured (running under linux). Locally on Windows dev machines we may get a single fail case after 30-40 runs which I can live with!
The application we are testing is currently built with angular 1.5 and we are using angular material 1.1.3
Due to the animations used in angular material and the performance hit these can have, we have already tried disabling animations following this approach here which certainly make the tests quicker but dont help with the fail cases we are seeing/
I am at a point now where I am running one test suite over and over, after 5 successful runs it then failed on it's 6th attempt on our Jenkins CI environment\linux box, locally I have run this test many times now and no failures yet.
The test suite in question is detailed below along with a page object file snippet:
//test suite
describe('Operators View', function () {
var operatorPage = require('./operators.po.js'),
loginView = require('../login/login.po.js'),
page = new operatorPage();
describe('Large screen tests', function () {
beforeAll(function () {
loginView.login();
});
afterAll(function () {
loginView.logout();
});
it('should create an operator', function () {
page.settlementBtn.click();
page.operatorsBtn.click();
page.fabBtn.click();
page.createOperator();
expect(page.headline.getText()).toEqual('Operators');
});
});
});
// operators.po.js
var operatorsSection = function() {
this.helper = new Helpers();
this.headline = element(by.css('.md-headline'));
this.settlementBtn = element(by.css('[ui-sref="settlement"]'));
this.operatorsBtn = element(by.css('[ui-sref="operators"]'));
this.fabBtn = element(by.css('.md-fab'));
// Form Elements
this.licenceNumber = element(by.model('vm.transportOperator.licenceNumber'));
this.tradingName = element(by.model('vm.tradingName'));
this.name = element(by.model('vm.name'));
this.operatorAddressFirstLine = element(by.model('vm.transportOperator.address.line1'));
this.operatorAddressCityTown = element(by.model('vm.transportOperator.address.line5'));
this.operatorAddressPostCode = element(by.model('vm.transportOperator.address.postcode'));
this.payeeAddressFirstLine = element(by.model('vm.transportOperator.payee.address.line1'));
this.payeeAddressCityTown = element(by.model('vm.transportOperator.payee.address.line4'));
this.payeeAddressPostCode = element(by.model('vm.transportOperator.payee.address.postcode'));
this.opID = element(by.model('vm.transportOperator.fields.opID'));
this.spID = element(by.model('vm.transportOperator.fields.spID'));
this.schemeSelect = element(by.model('reference.scheme'));
this.schemeOptions = element(by.exactRepeater('scheme in vm.schemes').row('0'));
this.alias = element(by.model('reference.alias'));
this.reference = element(by.model('reference.reference'));
this.saveBtn = element(by.css('.md-raised'));
this.createOperator = function() {
this.licenceNumber.sendKeys(this.helper.getRandomId(10));
this.tradingName.sendKeys('Protractor Trade Name LTD');
this.name.sendKeys('Protractor Trade Name');
this.operatorAddressFirstLine.sendKeys('Protractor Town');
this.operatorAddressCityTown.sendKeys('Cardiff');
this.operatorAddressPostCode.sendKeys('PT4 4TP');
this.payeeAddressFirstLine.sendKeys('Protractor Town');
this.payeeAddressCityTown.sendKeys('Cardiff');
this.payeeAddressPostCode.sendKeys('PT4 4TP');
this.opID.sendKeys('177');
this.spID.sendKeys('Protractor Spid');
this.schemeSelect.click();
this.schemeOptions.click();
this.alias.sendKeys('PTAlias');
this.reference.sendKeys('Protractor');
this.saveBtn.click();
}
};
module.exports = operatorsSection;
In this test suite after the call to createOperator from the PO file is invoked and the savteBtn is clicked, the application will transition to a state that shows a table of created entries (after successful creation of course). We are using angular ui-router also, currently on version 0.2.18
The expectation fails with:
Expected 'Create An Operator' to equal 'Operators'.
Yet the accompanying screenshot that was captured shows the table view with an 'Operators' heading, it seems the call to page.headline.getText() inside the expectation call is being invoked too soon, so before the database operation to create the item and the page change has had a chance to complete?
I have started wondering if this could be down to the order of promises executed by protractor. I have come across articles talking about control flow in protractor and why there may be occasions when you should hook into the result of a protractor call's promise using .then() - I found this
It got me wondering if I should move the call to my saveBtn.click(), that's called at the end of my page object's createOperator function, into the test suite, so doing something like:
it('should create an operator', function () {
page.settlementBtn.click();
page.operatorsBtn.click();
page.fabBtn.click();
page.createOperator();
page.saveBtn.click().then(function(){
expect(page.headline.getText()).toEqual('Operators');
});
});
I'm starting to clutch at straws here to be honest, so any thoughts\advice from the community here would be much appreciated.
Thanks!
As requested, here is the function I use for waiting for URLs to be as they should.
public waitForUrlToBeLike (urlPart: string, timeout: number = 10000) {
return browser.wait(() => {
return browser.driver.getCurrentUrl().then((url) => {
let regex = new RegExp(urlPart);
return regex.test(url);
});
}, timeout);
}
I also use the following a lot to wait for elements to be present before making assertions on them:
public waitTillPresent (element: ElementFinder, timeout: number = 10000) {
return browser.wait(() => {
return element.isPresent();
}, timeout);
}
Related
So I'm building an Apache Cordova app (Android), angularjs based, and using the WindowsAzure MobileServiceClient library to retrieve data from my Easy Table's, created in a SQL Database.
This works! Until I'd like to retrieve more then 50 records.
So I added the .take(100) to my table-read. Still 50 records..
Then I thought, maybe the take function doesn't work at all, so I changed the amount to 5, and I only got 5 records. So the take function is somewhat working, but not above 50 records.
Since it's a node.js backend, how do I increase the pagesize?
Here's my current code:
msc.tables.pokemonType = null;
msc.tables.pokemon = null;
msc.init = function () {
console.log('MobileServiceClient initializing...');
var mobileServiceClient = new WindowsAzure.MobileServiceClient("https://blablablabla");
msc.tables.pokemonType = mobileServiceClient.getTable('PokemonType');
msc.tables.pokemon = mobileServiceClient.getTable('Pokemon');
console.log('MobileServiceClient initialized!');
}
msc.getAllPokemonTypes = function() {
console.log('getAllPokemonTypes - called');
return msc.tables.pokemonType.read().then(function (pokemonTypes) {
console.log('getAllPokemonTypes - done');
return pokemonTypes;
}, msc.handleError);
}
msc.getAllPokemons = function () {
console.log('getAllPokemons - called');
return msc.tables.pokemon.take(100).read().then(function (pokemons) {
console.log('getAllPokemons - done');
return pokemons;
}, msc.handleError);
}
According the source code of table operations of Mobile Apps in node.js, the read operation ultimately receives context.query which is a queryjs object, which contains a take() function which can limit the number of items returned to the specified number.
Additionally, the take() function is contained in the mobile app server sdk, so it doesn't work on your client end code.
You can do some modification on your Easy Tables scripts, E.G.
table.read(function (context) {
context.query.take(100);
return context.execute();
});
I'm testing my angular application with Protractor.
Once the user is logged in to my app, I set a $timeout to do some job in one hour (so if the user was logged-in in 13:00, the $timeout will run at 14:00).
I keep getting these failures:
"Timed out waiting for Protractor to synchronize with the page after 20 seconds. Please see https://github.com/angular/protractor/blob/master/docs/faq.md. The following tasks were pending: - $timeout: function onTimeoutDone(){....."
I've read this timeouts page: https://github.com/angular/protractor/blob/master/docs/timeouts.md
so I understand Protractor waits till the page is fully loaded which means he's waiting for the $timeout to complete...
How can I make Protractor NOT wait for that $timeout?
I don't want to use:
browser.ignoreSynchronization = true;
Because then my tests will fail for other reasons (other angular components still needs the time to load...)
The solution will be to flush active timeouts (as #MBielski mentioned it in comments), but original flush method itself is available only in anuglar-mocks. To use angular-mocks directly you will have to include it on the page as a <script> tag and also you'll have to deal with all overrides it creates, it produces a lot of side effects. I was able to re-create flush without using angular-mocks by listening to any timeouts that get created and then reseting them on demand.
For example, if you have a timeout in your Angular app:
$timeout(function () {
alert('Hello World');
}, 10000); // say hello in 10 sec
The test will look like:
it('should reset timeouts', function () {
browser.addMockModule('e2eFlushTimeouts', function () {
angular
.module('e2eFlushTimeouts', [])
.run(function ($browser) {
// store all created timeouts
var timeouts = [];
// listen to all timeouts created by overriding
// a method responsible for that
var originalDefer = $browser.defer;
$browser.defer = function (fn, delay) {
// originally it returns timeout id
var timeoutId = originalDefer.apply($browser, arguments);
// store it to be able to remove it later
timeouts.push({ id: timeoutId, delay: delay });
// preserve original behavior
return timeoutId;
};
// compatibility with original method
$browser.defer.cancel = originalDefer.cancel;
// create a global method to flush timeouts greater than #delay
// call it using browser.executeScript()
window.e2eFlushTimeouts = function (delay) {
timeouts.forEach(function (timeout) {
if (timeout.delay >= delay) {
$browser.defer.cancel(timeout.id);
}
});
};
});
});
browser.get('example.com');
// do test stuff
browser.executeScript(function () {
// flush everything that has a delay more that 6 sec
window.e2eFlushTimeouts(6000);
});
expect(something).toBe(true);
});
It's kinda experimental, I am not sure if it will work for your case. This code can also be simplified by moving browser.addMockModule to a separate node.js module. Also there may be problems if you'd want to remove short timeouts (like 100ms), it can cancel currently running Angular processes, therefore the test will break.
The solution is to use interceptors and modify the http request which is getting timeout and set custom timeout to some milliseconds(your desired) to that http request so that after sometime long running http request will get closed(because of new timeout) and then you can test immediate response.
This is working well and promising.
as you may already know many of us who have a large quantity of written unit test has met with this not trivially solvable problem. I have around 3500+ unit tests written in the Jasmine syntax following the AngularJs unit testing guide. The tests are executed with Karma runner.
The problem is that they cannot be executed all at once due to some memory leaks. While running them the memory builds up no matter on what browser they are runned on and at some point the browser crashes and disconnects. The best workaround that I am aware of by now which is used in the community that have this problem is spliting the tests in multiple runs and at the end get the correct coverage by merging the results from the single runs.
When I first met with this problem I had around 1000 tests. After trying with all the available browsers for running I have split the tests in multiple runs, however it turned out that this is not good workaround for a long time. Now the tests are executed in 14+ single runs that are runned in parallel to reduce the time for completition and still IMO this cannot permanently solve the problem but delay it a litle bit because of resources limitation (RAM, CPU) and annoying time consumption.
Someone can argue that I have memory leaks in my code for which I cannot guarantee even though I don't have any problems alike when running the application in the browser. That is why I have created an example project that will highlight this problem.
There for reproducing this problem I am creating an Angular service which is heavy in memory consumption like this:
app.factory('heavyLoad', function () {
// init
var heavyList = [];
var heavyObject = {};
var heavyString = '';
// populate..
return {
getHeavyList: function () { return heavyList; },
getHeavyObject: function () { return heavyObject; },
getHeavyString: function () { return heavyString; }
};
});
After that I have a simple directive which uses this service to initialize many DOM elements:
app.directive('heavyLoad', function (heavyLoad) {
return {
scope: {},
template: '' +
'<div>' +
' <h1>{{title}}</h1>' +
' <div ng-repeat="item in items">' +
' <div ng-repeat="propData in item">' +
' <p>{{propData}}</p>' +
' </div>' +
' </div>' +
'</div>',
link: function (scope, element) {
scope.items = heavyLoad.getHeavyList();
scope.title = heavyLoad.getHeavyString();
// add data to the element
element.data(heavyLoad.getHeavyList());
}
};
});
And at the end I am dynamically registering 1000 test suites with the test definition for the directive which btw is written as suggested in the Angular unit testing guide.
// define multiple suits with the same definition just for showcase
for (var i = 0; i < 1000; i += 1) {
describe('heavyLoad directive #' + i, testDefinition);
}
To try the example just checkout the project from GitHub and before running karma start run:
$ npm install
$ bower install
I am looking forward to finding where the problem is and solving it finally.
Cheers
The problem was in the forgotten clean-up that needs to be done after each test.
After adding it the number of tests does not matter anymore because the memory consumption is stable and the tests can be run in any browser.
I have added a modification of the previous test definition here that shows the solution with successfully executing 3000 dinamically registered tests.
Here is how the test looks like now:
describe('testSuite', function () {
var suite = {};
beforeEach(module('app'));
beforeEach(inject(function ($rootScope, $compile, heavyLoad) {
suite.$rootScope = $rootScope;
suite.$compile = $compile;
suite.heavyLoad = heavyLoad;
suite.$scope = $rootScope.$new();
spyOn(suite.heavyLoad, 'getHeavyString').and.callThrough();
spyOn(suite.heavyLoad, 'getHeavyObject').and.callThrough();
spyOn(suite.heavyLoad, 'getHeavyList').and.callThrough();
}));
// NOTE: cleanup
afterEach(function () {
// NOTE: prevents DOM elements leak
suite.element.remove();
});
afterAll(function () {
// NOTE: prevents memory leaks because of JavaScript closures created for
// jasmine syntax (beforeEach, afterEach, beforeAll, afterAll, it..).
suite = null;
});
suite.compileDirective = function (template) {
suite.element = suite.$compile(template)(suite.$scope);
suite.directiveScope = suite.element.isolateScope();
suite.directiveController = suite.element.controller('heavyLoad');
};
it('should compile correctly', function () {
// given
var givenTemplate = '<div heavy-load></div>';
// when
suite.compileDirective(givenTemplate);
// then
expect(suite.directiveScope.title).toBeDefined();
expect(suite.directiveScope.items).toBeDefined();
expect(suite.heavyLoad.getHeavyString).toHaveBeenCalled();
expect(suite.heavyLoad.getHeavyList).toHaveBeenCalled();
});
});
There are two things that need to be cleaned-up:
compiled element when using $compile for testing directives
all variables in the describe functions scope
The two of them are tricky and hard to find out and take into consideration.
For the first one I already knew but it didn't helped much until I've discovered the second which is related with how Jasmine works inside.
I have created an issue on their GitHub repository which should help finding better solution or at least spread this information among developers faster.
I hope that this answer will be helpful for lot of people having this problem. I will write some info too after I finish refactoring all my other tests.
Cheers!
Due to Protractor 2.0's breaking changes, I have to refactor my at() function. I use this on non-angular pages to both wait for the page to load, and return true, so it's expectable (if that's a word).
Because 2.0 no longer returns a promise from a locator, it's breaking my wait (wait fires immediately). Thus, I've turned to the new ExpectedConditions... which works fine. My question is, am I using this correctly, or is there a better solution?
Page object locator:
this.pageLoaded = $('div#splash[style="display: none;"]');
Works in Protractor 1.8.0 (breaks in 2.0.0):
this.at = function() {
var that = this;
return browser.wait(function() {
return browser.isElementPresent(that.pageLoaded);
});
};
My working solution for Protractor 2.0.0:
this.at = function() {
var that = this;
return browser.wait(function() {
return protractor.ExpectedConditions.presenceOf(that.pageLoaded);
});
};
And for example, I would call this like so:
expect(mainPage.at()).toBeTruthy();
It looks like there was another breaking change in the Protractor 2.0 changelog, referring to a problem in WebDriver 2.45 that makes the wait parameter required:
Due to changes in WebDriverJS, wait without a timeout will now default to waiting for 0 ms instead of waiting indefinitely.
Before:
browser.wait(fn); // would wait indefinitely
After:
browser.wait(fn, 8000) // to fix, add an explicit timeout
This will be reverted in the next version of WebDriverJS.
In our Protractor tests, we always refer to a common helper class var helper = require("..\helpers.scenario.js"). To fix this problem, we just wrapped browser.wait in the helper class and passed in a timeout.
wait: function(f) {
return browser.wait(f, 30000);
},
We then used find-replace to change from browser.wait to helper.wait.
I'm creating users in some test. Since it is connected to the backend and create real users I need fixtures. I was thinking of using the browser name to create unique user. However, It has proven to be quite difficult to get to it...
Anyone can point me in the right direction?
Another case of rubber ducking :)
The answer was actually quite simple.
in my onPrepare function I added the following function and it works flawlessly.
browser.getCapabilities().then(function (cap) {
browser.browserName = cap.caps_.browserName;
});
I can get access the name in my test using browser.browserName.
This has changed in version of protractor starting from 3.2 (selenium webdriver 2.52)
Now one should call:
browser.driver.getCapabilities().then(function(caps){
browser.browserName = caps.get('browserName');
}
If you want to avoid the a browser, you may want to do this:
it('User should see a message that he has already been added to the campaing when entering the same email twice', function () {
browser.getCapabilities().then(function (capabilities) {
browserName = capabilities.caps_.browserName;
platform = capabilities.caps_.platform;
}).then(function () {
console.log('Browser:', browserName, 'on platform', platform);
if (browserName === 'internet explorer') {
console.log('IE Was avoided for this test.');
} else {
basePage.email.sendKeys('bruno#test.com');
console.log('Mande el mail');
basePage.subscribe.click().then(function () {
basePage.confirmMessage('Contact already added to target campaign');
});
}
});
});