Jmeter Webdriver sampler - selenium-webdriver

Jmeter has a webdriver sampler. We have to write the scripts. We have scripts written in opkey, a selenium based tool. Can we integrate both of them. So that we don't have to write scripts in jmeter.

Not familiar with opkey, but Selenium is a bad idea for any load tests beyond very trivial loads:
Selenium was never intended for large-scale performance testing. Selenium and its newer avatar webdriver, launch a browser engine per user and then replay all the user interactions inside it. This is great for functional testing because you are executing client side code inside a real browser engine - but at the same time this is very bad news for performance testing. Browser instances are resource intensive, and scaling becomes hard and expensive.
Even using something like Selenium Grid is really meant for cutting your test execution time by running in parallel, but not really for generating any sort of loads. They say this at the very top of their FAQ.
It is not just scale, when your load driver itself is very resource intensive, the applied load becomes inconsistent. If you see drop in performance, it could very well be that your load driver is the bottleneck and not the application under test.
Having said that, you can definitely use the JMeter Sampler, or Se Grid, or something else to drive your performance test, as long as you're in the scale of 10s of parallel users. Again quoting from Grid FAQ:
To simulate 200 concurrent users for instance, you would need 200 concurrent browsers with a load testing framework based on Selenium Grid. Even if you use Firefox on Linux (so the most efficient setup) you will probably need at least 10 machines to generate that kind of load. Quite insane when JMeter/Grinder/httperf can generate the same kind of load with a single machine.
Note that when they say JMeter, they are referring to the HTTP sampler or one of the simpler, more efficient samplers - Because even the webdriver sampler documentation says this:
JMeter allows the creation of multiple threads, and each thread is responsible for creating load on the server. However, for the Web Driver use case, the reader should be prudent in the number of threads they will create as each thread will have a single browser instance associated with it. Each browser consumes a significant amount of resources, and a limit should be placed on how many browsers the reader should create.
and then goes on to recommend using a maximum of (1 less than the #of processor cores) - which is a very small number for most non-elastic setups.

Related

What kind of server should I build to run Selenium and Appium automation tests?

I am building a server to run my Selenium/Appium automation scripts. I was told that I would need a more sophisticated server, like machine learning, because Webdrivers require more processing. Just wondering what everyone uses and if it works
Best
Using a server with multiple processing power like 4-8 cores or GPU will help for parallel testing. Also a sufficient amount of RAM is needed. At least 4-8 gigs.

Any known add-ins that increase IE Execution Speed?

My team is using Ruby Watir Webdriver to create automation scripts for our applications. IE11 is our browser that we must script to. We have noticed that our same script runs 30% faster in Firefox than it does in IE. This difference is speed, affects our script's ability to "see" elements fast enough. Are there any add-ins that increase IE's speed? We are using IEDriverServer version 2.48.0
Surely 30% is a tolerable difference between two completely different browsers? I'm very surprised the difference is so small.
As these posts from Jim Evans of Selenium show, working with the IE Driver hasn't always been easy:
http://jimevansmusic.blogspot.co.uk/2014/09/screenshots-sendkeys-and-sixty-four.html
http://jimevansmusic.blogspot.co.uk/2014/09/using-internet-explorer-webdriver.html
http://jimevansmusic.blogspot.co.uk/2012/06/whats-wrong-with-internet-explorer.html
Perhaps this is a legitimate reason for upping the hardware spec for your IE test environment, so that the cost is mitigated. Or else concentrating your tests so that only the IE-specific ones need IE, while the uncontroversial bulk can run on faster environments.
Speed should never be allowed to influence test reliability or variability, but provided you are using proper waits, that shouldn't be an issue. It wouldn't be fair to blame the IE Driver for that.
I doubt that the use of any particular testing framework (Watir) will make too much difference.

How to do load test with real user nearly 10.000 user?

I'm planning to do a load test for e-commerce site for valentine's day. I want to do this test with real user. I prepared it Jmeter WebDriver Sampler. I just found blazemeter.com but when i upload my scripts on blazemeter i got errors. I'm waiting a response from them. Do you know any platform, cloud servers provider or anything else to do this?
There's tons of tools, some of them are;
https://www.blitz.io/
http://loadstorm.com/
https://loader.io/
https://flood.io/
http://www.neotys.com/introduction/neoload-cloud-testing.html
http://www.soasta.com/products/cloudtest/
We work with BlazeMeter, which is easy to use, stable, and its staff are helpful if we stuck at dead-end. It creates nice reports, too.
Just to add one more to the mix:
http://loadimpact.com
Also in my 15 years experience of performance and load testing I have very rarely found a need for doing tests with full clients.
The virtual users are much more efficient and scale well in tests even if they possibly need some scripting and data management and reverse engineering of how the service you are testing works on the client.
And running full clients will consume huge amounts of server resources. One client per core in your load generators. Testing an e-commerce site I would estimate requires at least volumes in the thousands.
1.000 test users would require 250 4-core load generators. It quickly becomes large and expensive. For 500 virtual users you can get by with as little as a single 2-core load generator.

Load testing using Jmeter 5k users, website crashed

I was running 5K users on a website while load testing with Jmeter. It was taking too long time and my colleagues were not able to access the site. Then i stop the test. now the website is totally dead and is not accessible. What should I do now?
You should either restart your browser and/or your entire system.
The best would be use ramp up technique. This does not over load the server all at once but adds the load slowly till its reaches peak. This way it is easier to find the crash point.
Restarting the server will be a good way to bring up the application. Also check the server Logs.
First of all I would recommend you to try to understand how many users are going to use your system to set clear test objectives.
These articles I find useful:
stress testing web applications with JMeter (Part 1)
stress testing web applications with JMeter (part 2)
After you've set up your testing goals clearly follow JMeter Performance Testing best practices steps.
Also I would recommend you to get acquainted with Load Testing Cloud - altentive cloud solution for running your .jmx tests instead of local run, which often associated with resource demand on your PC machine.

Is there a framework for distributing browser automation testing over a cluster of EC2 instances?

I am aiming to simulate a large number of 'real users' hitting and realistically using our site at the same time, and ensuring they can all get through their use cases. I am looking for a framework that combines some EC2 grid management with a web automation tool (such as GEB/WATIR). Ideal 'pushbutton' operation would do all of this:
Start up a configurable number of EC2 instances (using a specified
AMI preconfigured with my browser automation framework and test
scripts)
Start the web automation framework test(s) running on all of them,
in parallel. I guess they would have to be headless.
Wait for completion
Aggregate results
Shut down EC2 instances.
While not a framework per se, I've been really happy with http://loader.io/
It has an API for your own custom integration, reporting and analytics for analysis.
PS. I'm not affiliated with them, just a happy customer.
However, in my experience, you need to do both load testing and actual client testing. Even loader.io will only hit your service from a handful of hosts. And, it skips a major part (the client-side performance from a number of different clients' browsers).
This video has more on that topic:
http://www.youtube.com/watch?v=Il4swGfTOSM&feature=youtu.be
BrowserMob used to offer such service. Looks like they got acquired.

Resources