What options do I have to performance test an Electron application in a similar way to how Load Runner does it?
I need to performance / load test an Electron application built using React. The application is deployed to Windows 7 (possibly to Windows 10 as well) and invokes ReST APIs.
I'm aware of the Spectron option but it doesn't sound like it is automated out of the box. I'd prefer to use a framework that already provides recording, playback, response time metrics, and configuring multiple scripts.
Many thanks in advance
REST means HTTP so you need a performance testing tool which supports recording HTTP traffic.
The majority of performance testing tools provide TCP or HTTP proxy server for capturing the traffic and converting it into internal form of tests
Assuming above 2 points you need to:
Choose a performance testing tool which suits your needs
Configure your Electron application to use the selected tool as a proxy, your application either has its own proxy settings or respects underlying operating system proxy setup.
This way your Electron application will communicate with the backend via performance testing tool proxy which will be capturing requests so you will be able to replay them with increased load later on. Check out How to Run Performance Tests of Desktop Applications Using JMeter for example instructions.
Related
Just an R&D question. We need to develop an application that can be run in a browser that has the capability of performing some system checks to gather support information to be emailed to us. These checks will include basic system information, but also will need to scan the filesystem and pull out version information about various DLLS, executables, and .NET assemblies that might be installed. The idea being that we can direct a client to a page and have the application gather the relevant information needed for support, and potentially even populate some database fields. We need it to have as small a footprint as possible.
I've worked with ActiveX before, and know it is capable of these things, but particularly on modern systems security is a nightmare to get around, with a lot of people blocking ActiveX altogether. Is Silverlight easier to deliver to clients? Does it have a lighter footprint? Is it even capable of doing these things?
Silveright has access to isolated storage, but I don't think it can do what you are looking for (I may be wrong). As for footprint, if I remember correctly, the runtime is reasonably small, and the .xap packages are limited to 4Mb.
Silverlight out-of-browser has access to the file system. http://msdn.microsoft.com/en-us/library/dd550721(v=vs.95).aspx#special_features_for_outofbrowser_applications
If you intend to run your app in the browser, you will still have to configure the trust as if it where oob. http://msdn.microsoft.com/en-us/library/gg192793(v=vs.95).aspx
However, iTunes has a neat way of doing something somewhat related. It has a custom protocol (itms://) that allows the browser to invoke a client side program (iTunes). Then you can embed html in a webpage that passes parameters as command line arguments to that app. The website also knows if the iTunes is installed by a cookie. We this in mind, you might be able to encourage your users to install some small app that setups the custom protocol on install. You could pass command-line parameters to it from the web, and the app will push information from the client back to the server.
To create a real-time experience, you could use sockets + more javascript to update the page with the info you just got off the machine.
HTH,
Silverlight runs in a pretty restricted silo and can't do a lot of low level things - such as checking the file system. So I would say it does not fit your use case, unfortunately.
Is there a good solution for A|B Testing in mobile apps like online? I know with iOS it's against the TOS to have different user experiences with identical actions, but what about Android? And what about firms like Apsalar which claim to offer A|B Testing in their analytics for apps? How would one implement that?
Artisan mobile makes an A/B testing solution for iOS and Android.
The basic idea is that you drop the SDK in your app and then put it out in the app store. You can use the service to create A/B tests and optimize your application without having to touch the code or go back through the app store for each test.
For mobile apps, A/B testing basically works by replacing static, hard-coded objects with dynamic objects that can be controlled from a remote server.
This methodology raises a potential performance issue: What if the end user's device is not connected to pull configuration data for an object being tested? We've built Splitforce (http://splitforce.com) to seamlessly setup and manage A/B testing in mobile apps while controlling for performance risk.
Los details
Once the SDK and experiment has been integrated, non-technical product or marketing folks can setup new tests or tweak existing tests on-the-fly - without having to resubmit to the app stores or hassle engineers.
On first app launch, the mobile app requests configuration data from the server and then caches that data locally on the device. This is to both ensure a consistent user experience on subsequent app launches, and prevent corrupt test results by guaranteeing accurate attribution of conversion events to variations.
If the end user's connection fails or is timed-out on first app launch, the library displays a hard-coded 'default' variation. And to make sure that everything is looking good before you go live, we've built a 'shake to preview' functionality in debug mode that does just that :-)
Once the app is deployed with Splitforce event data are stored locally and sent back to the website to be displayed for each variation alongside measurements of observed improvement and statistical confidence.
Instructions on integration of the SDKs and new tests can be found at https://splitforce.com/documentation.
And how is it used?
We've seen Splitforce used to A/B test:
UI elements + layouts (color, text, images, ad/menu placements)
UX workflows
Game dynamics + rules
Prices + promotions
We've also seen the tool used to control mobile apps remotely, by essentially setting one variation of a test subject to 100%.
Yes there is: E.g. the company Leanplum offers a Visual Interface Editor for iOS and Android: This requires no coding, and Leanplum will automatically detect the elements and allow you to change them. No engineers or app store resubmissions required.
Apple must have updated their TOS (https://developer.apple.com/app-store/review/guidelines/#user-interface) - At least I am not aware of anything that prohibits altering the UI in a way that the Leanplum Visual Editor is doing it.
Generally that is achieved by method swizzling (iOS) and reflection (Android).
To learn more about the Leanplum Visual Interface Editor, check out leanplum.com. They offer a free 30-day trial.
(Disclaimer: I am an Engineer at Leanplum.)
I wrote a small open source project called Switchboard.
It let's you A/B test, remote configure and stage rollout things in your native mobile app. It contains a server component that specifies what information the application should have and 2 native clients for android and iOS.
You can find the codebase at github.com/keepsafe/switchboard and a blog post about how you can use it HERE
The new kid around the block is Arise.io. They provide an A/B testing service for iOS and Android.
I wrote MSActiveConfig, an extremely flexible framework to do remote configuration + A/B testing on iOS, with a portable format to be able to implement clients on other platforms: https://github.com/mindsnacks/MSActiveConfig.
This framework is being used in applications with more than 5 million users.
There have been a spate of new entrants in this field...you could check out Swerve, Appiterate, leanplum...all of them seem to be having SDKs for iOS as well, not really sure whether and how Apple TOS allows for that, but since there are some many of them doing it, there must be a way.
Yes, new entrants are showing up in app A/B testing practically every week! But, I think Appiterate has gone two steps ahead of other competitors by creating a visual interface, without any need to re-write code. I have seen their platform (you can ask for an invite. I got a demo within 12 hours) and believe me, it is actual WYSIWYG that they are providing.
I'd like to build a multiplayer web game application in which it supports chat. I presume the application will have to handle hundreds of simultaneous connections.
I'm planning to host my application in a shared web hosting, which has these limitations (most likely similar to PHP + Comet (long-polling) scaling / hosts):
It does not seem I can change the web server. Most likely it's using Apache.
Supports MySQL 5, PHP 5.3.x, Perl, Python, Ruby on Rails, CGI
(To be more precise, I'll be using HawkHost's shared web hosting.)
And here are my result of research, followed by my questions:
Some resources (like Python Comet Server) say that PHP is not good for handling concurrent connections, while Python is better choice. Is this true?
I've tried the long polling technique in PHP (although I don't know whether it's correctly implemented or not, like Comet issue with abandoned open connections) using "Loop endlessly until the data changes." method. This almost works. The remaining problem is that the server process never dies when the browser is closed (the server does not know that the connection has been terminated, and the data never changes). Is there any way the PHP can detect whether the browser has been terminated so that it stops the loop?
I've been looking everywhere to look for answers but still I can't conclude anything. This topic has also been asked on StackOverflow so many times, I'm sorry if I may sound repeating >.<.
Currently I am able to code using PHP, MySQL, and JQuery for JS. I'm still new to the term Comet and Server Push. If necessary, I'm also willing to learn new scripting language like Python.
I appreciate any insights of what scripting language, framework, and techniques to use to start my project.
When you have a shared hosting environment and there are a number of restrictions enforced then it's a good idea to outsource the realtime functionality. I would say this since I work for one such company, Pusher. But I hope others will back me up on this.
When using a hosted solution you can push a notification by making a HTTP request to a RESTful API. The service will then deliver the message to the connected Web Client (browser). The browser does need to include a script tag or use a library that also connects to the hosted service.
The main benefits are:
No installation or maintenance
No need to handle persistent connections - no resource usage
Really simple usage: Script tag in app and call REST API
The hosted solution handles scaling
Also, here's a list of hosted realtime solutions.
So you can use Python. Then you can use Tornado. (psst... facebook uses it)
And I had same problem with open connections. Just don't spend time for search solution in PHP - later you will be sorry. I was. Just use what is made for Comet. If you more prefere JAVA, then there is: CometD.
And for game get a normal hosting. They cheap this days.
I just heard that a company I do work for may be bringing in the Pyxis Mobile application development system. When I google it most of what I find is from the company's web site and that is not very informative from a geek perspective. Can any one shed some light on what sort of programming environment it is and what programing language is involved (please let there be a text based language). Any additional information would be great.
Note: the company/product changed their name to Verivo in January.
Full Disclosure - I work as an engineer at Pyxis Mobile. However, I have been in the mobile space for 7+ years and have evaluated several approaches to mobile so hopefully this is helpful.
Pyxis Mobile provides a set of tools and components to build cross platform mobile applications. Let me outline them first.
1. Application Studio - All application development, backend integration, user provisioning and application maintenance/debugging is done w/in this tool. Application Studio (for now) is a Windows based desktop app.
2. Application Clients - Pyxis Mobile provides native client runtimes for iPhone, iPad, BlackBerry, and Android devices. These runtimes get branded for the customer through a build service and are primed to point to a specific Application Server URL.
3. Application Server - Pyxis Mobile App Server runs on the .NET stack (on IIS). All client communication is proxied via this server. This server is able to connect to varied of backend systems (via the Plugin Framework listed below) and respond to the client in a mobile optimized manner. This server needs a SQL Server (2005 or newer) for configuration access, session management, logging and more.
4. Plugin Framework - The Plugin Framework is a backend component that provides system specific pre-built access to several of the enterprise and cloud based systems (Oracle, Siebel, SAP, Salesforce.com, social feeds, REST/SOAP web services, etc.) and also offers an API layer in .NET and Python (using IronPython) to allow even further customization. A plugin is essentially comprised of one or more DLLs or a Python file. These assets are then dynamically loaded to normalize communication between Pyxis Mobile and the customers' backend systems.
5. Push Services - This provides a cross-platform push layer that can poll a backend system for change and alert a mobile device via BlackBerry Push, Apple Push Notification Services (APNS) or Android's Cloud to Deice Messaging (C2DM).
6. OverWatch Analytics - This is an optional (but included) component to track users/devices and provide integrated analytics on what the users are using and what kind of devices and locales makes up your users.
The application itself is "coded" via configuration that is build in App Studio. Pyxis Mobile abstracts away from the code so that you can work at a higher level without having to worry about the wide array of device variances (GPS, touch screens, camera, accelerometer, push, screen resolution, etc.). You can drag fields onto a from, connect screens via menus or buttons, set up caching rules and more in this graphical utility. This configuration (essentially think of an XML like document) is interpreted by the native client layer to produce a rich application. There is also a scripting layer in Lua that allows to really customize behavior via code.
The real value of Pyxis Mobile comes up when you have change to make. The clients check for new configuration at app startup or if the server forces the client to get new configuration. This gives you great agility. Lets say once your application is deployed you want start using the swipe gesture to go next/prev through a set of records. This change on other platforms would mean writing some platform specific code to trap and interpret the swipe to perform a navigation (you couldn't trap a swipe on a non-touch screen). However, in Pyxis Mobile this is a simple configuration change that can be quickly deployed to the App Server and the clients automatically download and use the new configuration. No compilation, no redeployment or re-download for the end users.
I could keep going, but hope this provides some level of guidance.
Beware of Pyxis Mobile. While many of the things they say do work, there are some serious platform issues (as a geek) which I've experienced.
1) No version control system process. The Application studio can basically only be developed on by one person at a time or you risk having your changes overwritten by a fellow developer. The "principle of last save" is very much in play.
2) No unit test coverage. This isn't the biggest issue for a lot of people, but it's a concern for anyone who wants to work in the Enterprise world.
3) The middleware server gets you some value, but it's also a PITA to work with. There is no concept of "client side storage" unless you consider the middleware server the client side. If your phone goes out of coverage, your app won't work. Again, this might not be an issue for you.
4) The application has no true scripting language to work with. The middleware server allows you to intercept requests and responses and modify what you're doing there, but it's not the most elegant solution considering that a native application can have something as simple as "if this then X else Y." This can be accomplished with Pyxis, but the whole process is convoluted and more complicated than one would think it needs to be.
5) Lack of documentation. There's some training guides and the GUI is easy enough to get around for simple apps; however, when you need to do something with guts, you're left relying on Pyxis professional services. There's really no developer community to pose questions to.
I have more complaints, but they are more opinion oriented than Q/A oriented.
I just got note about the most recent comments. I don't want to turn this into a thread of back and forth, but did want to throw a couple of quick notes.
Regarding the points on version control and documentation/developer community - no big contest there. We are definitely working on these shortcommings. We have some basic pieces in place, but we have big plans to focus on this.
Regarding unit testing - we provide a very open interface to our middleware and backend components and they can be very easily unit tested with a bit of instrumentation. We run a ton of unit and integration tests internally. However, mobile unit testing is extremely difficult to get right. We'll investigate this further.
Regarding #4 around middleware and offline capabilities - things are a lot different now. With version 7.1, 7.2 and 7.3 our products have increasing become more capabale offline and now features a secure local database if necessary. I can provide more details as necessary, but you can certainly login and operate the app even if you are out of coverage for weeks at a time!
Regarding #5, we've had a scripting engine for over 2 years. Its Lua based and its actually quite powerful and fast. It was BlackBerry only till the most recent release. Given Apple's change of stance on allowing scripting we now allow scripting on BlackBerry, iPad, iPhone and Android as well now!
#RockMeetHardplace - feel free to reach out to me directly and I'll be happy to give you more detailed live demos of our latest platform. I am at - arunSPAMNOTatpyxismobiledotcom (drop the "SPAMNOT" and replace the at and dot). I happen to be the Director of Software and interested in knowing more about the issues you had.
how can i record the web server log files and replay the the recorded server logs so that i can latter run the logs and check the load on sql server.
ex. number of request on and the number of resposes on sql server
Build a tool that can parse the log files from the web server and replay the requests again. Then you generate test cases by accessing your site using a normal web browser and record the log files.
You can then replay the same sequence over and over again and your tool could also read multiple logs and play them back simultaneously to simulate concurrency.
Make sure that the log files include all GET and POST request parameters as well as timestamps.
Here are a couple tools that you might find useful:
Selenium is a suite of tools specifically for testing web applications.
http://seleniumhq.org/
ab (Apache HTTP server benchmarking tool) is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving.
http://httpd.apache.org/docs/2.0/programs/ab.html
What's wrong with using a tool? Tools like selenium will exercise the front-end, you can probably setup a server and a collection of clients running on separate machines to simulate user interaction.
If you want to exercise the backend more directly you could write a unit test I suppose, but unit testing isn't really designed for doing that sort of thing.