I have a benchmarking situation that requires some advise.
This is basically a scenario.
I typically use Jmeter to benchmark web page loads.
However in this case, I intend to benchmark an URL that will make some API calls. Basically I'm interested to see the response time of each API call.
The tricky part is that one of the API calls requires a mobile data connection (3G/4G) because the connection will be redirected to carriers to identify which carrier the mobile phone number belongs to. If every carrier does not recognize the mobile phone number, the API call will fail.
I did a manual benchmark(with Jmeter) by connecting my machine to a tethered mobile phone. This worked, however, I find it impractical to have a machine to wirelessly connect to a mobile phone just to run benchmarks. I cannot imagine putting a mobile phone (for every carrier) in a server room.
Does anyone have any idea or any experience in benchmarking an API that requires 3g/4g connection? Are there any tools out there?
I tried googling around, but, did not come out with anything useful.
Any advise is appreciated.
Related
I'd like to retrieve a log of crashes for my mobile app from IBM Mobile Quality Assurance (MQA) so that I can filter/process these crashes based on certain characteristics. In particular, I want to excluded crashes that occur in the device simulator since these are most likely happening during automated tests and not actual user crashes.
Is there a REST API or other means to obtain a list of crashes (with details) for my app from IBM MQA?
We are working to improve the MQA filtering features however at this time we don't have a way to filter crashes by devices. We recently announced that MQA in Bluemix now supports integration with the following bug tracking systems: Jira, TFS, HP Quality Center, GitHub, FogBugz.
If you use one of these BTS's, its possible to use their filtering process, if any.
I will speak to the MQA Product Mgr to take this as a future enhancement request.
We need to develop a multi-player game with real-time performance.
This needs to be working worldwide (servers in America, Europe, Asia), and supporting a huge traffic. Using Google Cloud services for the hosting.
We're thinking of references like Jam with Chrome, Chrome Maze or Cube Slam.
The game :
2 players challenge a race
We need to simultaneously display the progression of the 2 players
Each match could last around 30 to 45 seconds
The hosting :
We will obviously host the website on AppEngine, automagically scaling,
but are thinking about 2 solutions for the real-time servers :
Using websocket servers with Compute Engine
Like they did for Jam with Chrome, Maze, etc.
Developing our own websocket servers (technology TBD), deploying on datacenters in Europe, US, Asia, handling scaling, syncing between them, computing latency issues on servers and clients, etc.
But it's pretty technically challenging as we are very short on time, and missing an admin sys and network guy for now.
Or using Channel API
We understand that it's not a websocket platform, and real-time performances are lower.
But it would be way more simple and secure for us and the time we have.
So, we would also like to know more about that.
In any case, we think we could use some graphical tricks on front ends, to make it look like real-time, but it really depends if we have a 100~500ms or a 500ms~10s latency.
Some questions :
What would the latency range values look like for the different solutions ?
(Jam w/ Chrome got 100ms with GCE, could Channel API reach several seconds ?)
How would Channel API servers handle high traffic, how does scaling work, could the latency go very high ? (no info about that on Channel docs ?)
What if someone in France play with someone in US, connecting to different servers, waiting them to sync, how to deal with it ?
Any advice or experience to share ?
Any interesting reading or viewing ? (seen some but not very precise)
Any other solution ?
Thank you for any helping comment !
EDIT :
Only 2 players connected together, potentially from different world zone, no broadcasting needed.
We could find some front side tricks to avoid server side processing. This is a race between 2 players, so we actually just need to compare their progression, and the real winner resolution is not that important as there is no real stuff to win, this is more for fun.
If you need a server for processing the data:
I would definitely go with websockets at Compute Engine!
The Channels API is much slower, and also quite unpredictable (latency differs from message to message)! Data has to go to the Channels server, which sends it to the App Engine instance, which has to do a request back to the Channels server, which will push the message to the client. There is too much going on there if you want to keep latency down!
Here is a Channels API stress test:
http://channelapistresstest.appspot.com/
Try clicking "send 5"-button a lot, and you will see latency numbers going up to several seconds.
The Channels API is also quite expensive under heavy load (it probably does not scale well, even if Google of course can solve that with more instances).
When keeping latency down, geolocation is quite important. With a websocket server at Compute Engine, you can send your european visitors to google's european datacenter and your american visitors to the US datacenter (using the geo location headers that AppEngine will provide). You have no such control with the Channels API (or app engine, which all your messages are relayed through). Maybe Google has edge servers for the Channels API (I don't know), but if your AppEngine instance is on the other side of the planet, that does not matter.
If you do NOT need a server for processing the data:
You should establish a peer-to-peer connection with WebRTC, sending stuff directly between the users' browsers. That is was Cube Slam does. (WebRTC requires some initial handshaking ("signaling") so the two peers can find each other, and Channels API would work fine for that handshaking, that's just a couple of messages to establish the peer-to-peer connection.)
WebRTC DataChannels API will give you a nice websocket-like interface like channel.onmessage = function(e) { yadayada()... }; and channel.send("yadayada"); to send your data between the peers.
Occasionally, WebRTC is not able to make a peer-to-peer connection. Then it will fall back to a TURN server, which relays traffic between the peers. Cube Slam is using TURN servers running on ComputeEngine (in both Europe and America to keep latency down), but that is just the fallback when true peer-to-peer is not possible.
It also depends on other things like scalability.
Ingress is built on app engine and a part from the occasional cache glitch it is pretty impressive.
Remember that the channel api is using talk.Google which is the service that hangouts is built on. Scalable and real time.
Personally if your traffic levels are going to be erratic and unpredictable, go app engine. If you think it can be controlled and predictable use compute engine or something else.
Alfred's answer is the best in the frame of the question I asked.
Thank you very much !
However, I forgot to mention a few important points and the scope changed a bit :
We have very little development time (about 1 week only)
This is for a campaign that will last 3 weeks only (we'll need to keep it online a few months afterward, but this is not like we need a long-lasting architecture)
We need to make it work on the broader browser audience as possible (WebRTC only runs on Chrome & Firefox for now)
According to these points, we eventually came up to a 3rd solution :
Using a real-time PAAS.
It's way easier and faster to develop, way cheaper as we don't need a solid backend developer and system/network admin, and we can concentrate more on the project than on the infrastructure and platform.
There are a couple of services that seems good out there, already hosting MMO RPG and the kind, worldwide, with low latency, and good scaling systems.
Here is a list of providers :
https://github.com/leggetter/realtime-web-technologies-guide/blob/master/guide.md
My problem is this...
I have two sites, one acting as an "Admin" site, the other as general "User" site. I need to broadcast live audio from the "Admin" site to all clients of the "User" site. I need to do this with <1 sec of latency.
Some restrictions include:
No install on "User" machines (the idea being the whole thing sits on the web)
If there needs to be a 3rd party plugin then Silverlight is preferred*
Any help much appreciated here
*I have tried IceCast with a flash client, IIS Smooth Streaming, Internet radio, all of which give us a latency of >5 secs.
Have you tried Flash with a server like Red5? You're generally going to get subsecond latency (though not much less than that), as it's designed for realtime communications. There's a learning curve with Flex and ActionScript, but if you're at all familiar with XAML, you can pick it up from the sample apps that come with Red5 pretty quickly.
Failing that, if there aren't too many clients, you can use one of the two real-time peer-to-peer solutions out there, namely Flash over RTMFP or WebRTC over JSEP/ICE/RTP. If you can ensure that all the clients are using Chrome, then WebRTC is probably your best bet. If you can ensure that they're not using Chrome, then Flash is a good choice. The current Flash Pepper client on Chrome is buggy up the wazoo when it comes to audio processing, and no sign of a fix in sight. (It doesn't support echo cancellation, and the volume of the audio goes up and down horribly.) So if you're using Flash, steer clear of recording and broadcasting your audio on Chrome. And I wouldn't recommend either approach if you have more than half a dozen clients - the number of audio streams is gonna overwhelm your "Admin" browser pretty quickly, I think. Better to push that out to something like a Red5 server.
Silverlight is a bad choice for more reasons than I can count. I'm saying this as a guy who spent several years trying to implement a realtime communication solution on Silverlight. Don't do it.
Many banks, airlines, etc. offer the possibility to receive text message notifications by entering your mobile number. Some other sites have you enter your mobile number and wireless carrier and send messages to your free per-provider email-to-mobile number. But some people definitely do it without asking for the provider. How is this accomplished? Are there commercial services that provide this functionality?
(if this doesn't belong on SO, feel free to migrate to another SE site)
Edit: I'm primarily concerned with mobile numbers in the US, but interested in answers for other countries as well.
Yep, providers of this service are called SMS gateways, good ones should be able to set you up with an API for bi-directional use
There are a few different ways this can happen. If you have access to cellular hardware, like Multitech's MTC-H5 cellular modem or their rCell 100 cellular gateway, you're sending SMS just like a cell phone does, so this bypasses the need to know the carrier of the phone number. If the program you're using can't directly access the hardware, you could try to use a program like ours, PageGate, to receiving alerting notifications from that program to interface with the hardware to deliver the SMS.
There are also message aggregators, companies who provide a web API that allows you to send SMS. You submit the phone number with message to their API, they handle the delivery. Examples of these companies would be Infobip and Clickatell or if you're looking for something for medical markets, you could try TigerText, or for emergency management, you could try Active911 or I Am Responding.
I am just brainstorming here. Let's say I have 2 mobile devices, iPhone, Android, WinPhone7, BlackBerry, Palm, whatever. I'd like to communicate my location (latitude/longitude) from my device to another one. What is the best way of doing this?
The assumption that each device has either built-in mapping capabilities or a custom-built native app.
I've thrown around ideas like SMS/MMS or email with links embedded in them. However, they all seem to be cludgy.
Any other ideas?
A web service (or, if you must and don't care about security, Twitter) that each phone could poll for changes to location might work out. The web service would have to accept the coordinates, obviously, plus be able to be polled for updates by each device. Security is obviously a concern, but that might be another way to go provided that each device had a custom app to accomplish this task as well as web access to poll for changes.
I would go for SMS. It is guaranteed to be private, reliable, low-cost and easy on battery.
Web service is also an option, however:
privacy is a concern, as the data will leave mobile operator internal network
reliability - well, you would have to build your own messaging system and that can always fail
end-user cost - in order to access web service, end-user would have to activate a data session and constant polling can be a source of significant traffic.
development cost - you would need to develop that web service
battery life - constant polling over the data connection will have an effect on the battery.
Compare that with SMS:
privacy - SMS never leaves operator network, so it is guaranteed to be as private as your phone calls are.
reliability - SMS center is guaranteed to always be there by your mobile operator to accept your messages
end-user cost - in most networks, SMS cost is negligible
development cost - well, as SMS center is already there, you can use it without any additional effort from your side (excluding your soft that needs to send and receive those SMSs)
battery life - no adverse effect on battery
In any case, both devices still have to have a "mapping app" that you can program to interface with the SMS/web-service data.