We need to develop a multi-player game with real-time performance.
This needs to be working worldwide (servers in America, Europe, Asia), and supporting a huge traffic. Using Google Cloud services for the hosting.
We're thinking of references like Jam with Chrome, Chrome Maze or Cube Slam.
The game :
2 players challenge a race
We need to simultaneously display the progression of the 2 players
Each match could last around 30 to 45 seconds
The hosting :
We will obviously host the website on AppEngine, automagically scaling,
but are thinking about 2 solutions for the real-time servers :
Using websocket servers with Compute Engine
Like they did for Jam with Chrome, Maze, etc.
Developing our own websocket servers (technology TBD), deploying on datacenters in Europe, US, Asia, handling scaling, syncing between them, computing latency issues on servers and clients, etc.
But it's pretty technically challenging as we are very short on time, and missing an admin sys and network guy for now.
Or using Channel API
We understand that it's not a websocket platform, and real-time performances are lower.
But it would be way more simple and secure for us and the time we have.
So, we would also like to know more about that.
In any case, we think we could use some graphical tricks on front ends, to make it look like real-time, but it really depends if we have a 100~500ms or a 500ms~10s latency.
Some questions :
What would the latency range values look like for the different solutions ?
(Jam w/ Chrome got 100ms with GCE, could Channel API reach several seconds ?)
How would Channel API servers handle high traffic, how does scaling work, could the latency go very high ? (no info about that on Channel docs ?)
What if someone in France play with someone in US, connecting to different servers, waiting them to sync, how to deal with it ?
Any advice or experience to share ?
Any interesting reading or viewing ? (seen some but not very precise)
Any other solution ?
Thank you for any helping comment !
EDIT :
Only 2 players connected together, potentially from different world zone, no broadcasting needed.
We could find some front side tricks to avoid server side processing. This is a race between 2 players, so we actually just need to compare their progression, and the real winner resolution is not that important as there is no real stuff to win, this is more for fun.
If you need a server for processing the data:
I would definitely go with websockets at Compute Engine!
The Channels API is much slower, and also quite unpredictable (latency differs from message to message)! Data has to go to the Channels server, which sends it to the App Engine instance, which has to do a request back to the Channels server, which will push the message to the client. There is too much going on there if you want to keep latency down!
Here is a Channels API stress test:
http://channelapistresstest.appspot.com/
Try clicking "send 5"-button a lot, and you will see latency numbers going up to several seconds.
The Channels API is also quite expensive under heavy load (it probably does not scale well, even if Google of course can solve that with more instances).
When keeping latency down, geolocation is quite important. With a websocket server at Compute Engine, you can send your european visitors to google's european datacenter and your american visitors to the US datacenter (using the geo location headers that AppEngine will provide). You have no such control with the Channels API (or app engine, which all your messages are relayed through). Maybe Google has edge servers for the Channels API (I don't know), but if your AppEngine instance is on the other side of the planet, that does not matter.
If you do NOT need a server for processing the data:
You should establish a peer-to-peer connection with WebRTC, sending stuff directly between the users' browsers. That is was Cube Slam does. (WebRTC requires some initial handshaking ("signaling") so the two peers can find each other, and Channels API would work fine for that handshaking, that's just a couple of messages to establish the peer-to-peer connection.)
WebRTC DataChannels API will give you a nice websocket-like interface like channel.onmessage = function(e) { yadayada()... }; and channel.send("yadayada"); to send your data between the peers.
Occasionally, WebRTC is not able to make a peer-to-peer connection. Then it will fall back to a TURN server, which relays traffic between the peers. Cube Slam is using TURN servers running on ComputeEngine (in both Europe and America to keep latency down), but that is just the fallback when true peer-to-peer is not possible.
It also depends on other things like scalability.
Ingress is built on app engine and a part from the occasional cache glitch it is pretty impressive.
Remember that the channel api is using talk.Google which is the service that hangouts is built on. Scalable and real time.
Personally if your traffic levels are going to be erratic and unpredictable, go app engine. If you think it can be controlled and predictable use compute engine or something else.
Alfred's answer is the best in the frame of the question I asked.
Thank you very much !
However, I forgot to mention a few important points and the scope changed a bit :
We have very little development time (about 1 week only)
This is for a campaign that will last 3 weeks only (we'll need to keep it online a few months afterward, but this is not like we need a long-lasting architecture)
We need to make it work on the broader browser audience as possible (WebRTC only runs on Chrome & Firefox for now)
According to these points, we eventually came up to a 3rd solution :
Using a real-time PAAS.
It's way easier and faster to develop, way cheaper as we don't need a solid backend developer and system/network admin, and we can concentrate more on the project than on the infrastructure and platform.
There are a couple of services that seems good out there, already hosting MMO RPG and the kind, worldwide, with low latency, and good scaling systems.
Here is a list of providers :
https://github.com/leggetter/realtime-web-technologies-guide/blob/master/guide.md
Related
I am using google play game services – real time multiplayer API to add multiplayer feature to my mobile games. The engine I am using is Unity3D, but my question does not have to do with Unity (I believe so) so it is not important.
What I would like to know is the delay of the messages that are received over the internet to make my games smooth and synchronized.
I know that in other APIs like Photon you can easily find the delay of the message that is being received but I don’t seem to find it on google play game services API.
Is there any way to know the delay of the received messages on google play game services API?
Thank you for your time!
Determining the latency of the messages is a bit complex in the case of Google Real Time multiplayer APIs since the connections are peer to peer, so most of the data travels directly from one player to the other. (see for details: https://developers.google.com/games/services/common/concepts/realtimeMultiplayer#messaging)
The short answer is you can estimate it yourself, by adding sequence numbers to the messages, and then exchange the time difference each client experienced between the messages. I recommend measuring several messages, and sizes, and not have too much memory since conditions will change. Something like the average time between each message for 30-100 messages and then plan for the slowest link.
To make a good real-time game, you really should assume the latency is variable (sometimes it is low, others high), and it is always longer than you want :)
You might want to checkout https://gamedev.stackexchange.com/questions/58450/mobile-multiplayer-games-and-coping-with-high-latency which has a good discussion on how to handle this situation.
My problem is this...
I have two sites, one acting as an "Admin" site, the other as general "User" site. I need to broadcast live audio from the "Admin" site to all clients of the "User" site. I need to do this with <1 sec of latency.
Some restrictions include:
No install on "User" machines (the idea being the whole thing sits on the web)
If there needs to be a 3rd party plugin then Silverlight is preferred*
Any help much appreciated here
*I have tried IceCast with a flash client, IIS Smooth Streaming, Internet radio, all of which give us a latency of >5 secs.
Have you tried Flash with a server like Red5? You're generally going to get subsecond latency (though not much less than that), as it's designed for realtime communications. There's a learning curve with Flex and ActionScript, but if you're at all familiar with XAML, you can pick it up from the sample apps that come with Red5 pretty quickly.
Failing that, if there aren't too many clients, you can use one of the two real-time peer-to-peer solutions out there, namely Flash over RTMFP or WebRTC over JSEP/ICE/RTP. If you can ensure that all the clients are using Chrome, then WebRTC is probably your best bet. If you can ensure that they're not using Chrome, then Flash is a good choice. The current Flash Pepper client on Chrome is buggy up the wazoo when it comes to audio processing, and no sign of a fix in sight. (It doesn't support echo cancellation, and the volume of the audio goes up and down horribly.) So if you're using Flash, steer clear of recording and broadcasting your audio on Chrome. And I wouldn't recommend either approach if you have more than half a dozen clients - the number of audio streams is gonna overwhelm your "Admin" browser pretty quickly, I think. Better to push that out to something like a Red5 server.
Silverlight is a bad choice for more reasons than I can count. I'm saying this as a guy who spent several years trying to implement a realtime communication solution on Silverlight. Don't do it.
I am about to begin development of a web app in New Zealand for a NZ market for which scalability is a key requirement. I am contemplating using Google Apps Engine which I have used in the past for smaller projects where latency was not a big issue, because half the apps are client side Java script.
However, the new project requires fast AJAX response times. The local web-app companies charge about $175/month (much more than in the US I would imagine) for a dedicated server.
Is there likely to be a significant difference between the latency for AJAX requests if I use Google Apps Engine (hosted in the US I presume??) vs the local hosting company who host here in New Zealand? If so how big?
A service which may interest you in this context is CloudSleuth. They measure page load times from multiple locations. But select Asia/Oceania for Location. Then drill down for GAE to see page load time from various location. Unfortunately the closest will be Sydney, where page load for GAE currently is almost 20s.
From your explanation you would like to use App Engine as your backend, there should not be any latency problems other that the time your app would take to load and serve a request. But as they say, there is no better test like the one you do it yourself, so go ahead play with App Engine and see it for yourself!
Happy coding!
It's unavoidably the case that the latency for a request within New Zealand is going to be lower than the latency for a request to the US and back, all else being equal. There are several mitigating factors to consider, though:
The speed-of-light delay may not be significant for your application. The round trip time to the US and back is under 100 milliseconds; the latency generated by your app serving the request may be large enough that this is not a significant factor on the end-user latency.
Although your app is only in a single location at any one time, Google has caching frontends all around the world. Requests typically get routed to the closest one, and if your app generates cacheable responses, the frontend may be able to return a response from its cache immediately, without having to ever send the request to your app.
Some ISPs, particularly in places like NZ where international bandwidth is expensive, run transparent proxies. Likewise, so do organisations, and your browser itself has a cache. Any of these can satisfy the request in less time than a roundtrip, if the response is cacheable.
In the end, the question is whether or not the extra 100 milliseconds or so is acceptable. More often than not, the answer is yes, and it's worth the tradeoff of not having to handle machine provisioning, maintenance, etc etc yourself.
App Engine is not globally distributed.
The whole application is hosted around North America by default.
It you pay for the service you may request hosting within Europe instead, but there is no option to select any other regions (from https://developers.google.com/appengine/docs/python/gettingstartedpython27/uploading).
I am just brainstorming here. Let's say I have 2 mobile devices, iPhone, Android, WinPhone7, BlackBerry, Palm, whatever. I'd like to communicate my location (latitude/longitude) from my device to another one. What is the best way of doing this?
The assumption that each device has either built-in mapping capabilities or a custom-built native app.
I've thrown around ideas like SMS/MMS or email with links embedded in them. However, they all seem to be cludgy.
Any other ideas?
A web service (or, if you must and don't care about security, Twitter) that each phone could poll for changes to location might work out. The web service would have to accept the coordinates, obviously, plus be able to be polled for updates by each device. Security is obviously a concern, but that might be another way to go provided that each device had a custom app to accomplish this task as well as web access to poll for changes.
I would go for SMS. It is guaranteed to be private, reliable, low-cost and easy on battery.
Web service is also an option, however:
privacy is a concern, as the data will leave mobile operator internal network
reliability - well, you would have to build your own messaging system and that can always fail
end-user cost - in order to access web service, end-user would have to activate a data session and constant polling can be a source of significant traffic.
development cost - you would need to develop that web service
battery life - constant polling over the data connection will have an effect on the battery.
Compare that with SMS:
privacy - SMS never leaves operator network, so it is guaranteed to be as private as your phone calls are.
reliability - SMS center is guaranteed to always be there by your mobile operator to accept your messages
end-user cost - in most networks, SMS cost is negligible
development cost - well, as SMS center is already there, you can use it without any additional effort from your side (excluding your soft that needs to send and receive those SMSs)
battery life - no adverse effect on battery
In any case, both devices still have to have a "mapping app" that you can program to interface with the SMS/web-service data.
Have you ever encountered something so easy to develop but stopped a while to think of server requirements for your project ? It is my case.
I want to compete with a gaming site, they have multiplayer Flash games like poker, rummy, backgammon, and other card games, 8 games in total. For each game they have rooms and tables.
I'll use Silverlight with Sockets. I already managed to develop the policy server, the Socket Server app using WinForms, the Client Socket app in Silverlight. I own a VPS for tests, so there is no problem in developing what I want, the problem is How to calculate server requirements, RAM, bandwidth, internet speed based on the following requirements:
Server should support 24.000 users / day or 1000 users / hour
Each game room should have it's own tables where users can play
Users should not lose scores and game speed should be fast in general
I just wonder how to handle the following situation: if 1000 users are connected through Socket connection to a room full of tables and one user leave a table, all 1000 users must be updated and UI should reflect the changes. Let's say that I'll update the clients by sending a small Message of 100 bytes to each user, this will eat 100 bytes * 1000 users = 100 kb, and this just for 1 UI change, for 1 Game and for 1 Room, not counting all my other games and rooms. Also 1000 iterations that sends bytes to clients should be very time consuming.I am a developer, but not experienced in those situations. Please advice. Numbers will be great.
Until you've built -- and optimized -- your actual applications, you cannot predict much about the hardware required for some level of performance.
You have to finish the apps first. Then you can measure their performance under load. Then you can decide how much to spend on what levels of performance.
The best answer I can offer you is to run stress tests and see how much load a single server can support. While running those tests, monitor memory, IO, CPU and disk activity (if relevant) to understand which resource is running out first.
We deploy our applications on Amazon's EC2 cloud infrastructure. That lets us easily (within minutes) add or remove capacity as needed. Perhaps it's worth considering for your situation.
Always follow these two rules
“The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet.” - Michael A. Jackson
First of all you should think more about how and when to send what information to which clients. Not every client needs to be informed about every table change.
That there are only so much informations that a client needs, and you need to decide when/how it will be transmitted. Also you should pack the informations into meaningfull packets. Whats happening at a table is only interesting for that table.
Also you need to profile your application to make sure you know what ressources it consumes. Cardgames should not eat up so much ressources. But the important point is to FIRST build it, and when you HAVE a bottleneck, then try to fix it.
It's very difficult to guess at these things at this point.
From a pragmatic standpoint, you may eventually want to look into a) a cloud-hosting type service for better bandwidth price-scaling for you, or b) a very experienced full-service hosting company that can help you calculate your needs based on prior experience.
Disclaimer: I work for Rackspace Hosting which provides both of the above.