I have an app that, when activated, uploads location data. Currently it sends the data to the server via REST, however I would like to save on server costs and send the data via iot-core.
Previously, I would queue location updates, and only send them about once every few minutes, this way the phone would only turn on its data broadcasting once every few minutes and not keep it constantly, and save on battery life.
Is there a way to enable similar battery saving when uploading to AWS iot-core? I haven't run tests, but I assume that constantly sending messages via mqtt, websockets, or http is just as battery draining as regular rest messages.
This is somewhat related to Aws IoT Message Delivery.
Related
I was implementing WebSockets just for practice and I encountered an architectural problem.
It's nice to have WebSockets, but I cannot figure out a simple scalable scenario.
Possible Scenario:
Browser users start some computationally difficult task over the frontend. It goes over the API server, API puts the task to a queue, some other GPU server with celery pulls the task and starts working on it. Somewhere on the way, possibly, there is a database saving a state. So I would say API and celery server writes in the DB under particular task information about what's going on.
Now the important part. There is a WebSocket server connected to the browser client. It would be great that WebSockets are simplex and only sends messages to browser clients about the progress of the task (status, progress bar % and etc). The WebSocket is clever and doesn't need periodical polling, but manages to send data to the browser client based on events that are triggered (by API and celery). Obviously, the WebSocket server needs to listen to this task state (Redis or something, certainly not something at the same place as is WebSocket server). This means that in the WebSocket loop there must be a listener for this state. But this ends up back to WebSocket server polling this redis or something for seeing the state of the task -> this is certainly connection killer in case of a lot of users as there will be a lot of WebSocket connections polling same database.
The question is then: How to solve this in terms of architecture(no polling, WebSockets sends messages only on the state change of some value in some DB)?
I'd propose that celery server also sends a task information to some queue. The WebSocket server would have to have a code responsible for reading from that queue and distributing that task information to its clients (WebSocket connections) that listen for that particular task information.
We are making a web application in Go with a MySQL database. Our users are allowed to only have one active client at a time. Much like Spotify allows you to only listen to music on one device at a time. To do this I made a map with as key the user ids and a reference to their active websocket connection as a value. Based on the websocket id that the client has to send in the header of the request we can identify weather the request comes from their active session.
My question is if it's a good practice to store data (in this case the map with the user ids and websockets) in a global space or is it better to store it in the database.
We don't expect to reach over 10000 simultaneously active clients. Average is probably gonna be around 1000.
If you only run one instance of the websocket server storing it in memory should be sufficient. Because if it for some reason goes down/restarts then all the connections will be lost and all the clients will have to create them again (and hence the list of connection will once again be populated by all the clients who want to use the service).
However, if you plan on scaling it horizontally so you have multiple websocket services behind a load balancer, then the connections may need to be stored in a database of some sort. And not because it necessarily needs to be more persistant but because you need to be able to check the request against all the services connections.
It is also possible to have a separate service which handles the incoming request and asks all the websocket services if any of them have the connection specified in the request. This could be done if you add a pub/sub queue and every websocket service subscribes to channels for all its websocket ids and the service that receives the request then publishes the websocket id, and the websocket services can then send back replies on a separate channel if they have that connection. You must decide how to handle if no one is responding (no websocket service has the websocket id). Either the channel does not exist, or you expect the answer within a specific time. Or you could publish the question on a general topic and expect all the websocket services to reply (yes or no).
And regarding whether you need to scale it I guess depends mostly on the underlying server you're running the service on. If I understand it correctly the websocket service will basically not do anything except from keeping track of its connections (you should add some ping pong to discover if connections are lost). Then your limitation should mainly be on how many file descriptors your system can handle at once. If that limit is much larger than your expected maximum number of users, then running only one server and storing everything in memory might be an OK solution!
Finally, if you're in the business of having a websocket open for all users, why not do all the "other" communication over that websocket connection instead of having them send HTTP requests with their websocket id? Perhaps HTTP fits better for your use case but could be something to think about :)
I have an IoT device that only sends data to the cloud (Google cloud function) infrequently. The data includes a time stamp. Once I turn off or I loose internet connection for the IoT device, I can't send a shut down notice to the cloud.
I would like to send a notification after not receiving any data for something like 10 minutes. Can my cloud function have a re-settable trigger that would send me a notification if it doesn't get reset by my IoT device checking in within that time? How do I create this delay?
I would prefer not to pay for the idle time.
Not knowing Cloud Functions well, my initial thought would be to use Task Queues on the App Engine.
On each incoming request from your device, you could enqueue a task with an eta of X minutes. When the task runs, it would check to see if any data has been written in the past X minutes. If not it would send a notification, and potentially queue up a fresh task to check again.
https://cloud.google.com/appengine/docs/standard/python/taskqueue/
My assumption here is that you can access the data written by your cloud function from an App Engine application.
We have an application which is experiencing much higher outgoing bandwidth than we would expect given the loads. We have over 10 GB/day of outgoing bandwidth with essentially 0 visitors/day on the front end and a bunch of back end processing (using backend servers and the task queue). We also use memcache.
Google says they bill as follows:
Outgoing Bandwidth (billable)
The amount of data sent by the application in response to requests.
This includes:
data served in response to both secure requests and non-secure requests by application servers, static file servers, or the Blobstore
data sent in email messages
data sent over XMPP or the Channel API
data in outgoing HTTP requests sent by the URL fetch service.
We are not serving static files (it only has a rest api), don't use the blob store, don't send emails, don't use XMPP. We do use the URL fetch service, but only with GET requests. I find it hard to believe that 6000 GET requests would amount to 10 GBs of data.
Does anyone know how I can track down the details of what goes into our outgoing bandwidth usage?
To get an idea of when this bandwidth is being consumed, on the appengine dashboard you can change the charts context to: Traffic (Bytes/Second)
Also, within the dashboard I would open the Quota Details page and give it a quick once over and see if you can isolate which service is consuming the bandwidth.
On a side note, have you reviewed tasks in flight to see if perhaps something is stuck in a queue?
How is the best way to handle messages through a server to multiple devices?
Scenario
It will be an app capable of running on multiple mobile platforms including online in a web browser. A type of instant messenger. The messages will be directed through a server to another mobile device.
The back-end structure/concept must be basically the same as WhatsApp. Sending messages to one-another like that.
What I think
Have the device send it to the web-server.
Server saves it in a queue table in a database.
When receiver device checks for new message (every second) it finds it in the queue.
Remove it from queue and put message in history table.
Final
What would be a efficient way to structure/handle such an app to get similar results as WhatsApp?
You may want to push messages instead of pull them every second. This has two big advantages:
Less bandwidth usage.
You can skip the database part if the sender and the receiver are both connected when the message is sent. Only queue the messages in the database if the receiver isn't connected.
So it's a huge performance boost if you use push.
If you have a web app using JavaScript you can use a JSON stream or, for new browsers, JavaScript WebSokets.