I am new to MQTT and I have some questions that I hope you guys could help me with. I'm working on a school project that will require me to utilize the MQTT protocol and the program needs to be written in C. (Just some background info)
Can a MQTT client be both a publisher and a subscriber at the same time? That is, while constantly waiting to receive messages from the broker and perform resulting actions, it is also able to publish messages to a broker when needed to.
My understanding of MQTT is as such: MQTT Publisher --> MQTT Broker --> MQTT Subscriber
What exactly is the Asynchronous mode of MQTT, in idiot terms?
Thanks!
1) Yes, there is no reason a MQTT Client can not be a Published and a Subscriber, this is a normal mode for a client to work in.
2) An Asynchronous MQTT client implementation is one that does not block when carrying out network operations (sending or receiving data). This means that all the network operations take place in the background, a token is returned from any call that would normally block that can be used to check if that operation succeeded later.
1) When you say your mqtt client has subscribed to a particular topic it means that It will keep on listening to that till either it is unsubscribed or the connection is Terminated. When you say your mqtt client is publishing, it publishes the MqttMessage to the Broker and all the clients who are subscribed i.e listening will receive the message. So you just need to keep yourself subscribed and start publishing.
2)Asynchronous mode doesn't block the thread for performing any action. It just acts on the given action and returns a token which can be used to track and wait for the action to get completed. This is opposite to Synchronous mode where the Thread is blocked till the action is completed on reached the timeOut period.
Eg: When you publish in Asynchronous mode, you will be returned a IMqttDeliveryToken which can be used later to verify if the publish action was completed.
Related
I was building a service that runs on Cloud Run that is triggered by PubSub through EventArc.
'PubSub' guarantees delivery at least one time and it would retry for every acknowledgement deadline. This deadline is set in the queue subscription details.
We could send an acknowledgement back at two points when a service receives a pub-sub request (which is received as a POST request in the service).
At the beginning of the request as soon as the request was received. The service would then continue to process the request at its own pace. However, this article points out that
When an application running on Cloud Run finishes handling a request, the container instance's access to CPU will be disabled or severely limited. Therefore, you should not start background threads or routines that run outside the scope of the request handlers.
So sending a response at the beginning may not be an option
After the request has been processed by the service. So this would mean that, depending on what the service would do, we cannot always predict how long it would take to process the request. Hence we cannot set the Acknowledgement deadline correctly, resulting in PubSub retries and duplicate requests.
So what is the best practice here? Is there a better way to handle this?
Best practice is generally to ack a message once the processing is complete. In addition to the Cloud Run limitation you linked, consider that if the endpoint acked a message immediately upon receipt and then an error occurred in processing it, your application could lose that message.
To minimize duplicates, you can set the ack deadline to an upper bound of the processing time. (If your endpoint ends up processing messages faster than this, the ack deadline won’t rate-limit incoming messages.) If the 600s deadline is not sufficient, you could consider writing the message to some persistent storage and then acking it. Then, a separate worker can asynchronously process the messages from persistent storage.
Since you are concerned that you might not be able to set the correct "Acknowledgement Deadline", you can use modify_ack_deadline() in your code where you can dynamically extend your deadline if the process is still running. You can refer to this document for sample code implementations.
Be wary that the maximum acknowledgement deadline is 600 seconds. Just make sure that your processing in cloud run does not exceed the said limit.
Acknowledgements do not apply to Cloud Run, because acks are for "pull subscriptions" where a process is continuously pulling the Cloud PubSub API.
To get events from PubSub into Cloud Run, you use "push subscriptions" where PubSub makes an HTTP request to Cloud Run, and waits for it to finish.
In this push scenario, PubSub already knows it made you a request (you received the event) so it does not need an acknowledgement about the receipt of the message. However, if your request sends a faulty response code (e.g. http 500) PubSub will make another request to retry (and this is configurable on the Push Subscription itself).
I was implementing WebSockets just for practice and I encountered an architectural problem.
It's nice to have WebSockets, but I cannot figure out a simple scalable scenario.
Possible Scenario:
Browser users start some computationally difficult task over the frontend. It goes over the API server, API puts the task to a queue, some other GPU server with celery pulls the task and starts working on it. Somewhere on the way, possibly, there is a database saving a state. So I would say API and celery server writes in the DB under particular task information about what's going on.
Now the important part. There is a WebSocket server connected to the browser client. It would be great that WebSockets are simplex and only sends messages to browser clients about the progress of the task (status, progress bar % and etc). The WebSocket is clever and doesn't need periodical polling, but manages to send data to the browser client based on events that are triggered (by API and celery). Obviously, the WebSocket server needs to listen to this task state (Redis or something, certainly not something at the same place as is WebSocket server). This means that in the WebSocket loop there must be a listener for this state. But this ends up back to WebSocket server polling this redis or something for seeing the state of the task -> this is certainly connection killer in case of a lot of users as there will be a lot of WebSocket connections polling same database.
The question is then: How to solve this in terms of architecture(no polling, WebSockets sends messages only on the state change of some value in some DB)?
I'd propose that celery server also sends a task information to some queue. The WebSocket server would have to have a code responsible for reading from that queue and distributing that task information to its clients (WebSocket connections) that listen for that particular task information.
I am trying to design a Client Server kind of application in which my Server is a daemon that accepts client requests, send client's data over a serial channel to the other side(which is an MCU and its firmware will reply to the Server request over the same serial channel). My client can be a CLI application or any other system program.
My idea of design is -
Use message queues for communication between Client and Server since this is a local application and message queues are bidirectional and fast.
Implement a LIBRARY that acts as an interface between multiple clients and the server. This basically does the stuff of packetizing client data into a message(own defined protocol), create message queues, connect to server, send/receive data and then pass it to the respective client(using call backs). This library also exposes API that can be used by clients. Thus this library gives me the flexibility to add support for any new clients keeping the server program unchanged.
Server gets the data over serial from other side and passes it to the library over message queue. The library uses callbacks to send data to the client.
EDIT:
I am thinking of creating Message queues on the fly when any client requests arrive. If I do this, how does the Server daemon(which has already started at linux boot up) gets information about this message queue? Does the message queue has a name that is persistent across and used by other programs? I want to implement clients that will be blocked until it gets response from the server.
Could you guys please review this design and tell me whether my approach is correct. Please reply if you have any other recommendations.
Thanks in advance.
Let's say we have several clients connected to App Engine using Channel API. Each client sends messages, which should be propagated to other conntected clients according to some rules. The tricky part is that clients may not be to the same App Engine instance.
Is there any way to push data from one instance to the others?
(Yes, I know about Memcache, but this would require some kind of polling.)
You're asking two questions here.
a. Can you push data from one instance to another without the use of polling. The answer is generally no.
b. Can one client send messages to the server that can be propagated to other clients? Yes, and this does not require propagating messages to other server-side instances.
Consider the Channel API as a service. Clients are connected to the Channel API service; they are not connected to any particular instance. Therefore any instance can send messages to any client.
You'll need to store the Channel tokens of your clients in the datastore, in some way that's queryable to match your rules.
Your client makes an HTTP request to send a message to your server.
The handler on the server queries for channel tokens that it needs to propagate the message to (either from memcache or datastore).
The handler on the server sends messages to all the clients.
If the list of destination clients is extremely large, you might want to do steps 3/4 in a task queue where the operation can run longer.
It does not matter what instance a client is connected to, that's hidden from you by the API.
Clients can only "reply" to message via standard HTTP commands, they don't actually have any way to respond via the channel API directly.
So Client A on server A1 wants to sent a message to client B on server B1.
Client A posts to a handler. That might be instance A1 or B1. It does not matter which as the server now passes the message on to client B whatever server client B is connected to via the Channel API.
The real point is that no App Engine instance has any data at all, in general. So it does not matter which instance you connect to, it might be the 99th instance or the very first to start up. So you have to design your application so that it's irrelevant what instance is in use.
Client sends message to server via HTTP.
Server sends message to N clients via the channel API.
Channel API does not make a fixed frontend-instance-to-client connection. Any frontend instance can push message to channel if it knows the channel ID.
What you need to do is pass messages cross-channel.
User one sends message normally to server (e.g. via GET)
Server looks up channel ID of second user and pushes the message
Repeat procedure in other direction: second user to first user.
we wrote in C++ a screen sharing application based on sending screenshots.
It works by establishing a TCP connection btw the server and client, where the server forwards every new screenshot received for a user through the connection, and this is popped-up by the client.
Now, we are trying to host this on google app engine, and therefore need 'servlet'-ize and 'sandbox' the server code, so to implement this forwarding through HTTP requests.
I immagine the following:
1. Post request with the screenshot as multiple-data form (apache uploads ..).
But now the server needs to contact the specified client (who is logged in) to send it/forward the screenshot.
I'm not sure how to 'initiate' such connection from the servlet to the client. The client doesn't run any servlet environment (of course).
I know HTTP 1.1 mantains a TCP connection, but it seems gapps won't let me use it.
1 approaches that comes to mind is to send a CONTINUE 100 to every logged in user at login, and respond with the screenshot once it arrives. Upon receival the client makes another request, and so on.
an alternative (insipired from setting the refresh header for a browser) would be to have the app pool on a regular basis (every 5 secs).
You're not going to be able to do this effectively on GAE.
Problem 1: All output is buffered until your handler returns.
Problem 2: Quotas & Limits:
Some features impose limits unrelated
to quotas to protect the stability of
the system. For example, when an
application is called to serve a web
request, it must issue a response
within 30 seconds. If the application
takes too long, the process is
terminated and the server returns an
error code to the user. The request
timeout is dynamic, and may be
shortened if a request handler reaches
its timeout frequently to conserve
resources.
Comet support is on the product roadmap, but to me your app still seems like a poor fit for a GAE application.
Long Polling is the concept used for such asynchronous communications between server and client.
In Long Polling, servlet keeps a map of client and associated messages. Key of Map being client id and value being list of messages to be sent to the client. When a client opens a connection with server (sends request to a servlet), the servlet checks the Map if there are any messages to be sent to it. If found, it sends the messages to the client exits from the method. On receiving messages, the client opens a new connection to the server. If the servlet does not find any messages for given client, it waits till the Map gets updated with messages for given client.
This is a late reply, I'm aware, but I believe that Google have an answer for this requirement: the Channel API.