In Search of an Understandable Consensus Algorithm (Extended Version):
Upon election: send initial empty AppendEntries RPCs (heartbeat) to
each server; repeat during idle periods to prevent election timeouts
Why sends empty AppendEntries RPCs but not carry logs? When does leader sends AppendEntries RPCS with logs?
AppendEntries RPCs with no entries are known as heartbeats, which the leader sends periodically to the followers to tell them that he's alive. The followers each have a different timer that resets everytime the they receive an heartbeat. If this timer reaches zero, it means that the leader may have crashed and the follower whose timer reached zero switches to the Candidate state to try to be the new Leader.
The leader sends AppendEntries RPC's with logs when the leader receives any requests from the clients and stores them in its own log. It then sends the new entries to its followers using AppendEntries.
Here you can visualize raft in action, they all start in a follower state and since there is no leader to send heartbeats, one of them switches to candidate state and becomes the Leader.
There could be another dedicated RPC to be the heartbeat but Raft uses AppendEntries because it does the job. The heartbeat is also used to update tell the followers who the current leader is and which term they are on.
Related
My Google-fu has failed me, Stack Overflow Gods...
I have a very basic bot running on 4 of my servers that handles scheduling and, as long as the bot is online, it works marvelously, but as soon as it goes offline for whatever reason, it stops listening to any of the messages already in the channel and will only listen to newly sent messages.
E.G. Let's say that every time the bot is online is starts a new session and when it goes offline, that session ends. So, if my bot is online for the first time, it's in Session 1 and listens to all messages sent during Session 1, but as soon as it goes offline, Session 1 ends. When it comes back online, it starts Session 2 and stops listening to messages that were sent during Session 1 and only listens to messages that are sent during Session 2.
How can I get the bot to listen to all the messages in one channel? (How can I get it to listen to all messages from all "Sessions")
Thank you kindly
I have a requirement to track the undelivered messages in PubSub. But when a subscriber to a PubSub Pull subscription is unavailable after the retention period the message will be lost forever from the subscription. It is not been captured by the dead letter topic created for the subscription.
It seems the PubSub only sends a message to a dead letter topic if the number of retries exceeds and the acknowledgement not been received by the subscriber.
Is there a way to push a message to dead letter topic before the message get lost for forever?
There is no way to send messages to a dead letter topic before the message is deleted due to the retention period expiring, no. The goal of the dead letter topic is to capture messages that are causing issues for subscribers and potentially preventing the processing of other messages, e.g., if the subscribers are crashing due to an unexpected message. The way this state is detected is via the retry count.
I found this docs: https://cloud.google.com/nodejs/docs/reference/pubsub/0.19.x/v1.SubscriberClient#acknowledge
If a message is acknowledged failed, will it be put back to the message queue and wait for redelivering later? Or this message is lost?
Acknowledgements in Google Cloud Pub/Sub are best effort and the service as a whole has at-least-once delivery of messages. What this means is that if an acknowledgement fails (and even in rare cases, if you get back that an ack succeeded), the messages will be redelivered to a subscriber. A message is only deleted from Pub/Sub if the service successfully receives and processes an ack for the messages messageRetentionDuration passes, which defaults to seven days.
We recently integrated google pubsub into our app, and some of our long running tasks are now under problem, as they take more than 1 minute sometimes. We have configured our subscriber's ack deadline to 600 seconds, yet, anything that is taking more than 600ms, is being retried by pubsub.
this is our config:
gcloud pubsub subscriptions describe name
ackDeadlineSeconds: 600
expirationPolicy: {}
messageRetentionDuration: 604800s
Not sure what is the issue. Most of our tasks will get repeated because of this
Pub/Sub has a built in At-least-once delivery system which will retry messages that were not acknowledged. In this case, after 600s have passed, the message you first sent becomes unacknowledged, thus Pub/Sub retries the message. It will keep retrying it for 600s until it reaches the messageRetentionDuration or you acknowledge it.
Keep in mind that it's specified in the documentation that your subscriber should be idempotent. So, making your code be able to handle multiple messages should be the best approach to this issue.
You could also decrease the messageRetentionDuration to 600s(it's minimum) so anything that passes the 10 min mark will not be retried.
Also, it is stated in the FAQs that:
Why are there too many duplicate messages?
Cloud Pub/Sub guarantees at-least-once message delivery, which means
that occasional duplicates are to be expected. However, a high rate of
duplicates may indicate that the client is not acknowledging messages
within the configured ack_deadline_seconds, and Cloud Pub/Sub is
retrying the message delivery. This can be observed in the monitoring
metrics.
pubsub.googleapis.com/subscription/pull_ack_message_operation_count
for pull subscriptions, and
pubsub.googleapis.com/subscription/push_request_count for push
subscriptions. Look for elevated expired or webhook_timeout values in
the /response_code. This is particularly likely if there are many
small messages, since Cloud Pub/Sub may batch messages internally and
a partially acknowledged batch will be fully redelivered.
Another possibility is that the subscriber is not acknowledging some
messages because the code path processing those specific messages
fails, and the Acknowledge call is never made; or the push endpoint
never responds or responds with an error.
I need this ability to send push notifications for an action in a mobile app but wait for the user to undo the action until say 10 seconds.
Is it possible to delay the processing of a message published in a topic by 10 seconds ? And then (sometimes, if user does undo) delete the message before 10 seconds, if it doesn't need to be processed ?
Depends on if you write the subscribers as well or not:
You have control over your subscriber's code:
In your PubSub messages add a timestamp for when you want that message to be processed.
In your clients (subscribers), have logic to acknowledge the message only if the timestamp to process the message is reached.
PubSub will retry delivering the message until it's acknowledged (or 10 days)
If you don't have control over your subscriber you can have a my-topic and my-delayed-topic. Folks can publish to the former topic and that topic will have only one subscriber which you will implement:
Publish message as before to my-topic.
You will have a subscriber for that topic that can do the same throttling as shown above.
If the time for that message has reached your handler will publish/relay that message to my-delayed-topic.
You can also implement the logic above with task-queue+pubsub-topic instead of pubsub-topic+pubsub-topic.
If architecturally possible at all, you could use Cloud Tasks. This API has the following features that might suit your usecase:
You can schedule the delivery of the message (task)
You can delete the tasks from the queue (before they are executed)
Assuming that your client has a storage for some task Ids:
Create a task with schedule_time set to 10s in the future.
Store the task name in memory (you can either assign a name to the task at creation time, or use the automatically generated ID returned from the create response).
If user undid the job, then call DeleteTask.
Just wanted to share that I noticed Pub/Sub supports retry policies 1 that are GA as of 2020-06-16 2.
If the acknowledgement deadline expires or a subscriber responds with a negative acknowledgement, Pub/Sub can send the message again using exponential backoff.
If the retry policy isn't set, Pub/Sub resends the message as soon as the acknowledgement deadline expires or a subscriber responds with a negative acknowledgement.
If the maximum backoff duration is set, the default minimum backoff duration is 10 seconds. If the minimum backoff duration is set, the default maximum backoff duration is 600 seconds.
The longest backoff duration that you can specify is 600 seconds.