I found this docs: https://cloud.google.com/nodejs/docs/reference/pubsub/0.19.x/v1.SubscriberClient#acknowledge
If a message is acknowledged failed, will it be put back to the message queue and wait for redelivering later? Or this message is lost?
Acknowledgements in Google Cloud Pub/Sub are best effort and the service as a whole has at-least-once delivery of messages. What this means is that if an acknowledgement fails (and even in rare cases, if you get back that an ack succeeded), the messages will be redelivered to a subscriber. A message is only deleted from Pub/Sub if the service successfully receives and processes an ack for the messages messageRetentionDuration passes, which defaults to seven days.
Related
I was building a service that runs on Cloud Run that is triggered by PubSub through EventArc.
'PubSub' guarantees delivery at least one time and it would retry for every acknowledgement deadline. This deadline is set in the queue subscription details.
We could send an acknowledgement back at two points when a service receives a pub-sub request (which is received as a POST request in the service).
At the beginning of the request as soon as the request was received. The service would then continue to process the request at its own pace. However, this article points out that
When an application running on Cloud Run finishes handling a request, the container instance's access to CPU will be disabled or severely limited. Therefore, you should not start background threads or routines that run outside the scope of the request handlers.
So sending a response at the beginning may not be an option
After the request has been processed by the service. So this would mean that, depending on what the service would do, we cannot always predict how long it would take to process the request. Hence we cannot set the Acknowledgement deadline correctly, resulting in PubSub retries and duplicate requests.
So what is the best practice here? Is there a better way to handle this?
Best practice is generally to ack a message once the processing is complete. In addition to the Cloud Run limitation you linked, consider that if the endpoint acked a message immediately upon receipt and then an error occurred in processing it, your application could lose that message.
To minimize duplicates, you can set the ack deadline to an upper bound of the processing time. (If your endpoint ends up processing messages faster than this, the ack deadline won’t rate-limit incoming messages.) If the 600s deadline is not sufficient, you could consider writing the message to some persistent storage and then acking it. Then, a separate worker can asynchronously process the messages from persistent storage.
Since you are concerned that you might not be able to set the correct "Acknowledgement Deadline", you can use modify_ack_deadline() in your code where you can dynamically extend your deadline if the process is still running. You can refer to this document for sample code implementations.
Be wary that the maximum acknowledgement deadline is 600 seconds. Just make sure that your processing in cloud run does not exceed the said limit.
Acknowledgements do not apply to Cloud Run, because acks are for "pull subscriptions" where a process is continuously pulling the Cloud PubSub API.
To get events from PubSub into Cloud Run, you use "push subscriptions" where PubSub makes an HTTP request to Cloud Run, and waits for it to finish.
In this push scenario, PubSub already knows it made you a request (you received the event) so it does not need an acknowledgement about the receipt of the message. However, if your request sends a faulty response code (e.g. http 500) PubSub will make another request to retry (and this is configurable on the Push Subscription itself).
I have a requirement to track the undelivered messages in PubSub. But when a subscriber to a PubSub Pull subscription is unavailable after the retention period the message will be lost forever from the subscription. It is not been captured by the dead letter topic created for the subscription.
It seems the PubSub only sends a message to a dead letter topic if the number of retries exceeds and the acknowledgement not been received by the subscriber.
Is there a way to push a message to dead letter topic before the message get lost for forever?
There is no way to send messages to a dead letter topic before the message is deleted due to the retention period expiring, no. The goal of the dead letter topic is to capture messages that are causing issues for subscribers and potentially preventing the processing of other messages, e.g., if the subscribers are crashing due to an unexpected message. The way this state is detected is via the retry count.
We recently integrated google pubsub into our app, and some of our long running tasks are now under problem, as they take more than 1 minute sometimes. We have configured our subscriber's ack deadline to 600 seconds, yet, anything that is taking more than 600ms, is being retried by pubsub.
this is our config:
gcloud pubsub subscriptions describe name
ackDeadlineSeconds: 600
expirationPolicy: {}
messageRetentionDuration: 604800s
Not sure what is the issue. Most of our tasks will get repeated because of this
Pub/Sub has a built in At-least-once delivery system which will retry messages that were not acknowledged. In this case, after 600s have passed, the message you first sent becomes unacknowledged, thus Pub/Sub retries the message. It will keep retrying it for 600s until it reaches the messageRetentionDuration or you acknowledge it.
Keep in mind that it's specified in the documentation that your subscriber should be idempotent. So, making your code be able to handle multiple messages should be the best approach to this issue.
You could also decrease the messageRetentionDuration to 600s(it's minimum) so anything that passes the 10 min mark will not be retried.
Also, it is stated in the FAQs that:
Why are there too many duplicate messages?
Cloud Pub/Sub guarantees at-least-once message delivery, which means
that occasional duplicates are to be expected. However, a high rate of
duplicates may indicate that the client is not acknowledging messages
within the configured ack_deadline_seconds, and Cloud Pub/Sub is
retrying the message delivery. This can be observed in the monitoring
metrics.
pubsub.googleapis.com/subscription/pull_ack_message_operation_count
for pull subscriptions, and
pubsub.googleapis.com/subscription/push_request_count for push
subscriptions. Look for elevated expired or webhook_timeout values in
the /response_code. This is particularly likely if there are many
small messages, since Cloud Pub/Sub may batch messages internally and
a partially acknowledged batch will be fully redelivered.
Another possibility is that the subscriber is not acknowledging some
messages because the code path processing those specific messages
fails, and the Acknowledge call is never made; or the push endpoint
never responds or responds with an error.
In a Pub/Sub 'push' model the docs say this:
If the push endpoint returns an error code, messages are retried for up to 7 days with an exponential backoff policy (capped at 10 seconds).
Is there a way to decide what to do with the message after the retry period ? i.e. send it to some error queue etc ?
The seven-day retry period represents the maximum amount of time unacknowledged messages are retrained in Cloud Pub/Sub to be delivered to subscribers. After the seven days pass, a message is automatically deleted from Cloud Pub/Sub and no longer delivered. The system does not currently support performing any actions on these deleted messages such as sending them to an error queue.
I am implementing an Apple notification push service and am processing an internal queue.
I would like to know if PushSharp notifications always raise the "NotificationSent" or "NotificationFailed" events (one or the other - at least one of them).
Also if the notification does successfully send, doe the NotificationSent event always fire for each successful notification sent or can there be circumstances where this does not occur even though it sent successfully?
I've been working with PushSharp (latest lib v2.0.4) and all my messages have either triggered NotificationFailed() or NotificationSent(). But this is no guarantee that the messages will reach the end user (device). Apple or Google are sending an ACK that their servers have received the notification successfully (or not) from my push service. Then it's in their hands to send on to the final device.
Furthermore, how are you closing your push services? If you call StopAllServices(true), the service will wait for all ACKS to be received before shutting down. If called as StopAllServices(false) then it shuts down not waiting for acks and therefore the events NotificationFailed() or NotificationSent() will not be triggered.
Hope this helps.