Google Pub Sub Message Pull latency Issue with .Net Client - google-cloud-pubsub

Question

I am pulling messages continuously from Google PubSub. All going good, except the time taken to fetch the message is around 12 to 15 seconds and its not acceptable in our case. Following is my CallTiming settings:
public CallSettings GetPullSetting()
{
CallTiming timing = CallTiming.FromRetry(new RetrySettings(
retryBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 0, 50), new TimeSpan(0, 0, 5), 1),
timeoutBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 18, 0), new TimeSpan(0, 0, 20), 1),
totalExpiration: Google.Api.Gax.Expiration.FromTimeout(TimeSpan.FromMilliseconds(600000))));
return CallSettings.FromCallTiming(timing);
}
I trying all kind of combination to reduce this latency to max 3 seconds.
One observation is that whenever a message is pulled successfully, and in the very next iteration of the pull if there is a message on pubsub, it fetches that message immediately. That means if message is found in consecutive pull latency is very low.
But the problem is, say in one iteration I get Deadline exceeded exception since pubsub has no message. Then I push a message in pubsub for the next iteration. At this point it takes lot of time (13 to 16 seconds). So the condition to reproduce this issue is I shall have one failed attempt to pull the message.
Code pasted here:
public void PullTest()
{
var cont = true;
SubscriberSettings settings = new SubscriberSettings()
{
PullSettings = GetPullSetting()
};
SubscriberClient subscriberClient = SubscriberClient.Create(settings: settings);
var subscriberName = new SubscriptionName("project-name", "subscription-name");
while (cont)
{
try
{
PullResponse response = subscriberClient.Pull(subscriberName, returnImmediately: false, maxMessages: 1);
System.Diagnostics.Trace.WriteLine(">>>>>> " + DateTime.Now.ToString());
System.Diagnostics.Trace.WriteLine(">>>>>> " + "Job Recieved" + response.ReceivedMessages.ToList().FirstOrDefault());
subscriberClient.Acknowledge(subscriberName, new List<string>() { response.ReceivedMessages.ToList().FirstOrDefault().AckId });
}
catch (Exception ex)
{
System.Diagnostics.Trace.WriteLine(">>>>>> " + DateTime.Now.ToString());
System.Diagnostics.Trace.WriteLine(">>>>>> " + ex.Message);
}
}
}
public CallSettings GetPullSetting()
{
CallTiming timing = CallTiming.FromRetry(new RetrySettings(
retryBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 0, 50), new TimeSpan(0, 0, 5), 1),
timeoutBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 18, 0), new TimeSpan(0, 0, 20), 1),
totalExpiration: Google.Api.Gax.Expiration.FromTimeout(TimeSpan.FromMilliseconds(600000))));
return CallSettings.FromCallTiming(timing);
}

Answer 1

You have a problem with back off between consecutive pulls when timeout deadline exceeds. You are essentially looking for a long polling solution, for which you either need to reduce backoff timings to close to 0, or use a custom connection/client to retry without delay.

Answer 2

To minimize latency, you are going to want to have multiple pull requests outstanding simultaneously. Depending on your throughput requirements, dozens of outstanding requests at once may be required. If your throughput is low, you'll still want to have multiple pull requests at once (at least two or three). As soon as any one of them returns, whether it be with a deadline exceeded or with a message, start another pull request. The goal is to always have a pull requests outstanding waiting to receive messages that have been published.