Closing Google Cloud PubSub client with future running - google-cloud-pubsub

Question

I'm using Cloud Pubsub and for one of our systems I'm starting to get "Too many files open". lsof shows a tons of requests to Google Cloud, which I'm pretty sure are pubsub.
Googling led me to https://github.com/googleapis/google-cloud-python/issues/5523 which indicates I need to close the transport explicitly.
The problem is that I'm using a helper python package (which is invoked by about 50-100 other services) to publish my messages, which looks roughly like this:
def pubsub_callback(future):
message_id = future.result()
LOGGER.info("Successfully published %s", message_id)
def send_oneoff_pubsub_message(self, client=None):
if not client:
client = self.get_client('pubsubpub') # Creates a pubsub publisher client
future = client.publish({...})
try:
future.exception(timeout=10)
except Exception as exc:
print("error")
future.add_done_callback(pubsub_callback)
Now in many places we're slowly refactoring to explicitly create a client outside of the function (so we're not creating too many clients). However I would still like to refactor this to close the client once the message is published.
The linked issue recommends client.api.transport._channel.close() after you'r finished with the client. However, in this case I'm only finished with it after the pubsub_callback has been triggered.
I'm not seeing any way to get the client from the future, and the callback add_done_callback doesn't (right so) doesn't allow to send arguments.
Any creative solutions?
I need to bite the bullet anyway and refactor the heavy pubsub clients, but it's not always clear cut.
Update:
Looking at the code, it appears as if this would successfully close the client after the future:
def send_oneoff_pubsub_message(self, client=None):
if not client:
client = self.get_client('pubsubpub') # Creates a pubsub publisher client
future = client.publish({...})
try:
future.exception(timeout=10)
except Exception as exc:
print("error")
future.add_done_callback(pubsub_callback)
future.result(timeout=10)
client.api.transport._channel.close()
Any downside of such asn approach? except that the function blocks until published (which is OK for me)

Answer 1

Your code does a mix of asynchronous operation and synchronous operation. After you call publish, you wait up to 10 seconds for an exception on the future. Within 10 seconds in most cases, publish will have completed anyway, so at that point, you might as well call future.result synchronously and not even bother with the add_done_callback:
def pubsub_callback(future):
try:
message_id = future.result()
LOGGER.info("Successfully published %s", message_id)
except Exception as exc:
print("error")
def send_oneoff_pubsub_message(self, client=None):
if not client:
client_created = True
client = self.get_client('pubsubpub') # Creates a pubsub publisher client
future = client.publish({...})
pubsub_callback(future)
if client_created:
client.api.transport._channel.close()
If you want to do things asynchronously, you could use functools.partial:
from functools import partial
def pubsub_callback(client, future):
try:
message_id = future.result()
LOGGER.info("Successfully published %s", message_id)
except Exception as exc:
print("error")
if client:
client.api.transport._channel.close()
def send_oneoff_pubsub_message(self, client=None):
if not client:
client_created = True
client = self.get_client('pubsubpub') # Creates a pubsub publisher client
if client_created:
callback = partial(pubsub_callback, client)
else:
callback = partial(pubsub_callback, None)
future = client.publish({...})
future.add_done_callback(callback)
Either way should allow you to close the client at the desired point.