Apache HttpAsyncClients DeadlineTimeoutException after several days of working application - apache-flink

Question

I have Kinesis Analytics application running with Flink and use external Flink Sink object to transfer data with Apache HttpAsyncClients. The below code is working fine for some time.
override def open(): Unit = {
uuid = randomUUID.toString
client = HttpAsyncClients
.custom()
.setKeepAliveStrategy(
DefaultConnectionKeepAliveStrategy.INSTANCE
)
.build()
client.start()
}
override def invoke(event: String): Unit = {
log.info(s"Received Value to sink: ${event.length} Head: ${event.take(20)}")
val request = createRequest(event)
client.execute(
request,
new FutureCallback[SimpleHttpResponse]() {
override def completed(response: SimpleHttpResponse): Unit = {
val status = StatusLine(response)
if (status.isError) {
log.error(s"Request: $request Status: $status Body: ${response.getBodyText}")
}
}
override def failed(ex: Exception): Unit = {
log.error(s"Request: $request Ex: $ex")
}
override def cancelled(): Unit = {
log.error(s"Request: $request Cancelled")
}
}
)
}
private def createRequest(event: String): SimpleHttpRequest = {
SimpleRequestBuilder
.post()
.setHttpHost(new HttpHost(URIScheme.HTTPS.id, host))
.setPath(path)
.setBody(event, ContentType.APPLICATION_JSON)
.build()
}
override def close(): Unit = {
client.close(CloseMode.GRACEFUL)
}
}
I have around 80k requests every minute and this app works fine for several days, before starting to raise DeadlineTimeoutException. Initially I thought it's related to the throughput, which my server can accept, but I don't see application errors on the backend. Instead, all requests just fail after some time.
I don't really know what is happening, because if socket connection is not closed properly and there is actual timeout I should see some requests dropping off. Instead, at the event of failure I stop to receive any traffic and zero requests successful.
I believe something has to be added for the configuration of the HTTP client, and sockets should not clog. From the documentation it seems like it shouldn't happen, socket connection and leased connection should be terminated. But at this point I have no idea what has to be changed.