I am doing a project in apache flink where I need to call multiple APIs so as to achieve my goal. The result of each API is required for the next API to work. Also as I am doing it on a KeyedStream, the same flow will be applicable to multiple data at once.
Below dig. can explain the scenario
/------API1---API2----
KeyedStream ----|------API1---API2----
\------API1---API2----
As I am doing all this, I am getting an exception saying "Buffer pool destroyed" after the job runs for sometime. Is it something related to API call, do I need to make use of Asynchronous function?? Please suggest. Thanks in advance.
a few things that are typically needed to help answer questions about Flink...
What version are you running?
How are you running it (from IDE, YARN cluster, stand-alone, etc)?
What's the complete stack trace for the exception?
(often) Can you share your code?
But at a high level, the "buffer pool destroyed" message you mentioned is not the root cause of failover, it's just a byproduct of Flink trying to kill off the workflow after an error has happened. So you need to dig deeper in the logs (typically Task Manager logs are where you'd look first).
Related
I've been trying to restart my Apache Flink from a previous checkpoint without much luck. I've uploaded the code to GitHub, here's the main class:
https://github.com/edu05/wordcount/blob/restart/src/main/java/edu/streaming/AppWithKafka.java
It's a simple word count program, only I'd like the program to continue with the counts it had already calculated after a restart.
I've read the docs and tried a few things but there must be something stupid missing, could someone please help?
Also: The end goal is to produce the output of the wordcount program into a compacted kafka topic, how would I go about loading the state of the app by first consuming the compacted topic, which in this case serves as both the output and the checkpointing mechanism of the program?
Many thanks
Flink's checkpoints are for automatic restarts after failures. If you want to do a manual restart, then either use a savepoint, or an externalized checkpoint.
If you've already tried this and are still having trouble, please provide more details about what you tried.
I'm currently using the RabbitMQ (3.6.2-1) on Ubuntu(16.04) in production. Producers publish messages and consumers consume messages and everything works correctly but sometimes RabbitMQ doesn't release memory and it touchs max memory and producers can not publish message into empty queues so I have to restart the service.
It's a bug or something else?
Update :
It is from the management plugin, so you can solve this issue by one of these solutions :
1.Update your RabbitMQ version (3.6.15 is stable)
2.Restarting statistics database periodically (in crontab hourly) https://www.rabbitmq.com/management.html#stats-db
3.set rates_mode to none in your rabbitmq.config file (it's not a good idea because in this case you can not see message rates)
You should check the number of the messages you have inside you queues.
RabbitMQ by default keeps the messages in memory to be fast.
if you have to handle lot of messages you can use the lazy queue:
https://www.rabbitmq.com/lazy-queues.html
With lazy queue you can handle milions of messages wihtout impact too much the node memory.
Or it could be a mamagement memory problem, see:
http://rabbitmq.com/management.html#stats-db
in your canse you can run:
rabbitmqctl eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup, rabbit_mgmt_sup), rabbit_mgmt_sup_sup:start_child().'
to reset the stats and free the memory.
You could call it periodically
Note:
There are different ways to reset the stats, it depends from the rabbitmq version, here all the detail
I have recently changed to use custom Go runtime on GAE, and noticed many errors like this from logs:
internal.flushLog: Flush RPC: Call error 3: invalid security ticket: 6c8027dc99b3ed3e
internal.flushLog: Flush RPC: Canceled: (timeout)
The server is still running well, but I have no idea about that error, as well as why it happens.
I'm using a custom Go runtime by using Dockerfile, and App Engine Release is 1.9.37.
Any help to clarify the error would be highly appreciated. Thanks.
This is a known issue with the Go runtime on App Engine Flexible. It tends to happen when a line is logged right before the end of a request/response.
What happens is that when the line is logged it is actually put in a list of log lines to be batched together and sent to the application server as an RPC at periodic intervals. The security ticket is canceled at the end of a request/response which sometimes can happen before the log lines have been flushed. It's harmless, except that you may lose a log line or two. :\
We're actively working on fixing it.
I have task queue setup for Google App Engine in Java. It works for a long time already, but I just notice URI error in admin dashboard.
A problem was encountered with the process that handled this request,
causing it to exit. This is likely to cause a new process to be used
for the next request to your application. (Error code 202)
What caused this error?
I'm seeing this as well. Little progress so far, but I did find the following, which may help:
Forum thread: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/jufkxPik1Js
Issues: https://code.google.com/p/googleappengine/issues/detail?id=8560
I bet that the instance is running out of memory. Are you using appstats? It can consume a large amount of memory.
I have a request in AppEngine that takes a little while to complete (many seconds). Is there a way to detect whether the user or some network problem has already aborted the request? This would allow me to save myself the server-load of continuing the result generation, which won't go anywhere anyways.
I tried the following in Dev-Mode, but neither worked (haven't checked yet whether it behaves differently in production mode):
Checking whether resp.getOutputStream completes without throwing an IOException
Checking whether there was an Interrupt sent to the servlet thread
Thanks, Markus
PS: I am really specifically interested in this question, not in ways to restructure my app to make the request faster or prevent aborts or other things.
I don't know if that is possible at all on the App Engine, app engine doesn't allow in progress request. The response is sent to the client after that the handler/servlat has returned.
No, there is no way to detect this from inside the app. I wouldn't worry about it.
Way late but this may be useful. In Golang you can detect interrupts using the Context package.
Here is a useful video of Francesc Campoy explaining it:
https://www.youtube.com/watch?v=LSzR0VEraWw