I am using the Apache Flink processElement1, processElement2 and onTimer streaming design pattern for implementing a timeout use case. I observed that the throughput of the system decreased by orders of magnitude when I included the timeout functionality.
Any hint on the internal implementation of the onTimer in Flink: is it one thread per key stream (unlikely), or a pool/single execution threads that continuously polls buffered callbacks and pick up the timed-out callbacks for execution.
To the best of my knowledge, Flink is based on the actor model and reactive patterns (AKKA) which encourages the judicious usage of few non-blocking threads, and hence one thread per key stream for onTimer or any other pattern is typically not used!
There are two kinds of timers in Flink, event time and processing time timers. The implementations are completely different, but in neither case should you see a significant performance impact. Something else must be going on. Can you share small, reproducible example, or at least show us more of what’s going on and how you are taking the measurements?
Related
I listen to mysql binlog through flink, then drop it to rabbitmq queue, consume rabbitmq messages in flink, set parallelism to 1 for sequential consumption of messages, but this will cause flink task oom, is there any way to support multiple parallelism and consume sequentially? Please advise, thanks!
According to your description of the problem, It seems like you want to use multiple event sources and process them sequentially.
But it depends on what order that sequence is in.
You may check the concept of time semantics in flink.
If you can define event time for each event sent from multiple parallel sources, you can use Event Time Semantics together with AssignedWatermark.
So that when flink received them, it knows to process them in event time order regardless of the time flink receive them ( which is processing time).
Keywords are: Event Time (which is the default) and Processing Time
We are building a stream processing pipeline to process/ingest Kafka messages. And we are using Flink v1.12.2. While defining a source watermark strategy, in the official documentation, I came across two out-of-the-box watermark strategies; forBoundedOutOfOrderness and forMonotonousTimestamps. I did go through javadoc, but did not fully understand when and why you should use one strategy over the other. Timestamps are based on event-time. Thanks.
You should use forMonotonousTimestamps if the timestamps are never out-of-order, or if you are willing for all out-of-order events to be considered late. On the other hand, if out-of-order timestamps are normal for your application, then you should use forBoundedOutOfOrderness.
With Kafka, if you are having the kafka source operator apply the watermark strategy (recommended), then it will apply the strategy to each partition separately. In that case, each instance of the Kafka source will produce watermarks that are the minimum of the per-partition watermarks (for the partitions handled by that instance). In this case, you can use forMonotonousTimestamps if the timestamps are in-order within each partition (which will be the case, for example, if you consuming from a producer that is using log-append timestamps).
You want to use forMonotonousTimestamps whenever possible, since it minimizes latency and simplifies things.
everyone.
Please help me.
I write apache flink streraming job, which reads json messages from apache kafka (500-1000 messages in seconds), deserialize them in POJO and performs some operations (filter-keyby-process-sink). I used RocksDB state backend with ExactlyOnce semantic. But I do not understand which checkpointing interval I need set?
Some forums peoples write mostly 1000 or 5000 ms.
I tried to set interval 10ms, 100ms, 500ms, 1000ms, 5000ms. I have not noticed any differences.
Two factors argue in favor of a reasonably small checkpoint interval:
(1) If you are using a sink that does two-phase transactional commits, such as Kafka or the StreamingFileSink, then those transactions will only be committed during checkpointing. Thus any downstream consumers of the output of your job will experience latency that is governed by the checkpoint interval.
Note that you will not experience this delay with Kafka unless you have taken all of the steps required to have exactly-once semantics, end-to-end. This means that you must set Semantic.EXACTLY_ONCE in the Kafka producer, and set the isolation.level in downstream consumers to read_committed. And if you are doing this, you should also increase transaction.max.timeout.ms beyond the default (which is 15 minutes). See the docs for more.
(2) If your job fails and needs to recover from a checkpoint, the inputs will be rewound to the offsets recorded in the checkpoint, and processing will resume from there. If the checkpoint interval is very long (e.g., 30 minutes), then your job may take quite a while to catch back up to the point where it is once again processing events in near real-time (assuming you are processing live data).
On the other hand, checkpointing does add some overhead, so doing it more often than necessary has an impact on performance.
In addition to the points described by #David, my suggestion is also to use the following function to configure the checkpoint time:
StreamExecutionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(milliseconds)
This way, you guarantee that your job will be able to make some progress in case the state gets bigger than planned or the storage where the checkpoints are made is slow.
I recommend reading the Flink documentation on Tuning Checkpointing to better understand these scenarios.
i am just beginning learning apache flink and meet the folling problem:
How can i suspend a flink job and then resume it ?
does flink support suspend a job using command line ?
Yes, you certainly can do this with Flink. You want to read about savepoints, which can be triggered from the command line or from the REST API.
Updated
Normally the goal of a stream processor is to do continuous, immediate processing of new elements as they become available. If you want to suspend processing, then I guess this might be with the goal of ignoring the source(s) for a while and dropping the arriving events, or with a desire to conserve computing resources for a time and to later resume without losing any input.
RichCoFlatmap and CoProcessFunction are building blocks you might find useful. You could setup a control stream connected to a socket (for example), and when you want to "suspend" the primary stream, send an event that causes the primary stream to either start dropping its input, or do a blocking read, or sleep, for example.
Or you might think about adding your own layer of abstraction on top of jobs, and cope with the fact that the jobids will change. Note that jobs can have names that remain unchanged across savepoints/restarts.
I have a gen_server process that maintains a pool, for each incoming request, I need to examining the pool to see if there is a match for this incoming request, if there is one, the matched one is removed from the pool and replies are made to both requests; if there is none, the new request is put to the pool for later examination.
The biz logic requires that, if a request, R, sits in the pool for T seconds without been matched, I need to make a reply to R saying something like "I cannot find a match for you".
Ideally, I want to do this with timers, specifically, for each incoming request, if there is no match, put it to the pool as before, but also start a timer to tell the gen_server to remove it if time is up, of course, if it is matched later, the timer should be cancelled.
My concern is that, If there are lots unmatched requests in the pool, then there would be lots of running timers, will this (too many timers) becomes a problem?
There were done big improvements in timers implementation in R18.
Besides the API changes and time warp modes a lot of
scalability and performance improvements regarding time
management has been made internally in the runtime system.
Examples of such improvements are scheduler specific timer
wheels, scheduler specific BIF timer management, parallel
retrieval of monotonic time and system time on systems with
primitives that are not buggy.
scheduler specific timer wheels is exactly what is interesting in your scenario. I doubt you would come around better performant solution of your problem in Erlang or any other language/environment. So your solution should be OK when you are using R18 or newer.