I am doing minute level partitioning of data with Flink S3 sink. I am noticing spike in job backpressure duration due to this minute level partioning. Is there any Flink s3 configs which we can tune to improve this performance. With Kinesis sink, we did increase the queue limit of kinesis sink in past to fix backpressure issue there. Is there any similar queue/buffer size configs with S3 sink which we can tune?
StreamingFileSink.forBulkFormat(
s3SinkPath,
new ParquetWriterFactory<>(builder))
.withBucketAssigner(new CustomDateTimeBucketAssigner())
.withBucketCheckInterval(60000)
.build();
Related
I use datastream connector KafkaSource and HbaseSinkFunction, consume data from kafka and write it to hbase.
I enable the checkpoint like this：
env.enableCheckpointing(3000,CheckpointingMode.EXACTLY_ONCE);
The data in kafka has already be successfully written to hbase，but checkpoints status on ui page is still “in progress” and has not changed.
Why does this happen and how to deal with it？
Flink version：1.13.3,
Hbase version：1.3.1,
Kafka version：0.10.2
Maybe you can post the complete checkpoint configuration parameters.
In addition, you can adjust the checkpoint interval and observe it.
The current checkpoint interval is 3s, which is relatively short.
so i tried look for it in the Flink documentation but couldn`t find any answer.
I have a Flink app "Kafka Source->Operations->Window->Kafka Sink".
Checkpoint Configuration is:
10 seconds interval
around 30MB of state size
each checkpoint take less than 1second+-
EXCACLY_ONCE on kafka producer configuration.
When i try to submit the job with checkpoint enabled on topic with 30+ partitions, the checkpoint success , but when i run the same code (with the same resources and parallelisem) on topic (the sink topic) with 2 partitions for example, it keeping failing (after reach the timeout).
So i try to look for it, but couldn`t find a good explanation, is the number of kafka sink partitions is related to checkpoint? if it does, then how?
In a Flink streaming application that is ingesting messages from Kafka,
1) How do I disable auto-committing?
2) How do I manually commit from Flink after successfully processing a message?
Thanks.
By default Flink commits offsets on checkpoints. You can disable it as follows:
val consumer = new FlinkKafkaConsumer011[T](...)
c.setCommitOffsetsOnCheckpoints(false)
If you don't have checkpoints enabled see here
Why would you do that though? Flink's checkpointing mechanism is there to solve this problem for you. Flink won't commit offsets in the presence of failures. If you throw an exception at some point downstream of the Kafka consumer Flink will attempt to restart the stream from previous successful checkpoint. If the error persists then Flink will repeatedly restart for the configured number of times before failing the stream.
This means that is unlikely you will lose messages due to Flink committing offsets of messages your code hasn't successfully processed.
I have a Flink job with a sink that is writing the data into MongoDB. The sink is an implementation of RichSinkFunction.
Externalized checkpointing enabled. The interval is 5000 mills and scheme is EXACTLY_ONCE.
Flink version 1.3,
Kafka (source topic) 0.9.0
I can't upgrade to the TwoPhaseCommitSink of Flink 1.4.
I have few doubts
At which point of time does the sink acknowledges the checkpoint barrier, at the start of the invoke function or when invoke completed? Means it waits for persisting (saving in MongoDB) response before acknowledging the barrier?
If committing checkpoint is done by an asynchronous thread, how can Flink guarantee exactly once in case of job failure? What if data is saved by the sink to MongoDB but the checkpoint is not committed? I think this will end up duplicate data on restart.
When I cancel a job from the Flink dashboard, will Flink complete the async checkpoint threads to complete or it's a hard kill -9 call?
First of all, Flink can only guarantee end-to-end exactly-once consistency if the sources and sinks support this. If you are using Flink's Kafka consumer, Flink can guarantee that the internal state of the application is exactly-once consistent. To achieve full end-to-end exactly-once consistency, the sink needs properly support this as well. You should check the implementation of the MongoDB sink if it is working correctly.
Checkpoint barriers are send a regular messages over the data transport channels, i.e., a barrier for checkpoint n separates the stream into records that go into checkpoint n and n + 1. A sink operator will process a barrier between two invoke() calls and trigger the state backend to perform a checkpoint. It is then up to the state backend, whether and how it can perform the checkpoint asynchronously. Once the call to trigger the checkpoint returns, the sink can continue processing. The sink operator will report to the JobManager that it completed checkpointing its state once it is notified by the state backend. An overall checkpoint completes when all operators successfully reported that they completed their checkpoints.
This blog post discusses end-to-end exactly-once processing and the requirements for sink operators in more detail.
I am running a streaming flink job which consumes the streaming data from kafka and do some process over the data in flink map function and write the data to the Azure data lake and the elastic search. For map function I used a parallelism of one because I need to process the incoming data one by one over the list of data I maintain as a global variable. Now when I run the job as the flink starts to get the streaming data from kafka it's backpressure becomes high in the map function. Is there any settings or configurations I could do to avoid the backpressure in flink?
Backpressure on a given operator indicates that the next operator is consuming elements slowly. From your description it would seem that one of the sinks is performing poorly. Consider scaling up the sink, commenting-out a sink for troubleshooting purposes, and/or investigating whether you're hitting an Azure rate limit.