In my program I have enabled checkpointing,
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(5000);
but I haven't configured any StateBackend.
Where is the checkpointed state stored? Can I somehow inspect this data?
The default state backend keeps the working state on the heaps of the various task managers, and backs that up to the job manager heap. This is the so-called MemoryStateBackend.
There's no API for directly accessing the data stored in the state backend. You can simulate a task manager failure and observe that the state is restored. And you can instead trigger a savepoint if you wish to externalize the state, though there is no tooling for directly inspecting these savepoints.
It's not the answer, but small addition to the correct answer. I can't write comments cause of reputation.
If you use flink version earlier then v1.5 then default state backend will MemoryStateBeckend with asynchronous snapshots set to false. So you will use synchronous saving checkpoints every 5 seconds in your case (your pipeline will block every 5 seconds for saving checkpoint).
To avoid this use explicit constructor:
env.setStateBackend(new MemoryStateBackend(maxStateSize, true));
Since flink version v1.5.0 MemoryStateBackend uses asynchronous snapshots by default.
For more information see flink_v1.4 docs
Related
I am following an example here: https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/StatefulSequenceSource.java
I am trying to build a source using a jdbc connection which extends RichParallelFunction and implements CheckpointedFunction, as I would like to be able to save my watermark from my source tables in the case of restart.
When testing locally with docker, I can call my run() method just fine and read data from my source database, but I am not sure where the snapshotState and initializeState methods actually get called. I have logic in those methods that should be setting the value of my watermark based on first startup/recovery - I just never see that being accessed, and not sure if I should be calling the methods externally?
Thanks for the help in advance!
The methods will be called by the Flink framework when it needs to (when performing a checkpoint or a save point).
See https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/state/#using-operator-state and https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/checkpoint/CheckpointedFunction.html
We are using StreamingFileSink in Flink 1.11 (AWS KDA) to write data from Kafka to S3.
Sometimes, even after a proper stopping of the application it will fail to start with:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
By looking at the code I can see that files are moved from in-progress to pending during a checkpoint: files are synced to S3 as MPU uploads and a _tmp_ objects when upload part is too small.
However, pending files are committed during notifyCheckpointComplete, after the checkpoint is done.
streamingFileSink will fail with the error above when an MPU which it has in state does not exists in S3.
Would the following scenario be possible:
Checkpoint is taken and files are transitioned into pending state.
notifyCheckpointComplete is called and it starts to complete the MPUs
Application is suddenly killed or even just stopped as part of a shutdown.
Checkpointed state still has information about MPUs, but if you try to restore from it it's not going to find them, because they were completed outside of checkpoint and not part of the state.
Would it better to ignore missing MPUs and _tmp_ files? Or make it an option?
This way the above situation would not happen and it would allow to restore from arbitrary checkpoints/savepoints.
The Streaming File Sink in Flink has been superseded by the new File Sink implementation since Flink 1.12. This is using the Unified Sink API for both batch and streaming use cases. I don't think the problem you've listed here would occur when using this implementation.
I have a spring statemachine (version 3.0.1) with an own instance for each user. With every HTTP request the statemachine is acquired by a DefaultStateMachineService from a JpaStateMachineRepository. At the end of the HTTP request the statemachine is released again. The state machine is configured to use a StateMachineRuntimePersister.
At one special state of the statemachine there is a transition triggered by a timer:
.withExternal()
.source( "S1" )
.target( "S2" )
.timerOnce( 60000 )
I want this timer to be executed only if the statemachine is still in state S1 after one minute (another cloud instance handling a later step of this statemachine instance could have changed the state in the meantime, making the timer obsolete).
I tried to release the statemachine without stopping it (stateMachineService.releaseStateMachine( machineId, false );). Then the timer always triggers after one minute, because it does not refresh the statemachine from the database leading to an inconsistent state.
I tried to release the statemachine stopped. Then the timer is killed and will not trigger anymore.
One solution could be to switch the trigger from timerOnce to event and to use a separate mechanism (e.g. Quartz scheduler) that reads the statemachine from database, checks the state and then explicitly fires the event.
But this would make the timer feature of spring statemachine useless in a cloud environment, so I guess there should be a better solution.
What would be the best solution here?
The problem you mentioned can be solved using Distributed States.
So, Spring Statemachine uses ZooKeeper to keep states of machines consistent in the app instances.
In your case, ZooKeeper will let your state machine know if its state changes on the other app instance and behave accordingly with the timer.
It's worth mentioning that the feature is not mature as of now according to docs:
Distributed state functionality is still a preview feature and is not yet considered to be stable in this particular release. We expect this feature to mature towards its first official release.
You can find some technical details of the concept in the Distributed State Machine Technical Paper
Personally speaking, I would stick with the solution you described (triggering event fire with a scheduler) since it gives more control over the process.
I'm trying to do one action after the flink job is finished (make some change in DB). I want to do it in the same flink application with no luck.
I found that there is JobStatusListener that is notified in ExecutionGraph about changed state but I cannot find how I can get this ExecutionGraph to register my listener.
I've tried to completely replace ExecutionGraph in my project (yes, bad approach but...) but as soon as it is runtime library it is not called at all in distributed mode, only in local run.
I have next flink application in short:
DataSource.output(RichOutputFormat.class)
ExecutionEnvironment.getExecutionEnvironment().execute()
Can please anybody help?
I need to know how Apache Flink restore its state from checkpoint, because I can't see any difference between the time of start and seeing first event in operator when running pure job verses restoring from savepoint.
Does state load lazily from checkpoint/savepoint?
The keyed state interfaces are designed to make this distinction transparent. As Dawid mentioned, the state is loaded during job start. Note that what it means to load the state depends on which state backend is being used.
In the case of operator state the CheckpointedFunction interface has this method
public void initializeState(FunctionInitializationContext context)
where the context has an isRestored() method that lets you know if you are recovering from a failure. See the docs on managed operator state for more details, including an example.