Joining stream events only once over a sliding window - apache-flink

Question

I'm trying to evaluate if apache flink would be usable for a distributed event driven system (only-once). The use case is that a user is signed up for a subscription and wants to change for a different subscription.
There are two separate processes that run asynchronously when the users clicks the submit button. One process cancels the existing subscription whilst another signs up for the new subscription. Once these two events have been triggered, the email notification is sent.
I've managed to create two streams in apache flink using the RabbitMQ connector. When I try joining these streams together using a sliding window, the events are duplicated for each slide in the window. I've tried setting a ValueStateDescriptor on the joined streams but this doesn't seem to expire after the window has passed.
Additionally I need to detect the events that have not been paired in the streams and send this event to a different RabbitMQ sink to cope with situations whereby the event has not be fired due to the process not completing successfully.
Do you have any tips/ideas on how I could achieve the above functionality?
final StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
environment.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
final RMQConnectionConfig rmqConnectionConfig = new RMQConnectionConfig.Builder()
.setHost("localhost")
.setPort(5672)
.setVirtualHost("/")
.setUserName("admin")
.setPassword("password")
.build();
final DataStream<String> cancellation = environment
.addSource(new RMQSource<>(rmqConnectionConfig, "scratchpad-cancellation", true, new SimpleStringSchema()))
.setParallelism(1);
final DataStream<String> subscription = environment
.addSource(new RMQSource<>(rmqConnectionConfig, "scratchpad-subscription", true, new SimpleStringSchema()))
.setParallelism(1);
cancellation
.join(subscription)
.where(value -> value).equalTo(value -> value)
.window(SlidingEventTimeWindows.of(Time.minutes(5), Time.seconds(15)))
.apply((left, right) -> left)
.keyBy(value -> value)
.process(new ProcessFunction<String, String>() {
private ValueStateDescriptor<Boolean> descriptor = new ValueStateDescriptor<>("seen", Boolean.class);
private ValueState<Boolean> state;
#Override
public void open(Configuration parameters) {
state = this.getRuntimeContext().getState(descriptor);
}
#Override
public void processElement(String value, Context ctx, Collector<String> out) throws Exception {
if (BooleanUtils.isNotTrue(state.value())) {
state.update(true);
out.collect(value);
ctx.timerService().registerEventTimeTimer(ctx.timestamp() + TimeUnit.MILLISECONDS.convert(10, TimeUnit.MINUTES));
}
}
#Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) {
state.clear();
}
})
.print()
.setParallelism(1);
environment.execute();

Answer 1

If you have duplicated values on the output of your window, you can add a reduce function into another window after your already defined slide window and that should be engough in most cases. But i think that should be a better solution than this, but we need an example fo your code to work on improvements.
On the other side, if you need to detect non paired events, i think that you need to work with the CoGroup operator, instead of use joins.