flink / spark stream - track user inactivity - apache-flink

Question

I am new to Flink and have a use case I do not know how to approach.
I have events coming
{
"id" : "AAA",
"event" : "someEvent",
"eventTime" : "2019/09/14 14:04:25:235"
}
I want to create a table (in elastic / oracle) that tracks user inactivity.
id || lastEvent || lastEventTime || inactivityTime
My final goal is to alert if some group of users are in active more then X minutes.
This table should be updated every 1 minute.
I do not have prior knowledge of all my id's. new ids can come at any time..
I thought maybe just use simple process function to emit event if present or else emit timestamp (that will update the inactivity column).
Questions
Regarding my solution - I still need to have another piece of code that check if event is null or not and update accordingly. If null --> update inactivity. else update lastEvent.
Can / should this code by in the same flink/spark job?
How do I deal with new ids?
Also, how can this use case can be dealt in spark structured stream?
input
.keyBy("id")
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.process(new MyProcessWindowFunction());
public class MyProcessWindowFunction
extends ProcessWindowFunction<Tuple2<String, Long>, Tuple2<Long, Object>> {
#Override
public void process(String key, Context context, Iterable<Tuple2<String, Long>> input, Collector<Tuple2<Long, Object>> out) {
Object obj = null;
while(input.iterator().hasNext()){
obj = input.iterator().next();
}
if (obj!=null){
out.collect(Tuple2.of(context.timestamp(), obj));
} else {
out.collect(Tuple2.of(context.timestamp(), null));
}
}

Answer 1

I would use a KeyedProcessFunction instead of the Windowing API for these requirements. [1] The stream is keyed by id.
KeyedProcessFunction#process is invoked for each record of the stream, you can keep state and schedule timers. You could schedule a timer every minute, and for each od store the last event seen in state. When the timer fires, you either emit the event and clear the state.
Personally, I would only store the last event seen in the database and calculate the inactivity time when querying the database. This way you can clear state after each emission and the possibly unbounded key space does not result in every growing managed state in Flink.
Hope this helps.
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html