I want to send notifications after every 30 seconds. I have scheduled my task using TimerTask but it is not working. Why is it so? My Searcher is given below:
#Override
public Result search(Query query, Execution execution) {
// pass it down the chain to get a result
Result result = execution.search(query);
execution.fill(result);
//the Date and time at which you want to execute
DateFormat dateFormatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Date date = null;
try {
date = dateFormatter.parse("2018-10-17 16:20:00");
} catch (ParseException e) {
e.printStackTrace();
}
//Now create the time and schedule it
Timer timer = new Timer();
assert date != null;
timer.schedule(new TimerTask() {
#Override
public void run() {
sendNotification();
}
}, date, 30000);
// return the result up the chain
return result;
}
The search() method is invoked on a per request basis and you are creating a new Timer instance per request which is garbage collected away by the Java VM some time after the search method returns as there are no references to it.
Related
I am newbie to Flink i am trying a POC in which if no event is received in x amount of time greater than time specified in within time period in CEP
public class MyCEPApplication {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment streamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment();
streamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("group.id", "consumer_group");
FlinkKafkaConsumer<String> inputSignal = new FlinkKafkaConsumer<>("consumer_topic",
new SimpleStringSchema(),
properties);
DataStream<String> inputSignalKafka = streamExecutionEnvironment.addSource(inputSignal);
DataStream<MyEvent> eventDataStream = inputSignalKafka.map(new MapFunction<String, MyEvent>() {
#Override
public MyEvent map(String value) throws Exception {
ItmAtomicEvent MyEvent = new ItmAtomicEvent();
MyEvent.setJsonObject(new JSONObject(value));
MyEvent.setAttribute("time",System.currentTimeMillis());
return MyEvent;
}
}).assignTimestampsAndWatermarks(new AssignerWithPunctuatedWatermarks<MyEvent>() {
#Override
public long extractTimestamp(MyEvent event, long currentTimestamp) {
System.out.println("TIMESTAMP: " +(long)event.getAttribute("time").get());
System.out.println("Time Difference: " +((long)(event.getAttribute("time").get()) - timeDifference));
timeDifference = (long)(event.getAttribute("time").get());
return (long)event.getAttribute("time").get();
}
#Override
public Watermark checkAndGetNextWatermark(MyEvent lastElement, long extractedTimestamp) {
return new Watermark(extractedTimestamp);
}
});
eventDataStream.print("Source=======================================>");
Pattern<MyEvent, ?> pattern =
Pattern.<MyEvent>begin("start")
.where(new SimpleCondition<MyEvent>() {
#Override
public boolean filter(MyEvent event) {
return event.equals("Event1");
}
})
.next("end")
.where(new SimpleCondition<MyEvent>() {
#Override
public boolean filter(MyEvent event) {
return event.equals("Event2");
}
}).within(Time.seconds(10));
PatternStream<MyEvent> patternStream = cepPatternMatching.compile(pattern, eventDataStream);
OutputTag<MyEvent> timedout = new OutputTag<MyEvent>("timedout") {};
SingleOutputStreamOperator<MyEvent> result = patternStream.flatSelect(
timedout,
new PatternFlatTimeoutFunction<MyEvent,MyEvent>() {
#Override
public void timeout(Map<String, List<MyEvent>> pattern, long timeoutTimestamp, Collector<MyEvent> out) throws Exception {
if(null != pattern.get("CustomerId")){
for (MyEvent timedOutEvent :pattern.get("CustomerId")){
System.out.println("TimedOut Event : "+timedOutEvent.getField(0));
out.collect(timedOutEvent);
}
}
}
},
new PatternFlatSelectFunction<MyEvent, MyEvent>() {
#Override
public void flatSelect(Map<String, List<MyEvent>> pattern,
Collector<MyEvent> out) throws Exception {
out.collect(pattern.get("CustomerId").get(0));
}
}
);
result.print("=============CEP Pattern Detected==================");
DataStream<MyEvent> timedOut = result.getSideOutput(timedout);
timedOut.print("=============CEP TimedOut Pattern Detected==================");
streamExecutionEnvironment.execute("CEPtest");
}
}
Even if no event received after 10 seconds its printing timedout event i tried even commenting out code PatternFlatSelectFunction method, is there a way or work around to make timeout of pattern if no event received in a given x seconds. Someone asked same question i referred below solution but nothing worked for me, please help me in solving the issue
1)Apache Flink CEP how to detect if event did not occur within x seconds?
2)https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/libs/cep.html#handling-timed-out-partial-patterns
3)https://github.com/ververica/flink-training-exercises/blob/master/src/main/java/com/ververica/flinktraining/solutions/datastream_java/cep/LongRidesCEPSolution.java
Your application is using event time, so you will need to arrange for a sufficiently large Watermark to be generated despite the lack of incoming events. You could use this example if you want to artificially advance the current watermark when the source is idle.
Given that your events don't have event-time timestamps, why don't you simply use processing time instead, and thereby avoid this problem? (Note, however, the limitation mentioned in https://stackoverflow.com/a/50357721/2000823).
Can someone please help me understand when and how is the window (session) in flink happens? Or how the samples are processed?
For instance, if I have a continuous stream of events flowing in, events being request coming in an application and response provided by the application.
As part of the flink processing we need to understand how much time is taken for serving a request.
I understand that there are time tumbling windows which gets triggered every n seconds which is configured and as soon as the time lapses then all the events in that time window will be aggregated.
So for example:
Let's assume that the time window defined is 30 seconds and if an event arrives at t time and another arrives at t+30 then both will be processed, but an event arrivng at t+31 will be ignored.
Please correct if I am not right in saying the above statement.
Question on the above is: if say an event arrives at t time and another event arrives at t+3 time, will it still wait for entire 30 seconds to aggregate and finalize the results?
Now in case of session window, how does this work? If the event are being processed individually and the broker time stamp is used as session_id for the individual event at the time of deserialization, then the session window will that be created for each event? If yes then do we need to treat request and response events differently because if we don't then doesn't the response event will get its own session window?
I will try posting my example (in java) that I am playing with in short time but any inputs on the above points will be helpful!
process function
DTO's:
public class IncomingEvent{
private String id;
private String eventId;
private Date timestamp;
private String component;
//getters and setters
}
public class FinalOutPutEvent{
private String id;
private long timeTaken;
//getters and setters
}
===============================================
Deserialization of incoming events:
public class IncomingEventDeserializationScheme implements KafkaDeserializationSchema {
private ObjectMapper mapper;
public IncomingEventDeserializationScheme(ObjectMapper mapper) {
this.mapper = mapper;
}
#Override
public TypeInformation<IncomingEvent> getProducedType() {
return TypeInformation.of(IncomingEvent.class);
}
#Override
public boolean isEndOfStream(IncomingEvent nextElement) {
return false;
}
#Override
public IncomingEvent deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
if (record.value() == null) {
return null;
}
try {
IncomingEvent event = mapper.readValue(record.value(), IncomingEvent.class);
if(event != null) {
new SessionWindow(record.timestamp());
event.setOffset(record.offset());
event.setTopic(record.topic());
event.setPartition(record.partition());
event.setBrokerTimestamp(record.timestamp());
}
return event;
} catch (Exception e) {
return null;
}
}
}
===============================================
main logic
public class MyEventJob {
private static final ObjectMapper mapper = new ObjectMapper();
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
MyEventJob eventJob = new MyEventJob();
InputStream inStream = eventJob.getFileFromResources("myConfig.properties");
ParameterTool parameter = ParameterTool.fromPropertiesFile(inStream);
Properties properties = parameter.getProperties();
Integer timePeriodBetweenEvents = 120;
String outWardTopicHostedOnServer = localhost:9092";
DataStreamSource<IncomingEvent> stream = env.addSource(new FlinkKafkaConsumer<>("my-input-topic", new IncomingEventDeserializationScheme(mapper), properties));
SingleOutputStreamOperator<IncomingEvent> filteredStream = stream
.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor<IncomingEvent>() {
long eventTime;
#Override
public long extractTimestamp(IncomingEvent element, long previousElementTimestamp) {
return element.getTimestamp();
}
#Override
public Watermark getCurrentWatermark() {
return new Watermark(eventTime);
}
})
.map(e -> { e.setId(e.getEventId()); return e; });
SingleOutputStreamOperator<FinalOutPutEvent> correlatedStream = filteredStream
.keyBy(new KeySelector<IncomingEvent, String> (){
#Override
public String getKey(#Nonnull IncomingEvent input) throws Exception {
return input.getId();
}
})
.window(GlobalWindows.create()).allowedLateness(Time.seconds(defaultSliceTimePeriod))
.trigger( new Trigger<IncomingEvent, Window> (){
private final long sessionTimeOut;
public SessionTrigger(long sessionTimeOut) {
this.sessionTimeOut = sessionTimeOut;
}
#Override
public TriggerResult onElement(IncomingEvent element, long timestamp, Window window, TriggerContext ctx)
throws Exception {
ctx.registerProcessingTimeTimer(timestamp + sessionTimeOut);
return TriggerResult.CONTINUE;
}
#Override
public TriggerResult onProcessingTime(long time, Window window, TriggerContext ctx) throws Exception {
return TriggerResult.FIRE_AND_PURGE;
}
#Override
public TriggerResult onEventTime(long time, Window window, TriggerContext ctx) throws Exception {
return TriggerResult.CONTINUE;
}
#Override
public void clear(Window window, TriggerContext ctx) throws Exception {
//check the clear method implementation
}
})
.process(new ProcessWindowFunction<IncomingEvent, FinalOutPutEvent, String, SessionWindow>() {
#Override
public void process(String arg0,
ProcessWindowFunction<IncomingEvent, FinalOutPutEvent, String, SessionWindow>.Context arg1,
Iterable<IncomingEvent> input, Collector<FinalOutPutEvent> out) throws Exception {
List<IncomingEvent> eventsIn = new ArrayList<>();
input.forEach(eventsIn::add);
if(eventsIn.size() == 1) {
//Logic to handle incomplete request/response events
} else if (eventsIn.size() == 2) {
//Logic to handle the complete request/response and how much time it took
}
}
} );
FlinkKafkaProducer<FinalOutPutEvent> kafkaProducer = new FlinkKafkaProducer<>(
outWardTopicHostedOnServer, // broker list
"target-topic", // target topic
new EventSerializationScheme(mapper));
correlatedStream.addSink(kafkaProducer);
env.execute("Streaming");
}
}
Thanks
Vicky
From your description, I think you want to write a custom ProcessFunction, which is keyed by the session_id. You'll have a ValueState, where you store the timestamp for the request event. When you get the corresponding response event, you calculate the delta and emit that (with the session_id) and clear out state.
It's likely you'd also want to set a timer when you get the request event, so that if you don't get a response event in safe/long amount of time, you can emit a side output of failed requests.
So, with the default trigger, each window is finalized after it's time fully passes. Depending on whether You are using EventTime or ProcessingTime this may mean different things, but in general, Flink will always wait for the Window to be closed before it is fully processed. The event at t+31 in Your case would simply go to the other window.
As for the session windows, they are windows too, meaning that in the end they simply aggregate samples that have a difference between timestamps lower than the defined gap. Internally, this is more complicated than the normal windows, since they don't have defined starts and ends. The Session Window operator gets sample and creates a new Window for each individual sample. Then, the operator verifies, if the newly created window can be merged with already existing ones (i.e. if their timestamps are closer than the gap) and merges them. This finally results with window that has all elements with timestamps closer to each other than the defined gap.
You are making this more complicated than it needs to be. The example below will need some adjustment, but will hopefully convey the idea of how to use a KeyedProcessFunction rather than session windows.
Also, the constructor for BoundedOutOfOrdernessTimestampExtractor expects to be passed a Time maxOutOfOrderness. Not sure why you are overriding its getCurrentWatermark method with an implementation that ignores the maxOutOfOrderness.
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Event> events = ...
events
.assignTimestampsAndWatermarks(new TimestampsAndWatermarks(OUT_OF_ORDERNESS))
.keyBy(e -> e.sessionId)
.process(new RequestReponse())
...
}
public static class RequestReponse extends KeyedProcessFunction<KEY, Event, Long> {
private ValueState<Long> requestTimeState;
#Override
public void open(Configuration config) {
ValueStateDescriptor<Event> descriptor = new ValueStateDescriptor<>(
"request time", Long.class);
requestState = getRuntimeContext().getState(descriptor);
}
#Override
public void processElement(Event event, Context context, Collector<Long> out) throws Exception {
TimerService timerService = context.timerService();
Long requestedAt = requestTimeState.value();
if (requestedAt == null) {
// haven't seen the request before; save its timestamp
requestTimeState.update(event.timestamp);
timerService.registerEventTimeTimer(event.timestamp + TIMEOUT);
} else {
// this event is the response
// emit the time elapsed between request and response
out.collect(event.timestamp - requestedAt);
}
}
#Override
public void onTimer(long timestamp, OnTimerContext context, Collector<Long> out) throws Exception {
//handle incomplete request/response events
}
}
public static class TimestampsAndWatermarks extends BoundedOutOfOrdernessTimestampExtractor<Event> {
public TimestampsAndWatermarks(Time t) {
super(t);
}
#Override
public long extractTimestamp(Event event) {
return event.eventTime;
}
}
I just encountered a very strange problem that I cannot get any results from stream window join when using EventTime with timestamps and watermarks assigner.
I am using Kafka as my data stream source and tried both AscendingTimestampExtractor and custom assigner which implemented AssignerWithPeriodicWatermarks as mentioned from Flink documentation here, and as what I have tested, there is no watermark emited and no joined result generated. If I change to use ProcessingTime and TumblingProcessingTimeWindows without any timestamps assigners, then I can get correct results.
My code for custom timestamps and watermarks assigner is like:
FlinkKafkaConsumer09<String> myConsumer1 =
new FlinkKafkaConsumer09<>(myTopic1, new SimpleStringSchema(), props);
myConsumer1.assignTimestampsAndWatermarks(new MyTimestampsAndWatermarks());
FlinkKafkaConsumer09<String> myConsumer2 =
new FlinkKafkaConsumer09<>(myTopic2, new SimpleStringSchema(), props);
myConsumer2.assignTimestampsAndWatermarks(new MyTimestampsAndWatermarks());
...
public static class MyTimestampsAndWatermarks implements AssignerWithPeriodicWatermarks<String> {
private long currentMaxTimestamp;
#Override
public long extractTimestamp(String element, long previousElementTimestamp) {
long timestamp = myFunctionToGetMillisFromString(element);
currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
return timestamp;
}
#Override
public Watermark getCurrentWatermark() {
return new Watermark(currentMaxTimestamp - 1L);
}
}
...
DataStream<myPOJO1> stream1 = env.addSource(myConsumer1).map(new MyMapper1());
DataStream<myPOJO2> stream2 = env.addSource(myConsumer2).map(new MyMapper2());
stream1.join(stream2)
.where(new KeySelector1())
.equalTo(new KeySelector2())
.window(TumblingEventTimeWindows.of(Time.seconds(windowSize)))
.apply(new JoinFunction<AdClick, GameCreate, TransferResult>() {...});
And my code for AscendingTimestampExtractor is like:
FlinkKafkaConsumer09<String> myConsumer1 =
new FlinkKafkaConsumer09<>(myTopic1, new SimpleStringSchema(), props);
myConsumer1.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<String>() {
#Override
public long extractAscendingTimestamp(String element) {
return myFunctionToGetMillisFromString(element);
}
});
FlinkKafkaConsumer09<String> myConsumer2 =
new FlinkKafkaConsumer09<>(myTopic2, new SimpleStringSchema(), props);
myConsumer2.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<String>() {
#Override
public long extractAscendingTimestamp(String element) {
return myFunctionToGetMillisFromString(element);
}
});
...
DataStream<myPOJO1> stream1 = env.addSource(myConsumer1).map(new MyMapper1());
DataStream<myPOJO2> stream2 = env.addSource(myConsumer2).map(new MyMapper2());
stream1.join(stream2)
.where(new KeySelector1())
.equalTo(new KeySelector2())
.window(TumblingEventTimeWindows.of(Time.seconds(windowSize)))
.apply(new JoinFunction<AdClick, GameCreate, TransferResult>() {...});
Thanks for helps!
I had the same problem, its a rather silly error, I found the solution here:
When you write:
myConsumer1.assignTimestampsAndWatermarks(new MyTimestampsAndWatermarks());
It creates a new data stream rather than modifying that stream and you didn't store this in a variable.
So bottomline is :
Store it in a new datastream and apply join to this datastream (which will have these timestamps and watermarks assigned).
myConsumer3 = myConsumer1.assign***
myConsumer4 = myConsumer2.assign***
and use myConsumer3/myConsumer4, which will be ok
Need help a bit, please. How to make it work in the following order: check value of an i, run Timer for it, ONLY when that timer is finished, do i++? For now two timers run at the same time. Tried to declare boolean isRunning, but did not help.
for(int i=0;i<pcArray.length;i++){
if(pcArray[i]==1){
blinkGreen.start(); }
else if(pcArray[i]==2){
blinkRed.start();
}
}
First add a boolean variable in blinkGreen and also in blinkRed Timer like this,
In BlinkGreen:
boolean isRunningGreen = false;
Timer blinkGreen = new Timer();
TimerTask task = new TimerTask() {
#Override
public void run() {
// do stuff
isRunningGreen = true;//This statement must be placed at the end/bottom of this method
}
};
In BlinkRed:
boolean isRunningRed = false;
Timer blinkRed = new Timer();
TimerTask task = new TimerTask() {
#Override
public void run() {
// do stuff
isRunningRed = true;//This statement must be placed at the end/bottom of this method
}
};
Now try this:
for(int i=0;i<pcArray.length;i++){
if(pcArray[i]==1){
blinkGreen.start();
while(!isRunningGreen){}//waiting until the task is completed
} else if(pcArray[i]==2){
blinkRed.start();
while(!isRunningRed){}//waiting until the task is completed
}
}
I would like to catch an observable thats loading an add and have it show after a few seconds. Im calling this is multiple places but in one particular place i'd like it to only run after a few seconds have elapsed. Here is the method i have that returns an observable:
private Observable fullScreenAdObservable(){
// Create the interstitial.
return Observable.create(new Observable.OnSubscribe<Object>() {
#Override
public void call(Subscriber<? super Object> subscriber) {
interstitial = new InterstitialAd(main.this);
interstitial.setAdUnitId(admob_publisherID);
// Create ad request.
AdRequest adRequest = new AdRequest.Builder().build();
// Begin loading your interstitial.
interstitial.loadAd(adRequest);
interstitial.setAdListener(new AdListener() {
#Override
public void onAdLoaded() {
super.onAdLoaded();
interstitial.show();
}
});
}
});
}
then to subscribe i do this but the timer one fails:
fullScreenAdObservable().subscribe();//this works
fullScreenAdObservable().timer(0,3, TimeUnit.SECONDS).subscribe(); //this fails to react,why ?
I want the timer to run the observerable after 3 seconds but it wont, why ?
There seems to be some scheduling issue when using delay and timer. So i had to explicity tell what thread i want to subscribe on BEFORE calling the delaySubscription or timer. Also modified the subscriber to listen for onNext so i can pass the subscriber a ad. Then i actually showed the ad in the onNext (as its triggered after the delay is elapsed.
private Observable fullScreenAdObservable(){
// Create the interstitial.
return Observable.create(new Observable.OnSubscribe<Object>() {
#Override
public void call(final Subscriber<? super Object> subscriber) {
interstitial = new InterstitialAd(main.this);
interstitial.setAdUnitId(admob_publisherID);
// Create ad request.
AdRequest adRequest = new AdRequest.Builder().build();
// Begin loading your interstitial.
interstitial.loadAd(adRequest);
interstitial.setAdListener(new AdListener() {
#Override
public void onAdLoaded() {
super.onAdLoaded();
//emit a loaded ad
subscriber.onNext(interstitial);
}
});
}
});
}
//and to call it :
fullScreenAdObservable().subscribeOn(AndroidSchedulers.mainThread()).
delaySubscription(13, TimeUnit.SECONDS).subscribe(new Action1<InterstitialAd>() {
#Override
public void call(InterstitialAd ad) {
ad.show();
}
});