I was looking at source code of ProcessingTimeoutTrigger which suppose to wrap any Trigger adding trigger-on-timeout functionality. I wonder how will it work with nestedTrigger that register, for example, its own Processing time timers.
According to implementation:
#Override
public TriggerResult onProcessingTime(long timestamp, W window, TriggerContext ctx) throws Exception {
TriggerResult triggerResult = this.nestedTrigger.onProcessingTime(timestamp, window, ctx);
if (shouldClearOnTimeout) {
this.clear(window, ctx);
}
return triggerResult.isPurge() ? TriggerResult.FIRE_AND_PURGE : TriggerResult.FIRE;
}
It will trigger WindowFunction no matter on nestedTrigger invocation result.
I wonder why? If this timer was registered with nestedTrigger and appears prior to the one registered with ProcessingTimeoutTrigger, why are we firing?
Same question goes to onEventTime implementation. ProcessingTimeoutTrigger does not register any eventTime timers. Which means that onEventTime will be called for a timer registered with nestedTrigger. Why, in this case, we are always firing?
Shouldn't we somehow check if current fired timer is actually the one registered by ProcessingTimeoutTrigger? What am I missing here?
I hadn't noticed the existence of ProcessingTimeoutTrigger before, so this is just my take based on reading the code and tests and FLINK-17058 (the ticket that added this feature).
testWindowPurgingWhenInnerTriggerIsPurging tests the case that I would be worried about with a nested ProcessingTimeTrigger, which is to test that the wrapping timeout trigger is purged when the nested trigger is purged. That way if the nested trigger fires and is purged before the outer one, the outer one will be purged. And if the inner trigger fires without purging, then the outer one should remain in place.
Looks to me like onEventTime is assuming that if the nested trigger registered an event time timer, then when that timer fires the window should FIRE. That feels a bit sloppy, but in practice is very unlikely to be wrong.
Related
In Flink job I have a KeyedProcessFunction.
I have implemented a watermark strategy
val wmStrategy: WatermarkStrategy<MyInput> =
WatermarkStrategy.forMonotonousTimestamps<MyInput>()
.withTimestampAssigner { event: MyInput, _: Long -> event.getTimestampEvent() }
and then i apply it to my source data:
mysource.assignTimestampsAndWatermarks(wmStrategy)
When processElement is called a timer may be registered ctx.timerService().registerEventTimeTimer(timerWakeUpInstant.toEpochMilli()) and after that the ValueState is updated. Update is successful.
The next time processElement is called, valueState.value() returns null instead of the last updated value.
No clear() is called explicitly on the value state.
The timer is never triggered.
At the moment, I'm testing in a 'clean' environment, reading from a text file with data referring to only a key, and with parallelism = 1 running into my IDE.
Can you help me? Why the state is nullified? And why timer is not triggered?
I have tried myself: OnTimer is not called until the Function that has registered the timer receives a message that advences the watermark.
With event-time timers, the onTimer(...) method is called when the current watermark is advanced up to or beyond the timestamp of the time
The "current" watermark actually refers to the operator, and not the job. This was misleading for me, as i thought it was centralized.
Looking at some code sample in the documentation we can find a useful comment that may give us a hint:
//trigger event time timers by advancing the event time of the operator with a watermark
My Flink job has to compute a certain aggregation after each working shift. Shifts are configurable and look something like:
1st shift: 00:00am - 06:00am
2nd shift: 06:00am - 12:00pm
3rd shift: 12:00pm - 18:00pm
Shifts are the same every day for operational purposes, there is no distinction between days of the week/year. The shifts configuration can vary over time and can be non-monotonous, so this leaves out of the table a trivial EventTime window like:
TumblingEventTimeWindows.of(Time.of(6, HOURS)) as some of the shifts might be shrunk or spanned overtime, or a couple hours break in between might be inserted...
I have come up with something based on a GlobalWindow and a custom Trigger:
LinkedList<Shift> shifts;
datastream.windowAll(GlobalWindows.create())
.trigger(ShiftTrigger.create(shifts))
.aggregate(myAggregateFunction)
where in my custom trigger I attempt to discern if an incoming event passes the end time of the on-going working shift, and fire the window for the shift:
#Override
public TriggerResult onElement(T element, long timestamp, GlobalWindow window, TriggerContext ctx) throws Exception {
// compute the end time of the on-going shift
final Instant currentShiftEnd = ...
// fire window for the shift if the event passes the end line
if (ShiftPredicate.of(currentShiftEnd).test(element)) {
return TriggerResult.FIRE_AND_PURGE;
}
return TriggerResult.CONTINUE;
}
Omitting the code for state management and some memoization optimizations, this seems to be working fine in a streaming use case: the first event coming in after a shift endtime, triggers the firing and the aggregation for the last shift.
However the job can be run bounded for date parameters (eg: for reprocessing past periods), or be shutdown prematurely for a set of expected reasons. When this sort of thing happens, I observe that the last window is not fired/flushed,
ie: the last shift of the day ends at midnight, and right over should
start the 1st shift of the next day. An event comes at 23:59pm and the
shift is about to end. However, the job is just running for the day of
today, and at 00:00 it finishes. Since no new element arrived to the
custom trigger passing the line to trigger the window firing, the
aggregation for the last shift is not calculated, however, some
partial results are still expected, even if nothing is happening in
the next shift or the job terminates in the middle of the on-going
shift.
I've read that the reason for this is:
Flink guarantees removal only for time-based windows and not for other
types, e.g. global windows (see Window Assigners)
I have taken a look inside the org.apache.flink.streaming.api.windowing package to look for something like a TumblingEventTimeWindows or DynamicEventTimeSessionWindows that I could use or extend with an end hour of the day, so that I can rely on the default event-time trigger of these firing when the watermark of the element passes the window limit, but I'm not sure how to do it. Intuitively I'd wish for something like:
shifts.forEach(shift -> {
datastream.windowAll(EventTimeWindow.fromTo(DAILY, shift.startTime, shift.endTime))
.aggregate(myAggregateFunction);
});
I know for use cases of arbitrary complexity, what some people do is ditching the Windows API in detriment of low-level process functions, where they "manually" compute the window by holding elements as managed state of the operator, while at given rules or conditions they fit and extract results from a defined aggregate function or accumulator. Also in a process function, is possible to pin point any pending calculations by tapping into the onClose hook.
Would there be a way to get this concept of recurrent event time windows for certain hours of a day every day by extending any of the objects in the Windows API?
If I understand correctly, there are two separate questions/issues here to resolve:
How to handle not having uniform window boundaries.
How to terminate the job without losing the results of the last window.
For (1), your approach of using GlobalWindows with a custom ShiftTrigger is one way to go. If you'd like to explore an alternative that uses a process function, I've written an example that you will find in the Flink docs.
For a more fluent API, you could create a custom WindowAssigner, which could then leverage the built-in EventTimeTrigger as its default trigger. To do this, you'll need to implement the WindowAssigner interface.
For (2), so long as you are relying on event time processing, the last set of windows won't be triggered unless a Watermark large enough to close them arrives before the job is terminated. This normally requires that you have an event whose timestamp is sufficiently after the window's end that a Watermark large enough to trigger the window is created (and that the job stays running long enough for that to happen).
However, when Flink is aware that a streaming job is coming to a natural end, it will automatically inject a Watermark with its timestamp set to MAX_WATERMARK, which has the effect of triggering all event time timers, and closing all event time windows. This happens automatically for any bounded sources. With Kafka (for example), you can also arrange for this by having your deserializer return true from isEndOfStream.
Another way to handle this is to avoid canceling such jobs when they are done, but to instead use ./bin/flink stop --drain [-p savepointPath] <jobID> to cleanly stop the job (with a savepoint), while draining all remaining window results (which it does by injecting one last big watermark (MAX_WATERMARK)).
I've added an event handler for the LoadedContent event. I'm a bit surprised that the event seems to fire multiple times for a page during a single load. Why does that happen?
protected void Application_Start()
{
AreaRegistration.RegisterAllAreas();
ServiceLocator.Current.GetInstance<IContentEvents>().LoadedContent += this.EPiServerApplication_LoadedContent;
}
EDIT
I'm using EPiserver 8.
With "a singe load" I mean going from DOPE mode to edit mode for a page that is a child but has no children. Last time I counted the event fired 17 times for that page.
The LoadedContent event is fired every time Get/TryGet on the IContentLoader (or IContentRepository) is called. This happens regardless if the data is loaded through from the cache or from the database.
As these APIs are used by many separate code branches, especially in edit mode, this event will be triggered multiple times just as you have found. Normally this should not be anything that you need to be worried about unless of course you are doing something resource intensive in your event handler.
Is it normal behavior that a SQLDependency onchange event is not fired if its thread is too busy?
private void NotificationOnChanged(...)
{
// get database results
// simulate long process
Thread.Sleep(10000);
}
During the sleep i add a new row and the notification is lost after the sleep expires.
Must I spawn a new single thread to do the work and use a flag to detect if new notifications arrived to restart the it?
This behavior is an artifact of the implementation on the ADO.net notification listener. See http://rusanu.com/2008/01/04/sqldependencyonchange-callback-timing/.
The SqlDependency internal thread that posts the WAITFOR(RECEIVE) is not going to post another one until the callback returns. So you have to do as little processing as possible and return in the OnChange event. Definetely nothing blocking.
Alternatively you can use the lower level SqlNotificationRequest that lets you manage everyting, including the notification handling. But you'll have to manage everything. See http://technet.microsoft.com/en-us/library/ms190270(v=sql.105).aspx
I know this question has been asked before, but I feel it wasn't asked correctly.
I have an intensive operation and I'd like the UI to remain responsive. I've read a few posts that say Background worker is the best way to go, however, I think this assumes you have the source code to the intensive task.
I have a library that I have no source code for, the only way I can check on the progress is to attach to events that get fired and get information that way.
The example I saw on the MSDN site assumed one would have the source.
I know how to get progress (which is a percentage value) by attaching to events, but how do I get that value back to the UI?
The following answer is based on my gut feeling and have not actually done it a test with third party libs.
Call your third party lib code as usual you call in a simple background (not BackGroundWorker) thread.
Attach the library components' events to normal event handlers in your code (meant to update UI).
In the event handler code should look like this:
private void EventHandler(object sender, DirtyEventArgs e)
{
if (myControl.InvokeRequired)
myControl.Invoke(new MethodInvoker(MethodToUpdateUI), e);
else
MethodToUpdateUI(e);
}
private void MethodToUpdateUI(object obj)
{
// Update UI
}
Attach to the progress events in the third party component and call ReportProgress on the BackgroundWorker. Have your UI attach to the BackgroundWorker.ProgressChanged event to update the UI.
You can have the desired effect by using a second thread and a thread safe queue.
You can create a second thread that will listen for the events.
When a new event happens it pushes the event information to a queue (thread safe- synchronized).
Using a Timer (Windows.Forms.Timer) that will check that queue every x time and in case new events exist can update the UI.
Because the timer runs in the main thread it can safely update the UI and if you make it light weight it will not block it long enough to be noticed.
A similar discussion is on this Q. If you can't or don't want to use the BackgroundWorker for whatever reason you can use your own thread and marshal events from it back onto your UI thread.
private void Initialise() {
MyLibClass myLibClass = new MyLibClass();
myLibClass.SomeEvent += SomeEventHandler;
ThreadPool.QueueUserWorkItem(myLibClass.StartLongTask);
}
private void SomeEventHandler(EventArgs e) {
if (this.Dispatcher.Thread != Thread.CurrentThread) {
this.Dispatcher.Invoke(delegate { DoStuffOnUIThread(e); });
}
else {
DoStuffOnUIThread(e);
}
}