I'm working on an application in Windows CVI that needs to run some code for a series of time intervals set by the user via text entry boxes. The boxes include three for how long to run each process, one to show the total time the processes will take, and one to show the time remaining.
My implementation currently is to have a function with static variables to track which process is running and how long is left in the current process, then move on when that time has elapsed. This function triggers on pushing a start button. Unfortunately, the code stops on the click of the start button as it seems to be waiting for the code to finish executing before it allows any further inputs.
Is there a "right" way to do this? Maybe something with multithreading or a pre-built timer application?
Got an answer here: "You are on the right way speaking about timers: place a timer control on your panel, set it to disabled, put your code in the timer callback and run the program. When the user inputs the required time, set the ATTR_INTERVAL attribute of the timer to this value, next enable the timer with ATRT_ENABLED attribute and you'll have your timed function up and running!"
Related
My Flink job has to compute a certain aggregation after each working shift. Shifts are configurable and look something like:
1st shift: 00:00am - 06:00am
2nd shift: 06:00am - 12:00pm
3rd shift: 12:00pm - 18:00pm
Shifts are the same every day for operational purposes, there is no distinction between days of the week/year. The shifts configuration can vary over time and can be non-monotonous, so this leaves out of the table a trivial EventTime window like:
TumblingEventTimeWindows.of(Time.of(6, HOURS)) as some of the shifts might be shrunk or spanned overtime, or a couple hours break in between might be inserted...
I have come up with something based on a GlobalWindow and a custom Trigger:
LinkedList<Shift> shifts;
datastream.windowAll(GlobalWindows.create())
.trigger(ShiftTrigger.create(shifts))
.aggregate(myAggregateFunction)
where in my custom trigger I attempt to discern if an incoming event passes the end time of the on-going working shift, and fire the window for the shift:
#Override
public TriggerResult onElement(T element, long timestamp, GlobalWindow window, TriggerContext ctx) throws Exception {
// compute the end time of the on-going shift
final Instant currentShiftEnd = ...
// fire window for the shift if the event passes the end line
if (ShiftPredicate.of(currentShiftEnd).test(element)) {
return TriggerResult.FIRE_AND_PURGE;
}
return TriggerResult.CONTINUE;
}
Omitting the code for state management and some memoization optimizations, this seems to be working fine in a streaming use case: the first event coming in after a shift endtime, triggers the firing and the aggregation for the last shift.
However the job can be run bounded for date parameters (eg: for reprocessing past periods), or be shutdown prematurely for a set of expected reasons. When this sort of thing happens, I observe that the last window is not fired/flushed,
ie: the last shift of the day ends at midnight, and right over should
start the 1st shift of the next day. An event comes at 23:59pm and the
shift is about to end. However, the job is just running for the day of
today, and at 00:00 it finishes. Since no new element arrived to the
custom trigger passing the line to trigger the window firing, the
aggregation for the last shift is not calculated, however, some
partial results are still expected, even if nothing is happening in
the next shift or the job terminates in the middle of the on-going
shift.
I've read that the reason for this is:
Flink guarantees removal only for time-based windows and not for other
types, e.g. global windows (see Window Assigners)
I have taken a look inside the org.apache.flink.streaming.api.windowing package to look for something like a TumblingEventTimeWindows or DynamicEventTimeSessionWindows that I could use or extend with an end hour of the day, so that I can rely on the default event-time trigger of these firing when the watermark of the element passes the window limit, but I'm not sure how to do it. Intuitively I'd wish for something like:
shifts.forEach(shift -> {
datastream.windowAll(EventTimeWindow.fromTo(DAILY, shift.startTime, shift.endTime))
.aggregate(myAggregateFunction);
});
I know for use cases of arbitrary complexity, what some people do is ditching the Windows API in detriment of low-level process functions, where they "manually" compute the window by holding elements as managed state of the operator, while at given rules or conditions they fit and extract results from a defined aggregate function or accumulator. Also in a process function, is possible to pin point any pending calculations by tapping into the onClose hook.
Would there be a way to get this concept of recurrent event time windows for certain hours of a day every day by extending any of the objects in the Windows API?
If I understand correctly, there are two separate questions/issues here to resolve:
How to handle not having uniform window boundaries.
How to terminate the job without losing the results of the last window.
For (1), your approach of using GlobalWindows with a custom ShiftTrigger is one way to go. If you'd like to explore an alternative that uses a process function, I've written an example that you will find in the Flink docs.
For a more fluent API, you could create a custom WindowAssigner, which could then leverage the built-in EventTimeTrigger as its default trigger. To do this, you'll need to implement the WindowAssigner interface.
For (2), so long as you are relying on event time processing, the last set of windows won't be triggered unless a Watermark large enough to close them arrives before the job is terminated. This normally requires that you have an event whose timestamp is sufficiently after the window's end that a Watermark large enough to trigger the window is created (and that the job stays running long enough for that to happen).
However, when Flink is aware that a streaming job is coming to a natural end, it will automatically inject a Watermark with its timestamp set to MAX_WATERMARK, which has the effect of triggering all event time timers, and closing all event time windows. This happens automatically for any bounded sources. With Kafka (for example), you can also arrange for this by having your deserializer return true from isEndOfStream.
Another way to handle this is to avoid canceling such jobs when they are done, but to instead use ./bin/flink stop --drain [-p savepointPath] <jobID> to cleanly stop the job (with a savepoint), while draining all remaining window results (which it does by injecting one last big watermark (MAX_WATERMARK)).
My use-case is quite simple I receive events that contain "event timestamp", and want them to be aggregated based on event time. and the output is a periodical processing time tumbling window every 10min.
More specific, the stream of data that is keyed and need to compute counts for 7 seconds.
a tumbling window of 1 second
a sliding window for counting 7 seconds with an advance of 1 second
a windowall to output all counts every 1s
I am not able to integration test it (i.e., similar to unit test but an end-to-end testing) as the input has fake event time, which won't trigger
Here is my snippet
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val oneDayCounts = data
.map(t => (t.key1, t.key2, 1L, t.timestampMs))
.keyBy(0, 1)
.timeWindow(Time.seconds(1))
.sum(2)
val sevenDayCounts = oneDayCounts
.keyBy(0,1)
.timeWindow(Time.seconds(3), Time.seconds(1))
.sum(2)
// single reducer
sevenDayCounts
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(1)))
.process(...)
I use EventTime as timestamp and set up an integration test code with MiniClusterWithClientResource. also created some fake data with some event timestamp like 1234l, 4567l, etc.
EventTimeTrigger is able to be fired for sum computation but the following TumblingProcessingTimeWindow is not able to trigger. I had a Thread.sleep of 30s in the IT test code but still not triggered after the 30s
In general it's a challenge to write meaningful tests for processing time windows, since they are inherently non-deterministic. This is one reason why event time windows are generally prefered.
It's also going to be difficult to put a sleep in the right place so that is has the desired effect. But one way to keep the job running long enough for the processing time window to fire would be to use a custom source that includes a sleep. Flink streaming jobs with finite sources shut themselves down once the input has been exhausted. One final watermark with the value MAX_WATERMARK gets sent through the pipeline, which triggers all event time windows, but processing time windows are only fired if they are still running when the appointed time arrives.
See this answer for an example of a hack that works around this.
Alternatively, you might take a look at https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/TumblingProcessingTimeWindowsTest.java to see how processing time windows can be tested by mocking getCurrentProcessingTime.
Given a simple textbox to answer a question, how would you go about saving answers as the user types? Assuming that the answer is upserted.
Using a somewhat naive example of an onChange handler and saving answers using an API within that onChange, I ran into two problems:
Sends what seems to be too many patch requests from client to API since it's every time the input textbox answer changes.
If using postgres upsert or creating your own, then the second request made by the onChange handler may not see the first if the user is typing quickly, creating duplicate records for that textbox (two answers for one question)
I need to execute queries beforehand to get necessary relationship values before updating, so the HTTP wait time is fairly long. Fixing this now, but seems to not address the above two issues.
I thought of only saving once the user presses spacebar, but that seems hacky. Normally of course I'd save after clicking a submit button, but in this case it must be as the user types.
Any suggestions / comments? Cheers.
EDIT:
Also tried saving within onBlur.. But this doesn't always get called depending on where the user clicks.
Does it have to patch the data for every keypress or can you use a delay?
For instance...
User starts typing: Register the event, set a timeout for say 3 seconds
If the user keeps typing: reset that timeout back to the full 3s
If the user pauses typing + 3s passes: patch the data with new value
When the user resumes typing: set another timeout
I got a special question regarding timer boundary events on a user task in Activiti/Camunda:
When starting the process I set the timer duration with a process variable and use expressions in the boundary definition to resolve the variable. The boundary event is defined on a user task.
<bpmn2:timerEventDefinition id="_TimerEventDefinition_11">
<bpmn2:timeDuration xsi:type="bpmn2:tFormalExpression">${hurry}</bpmn2:timeDuration>
</bpmn2:timerEventDefinition>
In some cases, when the timer is already running it can occur, that the deadline (dueDate) should be extended because the asignee has requested more time. For this purpose i want to change the value of the process variable defining the deadline.
As it happens, the variable is already resolved at the process-start and set to the boundary event.
Any further changes of the variable do not affect the dueDate of the boundary timer because it is stored in the database and is not updated when the value of the variable changes.
I know how to update the dueDate of the job element via the Java API, but i want to provide a generic approach like setting it with changing the value of the variable.
The most common use case for extending the deadline will be when the boundary timer is already running.
Any ideas how to cope with this problem?
Any tips are very apprechiated.
Cheers Chris
After some time of thinking I came up with a workaround like that:
I start the process with two variables. "hurry" is evaluated for the boundary timer. And "extendDeadline" is initialized with false. If the timer triggers and the process advances to the exclusive gateway, the value of "extendDeadline" is evaluated.
If a user has changed the value of "extendDeadline" to true during the the timer was running the process returns to the user task again where the boundary timer is set to the value of "hurry".
If the "extendDeadline" is still set to false, the process can proceed.
If the timer is running you can change dueDate of the timer by executing a signal. If a assginee requested more time, set new value of hurry and execute the signal. The old timer will be canceled and the new timer will be created with new due date.
runtimeService.setVariable(execution.getId(), "hurry", newDueDate);
runtimeService.signalEventReceived(signalName, execution.getId());
Solution is to have 2 out going sequence flow, one should be from boundary timer on task and another one should be from task it self, as shown in diagram added by #theFriedC. See following image also.
Then you can use some exclusive gateway on 2nd sequence flow and reroute that back to same task with a new timer value.
I am designing an application which has the potential to hang while waiting for data from servers (either Database or internet) the problem is that I don't know how best to cope with the multitude of different places things may take time.
I am happy to display a 'loading' dialog to the user while access is happening but ideally I don't want this to flick up and disappear for short running operations.
Microsoft word appears to handle this quite nicely, as if you click a button and the operation takes a long time, after a few seconds you get a 'working..' dialog. The operation is still synchronous and you can't interrupt the operation. However if the same operation happens quickly you obviously don't get the dialog.
I am happy(ish) to devise some generic background worker thread handler and 99% of my data processing is already done in static atomic methods, but I would like to go best practice on this one if I can.
If anyone has patterns, code or suggestions I welcome them all
Cheers
I would definitely think asynchronously using a pattern with 2 events. The first "event" is that you actually got your data from wherever/whenever you had to wait for it. The second event is a delay timer. If you get your data before this timer pops, all is well and good. If not, THEN you pop up your "I'm busy" and allow them to cancel the request. Usually cancel just mean "ignore" when you finally get the response.
Microsoft word appears to handle this quite nicely, as if you click a button and the operation takes a long time, after a few seconds you get a 'working..' dialog. The operation is still synchronous and you can't interrupt the operation. However if the same operation happens quickly you obviously don't get the dialog.
If this is the behavior your want...
You could handle this, fairly easily, by wrapping a class around BackgroundWorker. Just time the start of the DoWork event, and the time to the first progress report. If a certain amount of time passes, you could show your dialog - otherwise, block (since it's a short process).
That being said, any time you're doing work that can be processed asynchronously, I'd recommend doing it that way. It's much nicer to never block your UI for a noticable interval, even if it's short. This becomes much simpler in .NET 4 (or 3.5 with Rx framework) by using the task parallel library.
Ideally you should be running any IO or non-UI processing either in a background thread or asynchronously to avoid locking up the UI.