Could anyone help me answer the question that if there is a 5s time window executing aggregation operations every 2s. The first 2s handle data in window between n and n+5 while the second 2s handle data in window between n+2 and n+7. It seems that the Flink do duplicate work in time of n+2 to n+5. is it that? Any help would be appreciate!
the windows that Flink processes should be (n, n+2), (n, n+4), (n+1, n+6), (n+3, n+8). So in the beginning the windows are not 5 seconds wide. It has to "catch up" because there is not enough time-data available yet. The window is processed every two seconds and it looks at the last 5 seconds from that point.
In general it is easier to think about windows if the slide size and window size have a greatest common divisor (GCD). Also, windows can then potentially be evaluated faster, using a pane-based approach.
You are right. If you apply a function, that could potentially reuse the result of the first window to compute the second window, currently Flink does not exploit this. Each window in computed from scratch. (However, this optimization in on the development agenda already and will be supported in future releases.)
Related
I have a Flink streaming job, where I am creating windows based on some keys and adding the data points to a data point.
.window(SlidingProcessingTimeWindows.of(Time.days(1), Time.minutes(5)))
.trigger(CountTrigger.of(5))
.window(<ProcessWindowFunction>)
I'm using the above piece of code for creating sliding window of size 1 day with a slide of 5 minutes. Als, count trigger is triggering the process function once 5 data points are accumulated.
In addition to this, I want to trigger the process function for every slide that happens. This means, till 1 day of data points are accumulated (window size), CountTrigger shall trigger the process function and once 1 day worth points are created and window slides for every 5 minutes, I want to trigger the process function for every data point instead of waiting for CountTrigger to accumulate 10 data points. Can someone please help me on how to do this?
Be aware that this is going to be pretty painful. Every event is going to be assigned to a total of 288 windows (24 hours / 5 minutes). This means that every event is going to trigger 288 calls to your ProcessWindowFunction.
If you find you need to optimize this, you can probably get better performance with a carefully implemented KeyedProcessFunction.
Extend org.apache.flink.streaming.api.windowing.triggers.CountTrigger and override onProcessingTime method. Implement your processing time logic there. Then use this trigger instead of plain CountTrigger.
I am using react profiler to make my app more efficient. It will commonly spit out a graph like this:
I am confused because the timings do not add up. For example, it would make sense if the total commit time for "Shell" was 0.3ms then "Main" was "0.2ms of 0.3ms." But that is not the case.
What precisely do these timings mean and how do they add up?
(note: I have read "Introducing the React Profiler" but it appears from this section that this time-reporting convention is new since that article.)
The first number (0.2ms) is the self duration and the second number (0.3ms) is the actual duration. Mostly the self duration is the actual duration minus the time spent on the children. I have noticed that the numbers don't always add up perfectly, which I would guess is either a rounding artifact or because some time is spent on hidden work. For example, in your case, the Shell has an actual time of 3.1ms and a self duration of 0.3ms, which means the 2 children (Navbar and Main), should add up to 3.1ms - 0.3ms, or 2.8ms. However, we see that the Navbar is not re-rendered, so it's 0ms, but the actual duration for Main is only 2.7ms, not 2.8ms. It's not going to have any impact in practical terms when you're performance tuning, but it does violate expectations a bit.
I'm looking for a way to disable the PID.vi from running in Labview when the event case container is false.
The program controls motor position to maintain constant tension on a cable using target force and actual force as the input parameters. The output is motor position. Note that reinitialize is set to false since it needs previous instances to spool the motor.
Currently, when the event case is true the motor spools as expected and maintains the cable tension. But when the event case state is toggled the PID.vi seems to be running in the background when false causing the motor spool sporatically.
Is there a way to freeze the PID controls so that it continues from where it left off?
The PID VI does not run in the background. It only executes when you call it. That said, PID is a time-based calculation. It calculates the difference from the last time you called the VI and uses that to calculate the new values. If a lot of time passed, it will just try to fix it using that data.
If you want to freeze the value and then resume fixing smoothly, you can use the limits input on the top and set the max and min to your desired output. This will cause the PID VI to always output that value. You will probably need a feedback node or shift register to remember the last value output by the PID.
What Yair said is not entirely true - the integral and derivative terms are indeed time dependent, but the proportional is not. A great reference for understanding PIDs and how they are implemented in LabVIEW can be found here (not sure why it is archived). Also, the PID VIs are coded in G so you can simply open them to see how they operate.
If you take a closer look at the PID VI, you can see what is happening and why you might not get the response you expect. In the VI itself, dt will be either 1) what you set it to, or 2) an accumulation of time based on a tick count stored in the VI (the default). Since you have not specified a dt, the PID algorithm uses the accumulated time between calls. If you have "paused" calculation for some time, this will have an impact on the integral and derivative output.
The derivative output will kick in when there is a change in the process variable (use of the process variable prevents derivative kick). The effect of a large accumulated time between calls will be to reduce the response of this term. The time that you have paused will have a more significant impact on the integral term. Since the response of the integral portion of the controller is the proportional to the integral of the error over dt, the longer you pause the larger the response simply because because the the algorithm is performing a trapezoidal integration over dt.
My first suggestion is don't pause the controller - let the PID do what it is supposed to do. If you are using it properly, then you should not have to stop the controller action. But, if you must pause the controller action, consider re-initializing the controller. This will force the controller to reset the accumulated time term and the response in the first iteration will be purely proportional.
Hope this helps.
The classic advice in multithreading programing is to do processor heavy work on a background thread and return the result to the UI thread for minor processing (update a label, etc). What if generating the WPF element itself is the operation which is expensive?
I'm working with a third party library which generates some intense elements, which can take around to 0.75s - 1.5s to render. Generating one isn't too bad, but when I need to create 5 of them to show at once it noticeably locks the UI (including progress spinners). Unfortunately, there isn't any other place to create them because WPF is thread affine.
I've already tried DispatcherPriority.Background but its not enough. What is the recommended way to deal with this problem?
If the objects being created derived from Freezable, then you can actually create them on a different thread than the UI thread - you just have to call Freeze on them while you're on the worker thread, and then you can transfer them over. However, that doesn't help you for items that don't derive from Freezable.
Have you tried creating them one at a time? The following example doesn't do any useful work but it does show how the basic structure for doing a lot of work in little bits:
int count = 100;
Action slow = null;
slow = delegate
{
Thread.Sleep(100);
count -= 1;
if (count > 0)
{
Dispatcher.BeginInvoke(slow, DispatcherPriority.Background);
}
};
Dispatcher.BeginInvoke(slow, DispatcherPriority.Background);
The 'work' here is to sleep for a tenth of a second. (So if you replace that with real work that takes about as long, you'll get the same behaviour.) This does that 100 times, so that's a total of 10 seconds of 'work'. The UI remains reasonably responsive for the whole time - things like dragging the window around become a bit less smooth, but it's perfectly usable. Change both those Background priorities to Normal, and the application locks up.
The key here is that we end up returning after doing each small bit of work having queued up the next bit - we end up calling Dispatcher.BeginInvoke 100 times in all instead of once. That gives the UI a chance to respond to input on a regular basis.
Is there some reason that identical math operations would take significantly longer in one Silverlight app than in another?
For example, I have some code that takes a list of points and transforms them (scales and translates them) and populates another list of points. It's important that I keep the original points intact, hence the second list.
Here's the relevant code (scale is a double and origin is a point):
public Point transformPoint(Point point) {
// scale, then translate the x
point.X = (point.X - origin.X) * scale;
// scale, then translate the y
point.Y = (point.Y - origin.Y) * scale;
// return the point
return point;
}
Here's how I'm doing the loop and timing, in case it's important:
DateTime startTime = DateTime.Now;
foreach (Point point in rawPoints) transformedPoints.Add(transformPoint(point));
Debug.Print("ASPX milliseconds: {0}", (DateTime.Now - startTime).Milliseconds);
On a run of 14356 points (don't ask, it's modeled off a real world number in the desktop app), the breakdown is as follows:
Silverlight app #1: 46 ms
Silverlight app #2: 859 ms
The first app is an otherwise empty app that is doing the loop in the MainPage constructor. The second is doing the loop in a method in another class, and the method is called during an event handler in the GUI thread, I think. But should any of that matter, considering that identical operations are happening within the loop itself?
There maybe something huge I'm missing in how threading works or something, but this discrepancy doesn't make sense to me at all.
In addition to the other comments and answers I'm going to read between the lines a little.
In the first app you have pretty much this code in isolation running in the MainPage constructor. IWO you've create a fresh Silverlight app and slapped this code in it and thats it.
In the second app you have more actual real world stuff. At the very least you have this code running as the result of a button click on a rudimentory UI. Therein lies the clue.
Take a blank app and drop a button on it. Run it and click the button, what does the button do? There are animations attached to visual states of the button. This animation (or other animations or loops) are likely running in parrallel with your code when you click the button. Timers (whether you do it properly with StopWatch or not) record elapsed time, not just the time your thread takes. Hence when other threads are doing other things (like animations) your timing will be off.
My first suspicion would be that Silverlight App #2 triggers a garbage collection. Scaling ~15,000 points should be taking a millisecond, not nearly a second.
Try to reduce memory allocations in your code. Can transformedPoints be an array, rather than a dynamically grown data structure?
You can also look at the GC performance counters, but simply reducing the memory allocation may turn out to be simpler.
Could it be possible your code is not being inlined in the CLR by the app that is running slower?
I'm not sure how the CLR in SL handles inlining, but here is a link to some of the prerequisites for inlining in 3.5 SP1.
http://udooz.net/blog/2009/04/clr-improvements-in-net-35-sp1/