Let's say we want to compute the sum and average of the items,
and can either working with states or windows(time).
Example working with windows -
https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#example-program
Example working with states -
https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataartisans/flinktraining/exercises/datastream_java/ride_speed/RideSpeed.java
Can I ask what would be the reasons to make decision? Can I infer that if the data arrives very irregularly (50% comes in the defined window length and the other 50% don't), the result of the window approach is more biased (because the 50% events are dropped)?
On the other hand, do we spend more time checking and updating the states when working with states?
First, it depends on your semantics... The two examples use different semantics and are thus not comparable directly. Furthermore, windows work with state internally, too. It is hard to say in general with approach is the better one.
As Flink's window semantics are very rich, I would suggest to use windows. If you cannot express your semantics with windows, using state can be a good alternative. Using windows, has the additional advantage that state handling---which is hard to get done right---is done automatically for you.
The decision is definitely independent from your data arrival rate. Flink does not drop any data. If you work with event time (rather than with processing time) your result will be the same independently of the data arrival rate after all.
Related
Embedded C. I have a list of things I want to do, procedurally, mostly READ and WRITE and MODIFY actions, acting on the results of the last statement. They can take up to 2 seconds each, I can’t block.
Each action can have states of COMPLETE and ERROR which has sub-states for reason the error occurred. Or on compete I’ll want to check or modify some data.
Each list of actions is a big switch and to re-enter I keep a list of which step I’m on, a success step++ and I come back in further down the list next time.
Pretty simple, but I’m finding that to not block I’m spending a ton of effort checking states and errors and edges constantly. Over and over.
I would say 80% of my code is just checks and moving the system along. There has to be a better way!
Are there any design patterns for async do thing and come back later for results in a way that efficiently handles some of the exception/edge/handling?
Edit: I know how to use callbacks but don’t really see that as “a solution” as I just need to get back to a different part of the same list for the next thing to do. Maybe it’s would be beneficial to know the backend to how async and await in other languages work?
Edit2: I do have an RTOS for other projects but this specific question, assume no threads/tasks, just bare metal superloop.
Your predicament is a perfect fit for state machines (really, probably UML statecharts). Each different request can each be handled in its own state machine, which handle events (such as COMPLETE or ERROR indications) in a non-blocking, run-to-completion manner. As the events come in, the request's state machine moves through its different states towards completion.
For embedded systems, I often use the QP event-driven framework for such cases. In fact, when I looked up this link, I noticed the very first paragraph uses the term "non-blocking". The framework provides much more than state machines with hierarchy (states within states), which is already very powerful.
The site also has some good information on approaches to your specific problem. I would suggest starting with the site's Key Concepts page.
To get you a taste of the content and its relevance to your predicament:
In spite of the fundamental event-driven nature, most embedded systems
are traditionally programmed in a sequential manner, where a program
hard-codes the expected sequence of events by waiting for the specific
events in various places in the execution path. This explicit waiting
for events is implemented either by busy-polling or blocking on a
time-delay, etc.
The sequential paradigm works well for sequential problems, where the
expected sequence of events can be hard-coded in the sequential code.
Trouble is that most real-life systems are not sequential, meaning
that the system must handle many equally valid event sequences. The
fundamental problem is that while a sequential program is waiting for
one kind of event (e.g., timeout event after a time delay) it is not
doing anything else and is not responsive to other events (e.g., a
button press).
For this and other reasons, experts in concurrent programming have
learned to be very careful with various blocking mechanisms of an
RTOS, because they often lead to programs that are unresponsive,
difficult to reason about, and unsafe. Instead, experts recommend [...] event-driven programming.
You can also do state machines yourself without using an event-driven framework like the QP, but you will end up re-inventing the wheel IMO.
I have an always one application, listening to a Kafka stream, and processing events. Events are part of a session. And I need to do calculations based off of a sessions data. I am running into a problem trying to correctly run my calculations due to the length of my sessions. 90% of my sessions are done after 5 minutes. 99% are done after 1 hour. Sessions may last more than a day, due to this being a real-time system, there is no determined end. Session are unique, and show never collide.
I am looking for a way where I can process a window multiple times, either with an initial wait period and processing any later events after that, or a pure process per event type structure. I will need to keep all previous events around(ListState), as well as previously processed values(ValueState).
I previously thought allowedLateness would allow me to do this, but it seems the lateness is only considered for when the event should have been processed, it does not extend an actual window. GlobalWindows may also work, but I am unsure if there is a way to process a window multiple times. I believe I can used an evictor with GlobalWindows to purge the Windows after a period of inactivity(although admittedly, I did not research this yet, because I was unsure of how to trigger a GlobalWindow multiple times.
Any suggestions on how to achieve what I am looking to do would be greatly appreciated, I would also be happy to clarify any points needed.
If SessionWindows won't do the job, then you can use GlobalWindows with a custom Trigger and Evictor. The Trigger interface has onElement and timer-based callbacks that can fire whenever and as often as you like. If you go down this route, then yes, you'll also need to implement an Evictor to dispose of elements when they are no longer needed.
The documentation and the source code are helpful when trying to understand how this all fits together.
I'm wondering if there is any widespread timer solution.
I want to write a simple game engine which would process users input on tick every 20ms (or perform some actions once after 20ms (or any other period)) and basically update "global" state via transactions, and also I plan to use futures so this solution should be able to deal with concurrency caveats.
Can you give me an advice?
You have actually got two distinct issues here.
The first is the issues of timers. You have lots of choices here:
Start a thread that sleeps between actions, something like (future (loop [] (do-something) (Thread/sleep 20) (when (game-still-running) (recur))))
Use a Java TimerTask - easy to call from Clojure
Use a library like my little utility Task that includes a DSL for repeating tasks
Use the timer functionality from whatever game engine you are using - most of these provide some tools for setting up a game loop
I'd probably just use the simple thread option - it's pretty easy to set up and easy to hack more features in later if you need to.
The second issue is handling game state. This is actually the trickier problem and you will need to design it around the particular type of game you are making, so I'll just give a few points of advice:
If your game state is immutable, then you get a lot of advantages: your rendering code can draw the current game state independently while the game update is computing the next game state. Immutability has some performance costs, but the concurrency advantages are huge if you can make it work. Both of my mini Clojure games (Ironclad and Alchemy) use this approach
You should probably try and store your game state in a single top-level var. I find this works better than splitting different parts of the game state over different vars. It also means that you don't really need ref-based transactions: an atom or agent will usually do the trick.
You may want to implement a queue of events that need to get processed sequentially by the game state update function. This is particularly important if you have multiple concurrent event sources (e.g. player actions, timer ticks, network events etc.)
Nowadays I would consider core/async to be a good choice, since
it scales very well when it comes to complex tasks that can be separated into activities that communicate using channels
avoids binding activity to a thread
here is the sketch
(require '[clojure.core.async :refer [go-loop]])
(go-loop []
(do
(do-something ...)
(Thread/sleep 1000)
(recur))))
This solution assumes you're writing Clojure on the JVM.
Something like this could work:
(import '(java.util TimerTask Timer))
(let [task (proxy [TimerTask] []
(run [] (println "Here we go!")))]
(. (new Timer) (schedule task (long 20))))
I'm encountering a performance issue in my application. Essentially I click a button, and a list is populated with databound data (this is virtualized because of the large amount of data), and then click another button which will add a row to an associated list view. I'm being vague because I think it's just necessary to illustrate how little is actually going on in the UI.
Here's what I know:
I don't see the issue on my my beefy dev computer running Win 7 Pro, nor on XP SP3 machines with decent specs. I only see it on a certain brand of laptops (Lenovo ThinkPads) running Win 7 enterprise with 4 GB RAM and Core i5 CPU (much beefier than the XP desktop).
Because of the aforementioned findings, I'm not thinking this is an issue with code.
I profiled with Microsoft's PerfView tool and noticed what I would assume to be an incredibly large number of calls to UIElement.Measure (not ever invoked directly by our code), something I don't see when I profile on the other machines.
The laptop has a 1360x780 resolution, so I thought that perhaps the small resolution was causing the GPU to unnecessarily render the controls because of some data binding that I might be doing (which might explain the large number of calls to Measure()). I extended the laptop's display to my 24" monitor and didn't see any improvement.
Right now I'm assuming that the issue is with the GPU. I've updated the driver with no improvements.
Even though I don't think it's an issue with code, is there a WPF equivalent to "SuspendLayout()"
Is there a way to profile GPU performance to see if it is being hammered during certain processes
(far shot) Has anyone had similar performance issues that seem to be computer specific and suggestions on how to track them down?
Sorry if this is a vague question. I tried to make it comply with SO's usage reqs. Let me know if you want any more info.
Just as an addendum: The program is using WPF, C# 4.0, the issue seems to be around Telerik controls (though I don't think they're suspect since we use them elsewhere without issue).
Turns out it's caused by a known Microsoft issue. I’d try to explain, but I won’t. Mainly because I can’t.
Article talking about fix (see post by Viðar on 3 August 2010):
Microsoft Hotfix site: http://support.microsoft.com/kb/2484841/en-us
Fix: http://archive.msdn.microsoft.com/KB2484841/Release/ProjectReleases.aspx?ReleaseId=5583
1. Answer
To to prevent rampant MeasureOverride calls originating from the WPF ContextLayoutManager:
protected override void OnChildDesiredSizeChanged(UIElement el)
{
/* base.OnChildDesiredSizeChanged(el); */ // avoid rampant remeasuring
}
2. Relevant citation
UIElement.OnChildDesiredSizeChanged(UIElement) Method ...
The OnChildDesiredSizeChanged(UIElement) method has the default implementation of calling InvalidateMeasure() on itself. A typical implementation would be: do whatever optimization your own element supports, and then typically call base OnChildDesiredSizeChanged(UIElement) from a̲t̲ l̲e̲a̲s̲t̲ o̲n̲e̲ of the code branches...
...the implication (and fact-of-the-matter) being that, for any single parent layout pass originated by any one of its children, the parent's MeasureOverride will be called additionally—and likely extraneously—once for each of its children whose size(s) have changed as well.
3. Discussion
In the case where multiple children change their sizes "at the same time", the parent will typically detect and account for the new overall layout amongst all of its children entirely during just the first of these calls. This is standard practice in WPF, and is encouraged by MeasureOverride(…) deliberately excluding any indication of some specific triggering child. Besides the fact that in the most common cases there is no such child (see above link for details), it makes code for attempting any sort of "partial layout" onerous. But mostly, why would any layout calculation ever want to proceed without first obtaining all the very latest available measurements anyway?
So we see that after a single MeasureOverride call, triggered by whichever child happened to be "first" (it shouldn't matter), the layout of the parent should actually be final regarding all of its latest child size information. But this doesn't mean that any queued OnChildDesiredSizeChanged notifications—for other children whose size(s) had also changed—have gone away. Those calls are still pending with the parent, and unless the virtual base call is explicitly abandoned (as shown in bullet #1), each will generate one additional, now-extraneous MeasureOverride call.
4. Caveat
The code shown here disables a̲l̲l̲ child-initiated measure invalidations, which is appropriate for cases where the parent either willfully forbids such changes, or is inherently already aware of them. This is not uncommon; for example, it includes any parent that always fully determines and enforces the size of its children, or more generally, any parent that only adopts DesiredSize values from its children during its own measure pass. What's important is that the measuring of the parent be sufficiently guaranteed by its own parent only.
The situation where a parent might wish to cancel some of the child-initiated measure notifications while preserving/allowing others will naturally depend on additional particular circumstances. It seems more obscure, so it is not addressed here.
In many embedded applications there is a tradeoff between making the code very efficient or isolating the code from the specific system configuration to be immune to changing requirements.
What kinds of C constructs do you usually employ to achieve the best of both worlds (flexibility and reconfigurabilty without losing efficiency)?
If you have the time, please read on to see exactly what I am talking about.
When I was developing embedded SW for airbag controllers, we had the problem that we had to change some parts of the code every time the customer changed their mind regarding the specific requirements. For example, the combination of conditions and events that would trigger the airbag deployment changed every couple weeks during development. We hated to change that piece of code so often.
At that time, I attended the Embedded Systems Conference and heard a brilliant presentation by Stephen Mellor called "Coping with changing requirements". You can read the paper here (they make you sign-up but it's free).
The main idea of this was to implement the core behavior in your code but configure the specific details in the form of data. The data is something you can change easily and it can even be programmable in EEPROM or a different section of flash.
This idea sounded great to solve our problem. I shared this with my colleague and we immediately started reworking some of the SW modules.
When trying to use this idea in our coding, we encountered some difficulty in the actual implementation. Our code constructs got terribly heavy and complex for a constrained embedded system.
To illustrate this I will elaborate on the example I mentioned above. Instead of having a a bunch of if-statements to decide if the combination of inputs was in a state that required an airbag deployment, we changed to a big table of tables. Some of the conditions were not trivial, so we used a lot of function pointers to be able to call lots of little helper functions which somehow resolved some of the conditions. We had several levels of indirection and everything became hard to understand. To make a long story short, we ended up using a lot of memory, runtime and code complexity. Debugging the thing was not straightforward either. The boss made us change some things back because the modules were getting too heavy (and he was maybe right!).
PS: There is a similar question in SO but it looks like the focus is different. Adapting to meet changing business requirements?
As another point of view on changing requirements ... requirements go into building the code. So why not take a meta-approach to this:
Separate out parts of the program that are likely to change
Create a script that will glue parts of source together
This way you are maintaining compatible logic-building blocks in C ... and then sticking those compatible parts together at the end:
/* {conditions_for_airbag_placeholder} */
if( require_deployment)
trigger_gas_release()
Then maintain independent conditions:
/* VAG Condition */
if( poll_vag_collision_event() )
require_deployment=1
and another
/* Ford Conditions */
if( ford_interrupt( FRONT_NEARSIDE_COLLISION ))
require_deploymen=1
Your build script could look like:
BUILD airbag_deployment_logic.c WITH vag_events
TEST airbag_deployment_blob WITH vag_event_emitter
Thinking outloud really. This way you get a tight binary blob without reading in config.
This is sort of like using overlays http://en.wikipedia.org/wiki/Overlay_(programming) but doing it at compile-time.
Our system is subdivided into many components, with exposed configuration and test points. There is a configuration file that is read at start-up that actually helps us instantiate components, attach them to each other, and configure their behavior.
It's very OO-like, in C, with the occasional hack to implement something like inheritance.
In the defense/avionics world software upgrades are very strictly controlled, and you can't just upgrade SW to fix issues... however, for some bizarre reason you can update a configuration file without a major fight. So it's been darn useful for us to be able to specify a lot of our implementation in those configuration files.
There is no magic, just good separation of concerns when designing the system and a bit of foresight on the part of the developers.
What are you trying to save exactly? Effort of code re-work? The red tape of a software version release?
It's possible that changing the code is reasonably straight-forward, and quite possibly easier than changing data in tables. Moving your often-changing logic from code to data is only helpful if, for some reason, it's less effort to modify data rather than code. That might be true if the changes are better expressed in a data form (e.g. numeric parameters stored in EEPROM). Or it might be true if the customer's requests make it necessary to release a new version of software, and a new software version is a costly procedure to build (lots of paperwork, or perhaps OTP chips burned by the chip maker).
Modularity is very good principle for these sort of things. Sounds as though you're already doing it to some degree. It's good to aim to isolate the often-changing code to as small an area as possible, and try to keep the rest of the code ("helper" functions) separate (modular) and as stable as possible.
I don't make the code immune to requirements changes per se, but I always tag a section of code that implements a requirement by putting a unique string in a comment. With the requirements tags in place, I can easily search for that code when the requirement needs a change. This practice also satisfies a CMMI process.
For example, in the requirements document:
The following is a list of
requirements related to the RST:
[RST001] Juliet SHALL start the RST with 5 minute delay when the ignition
is turned OFF.
And in the code:
/* Delay for RST when ignition is turned off [RST001] */
#define IGN_OFF_RST_DELAY 5
...snip...
/* Start RST with designated delay [RST001] */
if (IS_ROMEO_ON())
{
rst_set_timer(IGN_OFF_RST_DELAY);
}
I suppose what you could do is to specify several valid behaviors based on a byte or word of data that you could fetch from EEPROM or an I/O port if necessary and then create generic code to handle all possible events described by those bytes.
For instance, if you had a byte that specified the requirements for releasing the airbag it could be something like:
Bit 0: Rear collision
Bit 1: Speed above 55mph (bonus points for generalizing the speed value!)
Bit 2: passenger in car
...
Etc
Then you pull in another byte that says what events happened and compare the two. If they're the same, execute your command, if not, don't.
For adapting to changing requirements I would concentrate on making the code modular and easy to change, e.g. by using macros or inline functions for parameters which are likely to change.
W.r.t. a configuration which can be changed independently from the code, I would hope that the parameters which are reconfigurable are specified in the requirements, too. Especially for safety-critical stuff like airbag controllers.
Hooking in a dynamic language can be a lifesaver, if you've got the memory and processor power for it.
Have the C talk to the hardware, and then pass up a known set of events to a language like Lua. Have the Lua script parse the event and callback to the appropriate C function(s).
Once you've got your C code running well, you won't have to touch it again unless the hardware changes. All of the business logic becomes part of the script, which in my opinion is a lot easier to create, modify and maintain.