I want implements the aggregationFunction by the processKeyedFunction, because the default aggregationFunction does not support rich function,
Besides, I tryed the aggreagationFunction + processWindowFunction(https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html), but it also cannot satisfy my needs, so I have to use the basic processKeyedFunction to implement the aggregationFunction, the detail of my problem is as followed:
in processFunction, , I define a windowState for stage the aggregation value of elements, the code is as followed:
public void open(Configuration parameters) throws Exception {
followCacheMap = FollowSet.getInstance();
windowState = getRuntimeContext().getMapState(windowStateDescriptor);
currentTimer = getRuntimeContext().getState(new ValueStateDescriptor<Long>(
"timer",
Long.class
));
in processElement() function, I use the windowState (which is a MapState initiate in open function) to aggregate the window element, and register the first timeServie to clear current window state, the code is as followed:
#Override
public void processElement(FollowData value, Context ctx, Collector<FollowData> out) throws Exception
{
if ( (currentTimer==null || (currentTimer.value() ==null) || (long)currentTimer.value()==0 ) && value.getClickTime() != null) {
currentTimer.update(value.getClickTime() + interval);
ctx.timerService().registerEventTimeTimer((long)currentTimer.value());
}
windowState = doMyAggregation(value);
}
in onTimer() function, first, I register the next timeService in next One minute, and clear the window State
#Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<FollowData> out) throws Exception {
currentTimer.update(timestamp + interval); // interval is 1 minute
ctx.timerService().registerEventTimeTimer((long)currentTimer.value());
out.collect(windowState);
windowState.clear();
}
but when the program is running , I found that all the windowState in onTimer is empty, but it is not empyt in processElement() function, I don't know why this happens, maybe the execution logic is different, how can I fix this,
Thanks in advance !
new added code about doMyAggregation() part
windowState is a MapState , key is "mykey", value is an self-defined Object AggregateFollow
public class AggregateFollow {
private String clicked;
private String unionid;
private ArrayList allFollows;
private int enterCnt;
private Long clickTime;
}
and the doMyAggregation(value) function is pretty much like this , the function of doMyAggregation is to get all the value whose source field is 'follow', but if there are no values whose field is 'click' during 1 minute, the 'follow' value should be obsolete, in a word , it's like a join operation of 'follow' data and 'click' data,
AggregateFollow acc = windowState.get(windowkey);
String flag = acc.getClicked();
ArrayList<FollowData> followDataList = acc.getAllFollows();
if ("0".equals(flag)) {
if ("follow".equals(value.getSource())) {
followDataList.add(value);
acc.setAllFollows(followDataList);
}
if ("click".equals(value.getSource())) {
String unionid = value.getUnionid();
clickTime = value.getClickTime();
if (followDataList.size() > 0) {
ArrayList listNew = new ArrayList();
for (FollowData followData : followDataList) {
followData.setUnionid(unionid);
followData.setClickTime(clickTime);
followData.setSource("joined_flag"); //
}
acc.setAllFollows(listNew);
}
acc.setClicked("1");
acc.setUnionid(unionid);
acc.setClickTime(clickTime);
windowState.put(windowkey, acc);
}
} else if ("1".equals(flag)) {
if ("follow".equals(value.getSource())) {
value.setUnionid(acc.getUnionid());
value.setClickTime(acc.getClickTime());
value.setSource("joined_flag");
followDataList.add(value);
acc.setAllFollows(followDataList);
windowState.put(windowkey, acc);
}
}
because of performance problem, original windowAPI is not a valid choice for me, the only way here I think is to use processFunction + ontimer and Guava Cache ,
Thanks a lot
If windowState is empty, it would be helpful to see what doMyAggregation(value) is doing.
It's difficult to debug this, or propose good alternatives, without more context, but out.collect(windowState) isn't going to work as intended. What you might want to do instead would be to iterate over this MapState and collect each key/value pair it contains to the output.
I changed the type of windowState from MapState to ValueState, and the problem is solved, maybe it is a bug or something, can anyone can explain this?
Related
I have a keyd stream of data that looks like:
{
summary:Integer
uid:String
key:String
.....
}
I need to aggregate the summary values in some time range, and once I achieved a specifc number , to flush the summary and all the of the UID'S that influenced the summary to database/log file.
after the first flush , I want to discare all the uid's from the memory , and just flush every new item immediatelly.
So I tried this aggregate function.
public class AggFunc implements AggregateFunction<Item, Acc, Tuple2<Integer,List<String>>>{
private static final long serialVersionUID = 1L;
#Override
public Acc createAccumulator() {
return new Acc());
}
#Override
public Acc add(Item value, Acc accumulator) {
accumulator.inc(value.getSummary());
accumulator.addUid(value.getUid);
return accumulator;
}
#Override
public Tuple2<Integer,List<String>> getResult(Acc accumulator) {
List<String> newL = Lists.newArrayList(accumulator.getUids());
accumulator.setUids(Lists.newArrayList());
return Tuple2.of(accumulator.getSum(), newL);
}
#Override
public Acc merge(Acc a, Acc b) {
.....
}
}
and in the aggregate process function , I flush the list to state, and if I need to save to dataBase I'm clearing the state and save flag in the state to indicate it.
But it seems crooked to me. And I'm not sure if that would work well for me.
Is there a better solution to this situation?
Work with a state inside a rich function. Keep adding the uid in your state and when the window triggers to flush the values. This page from the official documentation has an example.
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/state/state.html#using-keyed-state
For your case a ListState will work well.
EDIT:
The solution above is for non-window case. for window case simply use the aggrgation with apply function that can have a rich window function
I need to count events within a tumbling window. But I also want to send events with 0 Value if there were no events within the window.
Something like.
windowCount: 5
windowCount: 0
windowCount: 0
windowCount: 3
windowCount: 0
...
import com.google.protobuf.Message;
import org.apache.flink.api.common.functions.AggregateFunction;
import org.skydivin4ng3l.cepmodemon.models.events.aggregate.AggregateOuterClass;
public class BasicCounter<T extends Message> implements AggregateFunction<T, Long, AggregateOuterClass.Aggregate> {
#Override
public Long createAccumulator() {
return 0L;
}
#Override
public Long add(T event, Long accumulator) {
return accumulator + 1L;
}
#Override
public AggregateOuterClass.Aggregate getResult(Long accumulator) {
return AggregateOuterClass.Aggregate.newBuilder().setVolume(accumulator).build();
}
#Override
public Long merge(Long accumulator1, Long accumulator2) {
return accumulator1 + accumulator2;
}
}
and used here
DataStream<AggregateOuterClass.Aggregate> aggregatedStream = someEntryStream
.windowAll(TumblingEventTimeWindows.of(Time.seconds(5)))
.aggregate(new BasicCounter<MonitorOuterClass.Monitor>());
TimeCharacteristics are ingestionTime
I read about a TiggerFunction which might detect if the aggregated Stream has received an event after x time but i am not sure if that is the right way to do it.
I expected the aggregation to happen even is there would be no events at all within the window. Maybe there is a setting i am not aware of?
Thx for any hints.
I chose Option 1 as suggested by #David-Anderson:
Here is my Event Generator:
public class EmptyEventSource implements SourceFunction<MonitorOuterClass.Monitor> {
private volatile boolean isRunning = true;
private final long delayPerRecordMillis;
public EmptyEventSource(long delayPerRecordMillis){
this.delayPerRecordMillis = delayPerRecordMillis;
}
#Override
public void run(SourceContext<MonitorOuterClass.Monitor> sourceContext) throws Exception {
while (isRunning) {
sourceContext.collect(MonitorOuterClass.Monitor.newBuilder().build());
if (delayPerRecordMillis > 0) {
Thread.sleep(delayPerRecordMillis);
}
}
}
#Override
public void cancel() {
isRunning = false;
}
}
and my adjusted AggregateFunction:
public class BasicCounter<T extends Message> implements AggregateFunction<T, Long, AggregateOuterClass.Aggregate> {
#Override
public Long createAccumulator() {
return 0L;
}
#Override
public Long add(T event, Long accumulator) {
if(((MonitorOuterClass.Monitor)event).equals(MonitorOuterClass.Monitor.newBuilder().build())) {
return accumulator;
}
return accumulator + 1L;
}
#Override
public AggregateOuterClass.Aggregate getResult(Long accumulator) {
AggregateOuterClass.Aggregate newAggregate = AggregateOuterClass.Aggregate.newBuilder().setVolume(accumulator).build();
return newAggregate;
}
#Override
public Long merge(Long accumulator1, Long accumulator2) {
return accumulator1 + accumulator2;
}
}
Used them Like this:
DataStream<MonitorOuterClass.Monitor> someEntryStream = env.addSource(currentConsumer);
DataStream<MonitorOuterClass.Monitor> triggerStream = env.addSource(new EmptyEventSource(delayPerRecordMillis));
DataStream<AggregateOuterClass.Aggregate> aggregatedStream = someEntryStream
.union(triggerStream)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
.aggregate(new BasicCounter<MonitorOuterClass.Monitor>());
Flink's windows are created lazily, when the first event is assigned to a window. Thus empty windows do not exist, and can't produce results.
In general there are three ways to workaround this issue:
Put something in front of the window that adds events to the stream, ensuring that every window has something in it, and then modify your window processing to ignore these special events when computing their results.
Use a GlobalWindow along with a custom Trigger that uses processing time timers to trigger the window (with no events flowing, the watermark won't advance, and event time timers won't fire until more events arrive).
Don't use the window API, and implement your own windowing with a ProcessFunction instead. But here you'll still face the issue of needing to use processing time timers.
Update:
Having now made an effort to implement an example of option 2, I cannot recommend it. The issue is that even with a custom Trigger, the ProcessAllWindowFunction will not be called if the window is empty, so it is necessary to always keep at least one element in the GlobalWindow. This appears then to require implementing a rather hacky Evictor and ProcessAllWindowFunction that collaborate to retain and ignore a special element in the window -- and you also have to somehow get that element into the window in the first place.
If you're going to do something hacky, option 1 appears to be much simpler.
I am using two AutocompleteTextFilters as depended filters. I want the second one filter to change its options depending on the suggestion of the first filter.
I have bind an event listener on the first filter so as when it loose focus it triggers a proccess on the second filter.
The proble is that the second filter never changes its options. I even have setup hardcoded values in case somethig was wrong on my code but no luck.
The code I use is below:
public CreateSubmission(com.codename1.ui.util.Resources resourceObjectInstance, Map<String, ProjectType> projectTypes) {
this.projectTypes = projectTypes;
initGuiBuilderComponents(resourceObjectInstance);
gui_ac_projecttype.clear();
gui_ac_projecttype.setCompletion( this.projectTypes.keySet().toArray( new String[0]) );
gui_ac_projecttype.addFocusListener( new ProjectTypeFocusListener( this ));
gui_ac_steps.setCompletion( new String[]{"t10", "t20"});
}
public void makeSteps (String selection) {
ProjectType projectType = this.projectTypes.get( selection );
if (projectType != null) {
this.selectedProjectType = selection;
int length = projectType.projectSteps.length;
String[] steps = new String[ length ];
for(int i =0; i < length; i ++) {
steps[i] = projectType.projectSteps[i].projectStep;
}
// String[] s = gui_ac_steps.getCompletion();
gui_ac_steps.setCompletion( new String[]{"t1", "t2"} );
gui_ac_steps.repaint();
}
else {
}
}
public class ProjectTypeFocusListener implements FocusListener{
private CreateSubmission parent;
public ProjectTypeFocusListener( CreateSubmission parent ) {
this.parent = parent;
}
#Override
public void focusGained(Component cmp) {
//throw new UnsupportedOperationException("Not supported yet."); //To change body of generated methods, choose Tools | Templates.
}
#Override
public void focusLost(Component cmp) {
this.parent.makeSteps (
((AutoCompleteTextField)cmp).getText()
);
//throw new UnsupportedOperationException("Not supported yet."); //To change body of generated methods, choose Tools | Templates.
}
}
On the above code the initialization happens on "public CreateSubmission" method.
"gui_ac_projecttype" is the first AutocompletionTextField that triggers the whole proccess through it's FocusListener handler (class ProjectTypeFocusListener )
"gui_ac_steps" is the second AutocompleteTextField filter that must change its values. On the code above I initialize it's suggestions to "t10", "t20". Those two values are shown correctly.
Later from iside the FoculListenerHandler's method "ProjectTypeFocusListener.focusLost" I call method "makeSteps" which sets the suggestion options to "t1", "t2 and then I repaint the component. These two last values are never shown. It remains on the first values "t10", "t20".
The Strange thing is that in debugger when I ask gui_ac_steps.getCompletion(); to see the current options ( the code that is commentd out into makeSteps method) I get the correct values "t1", "t2".
But on the screen it keeps showing "t10", "t20".
any help is aprreciated.
You shouldn't do anything "important" in a focus listener. Especially not with a text field. They are somewhat unreliable because the text field switches to native editing and in effect transfers the focus there. The problem is that some events are delayed due to the back and forth with the native editing so by the time the focus event is received you've moved on to the next field.
Try something like this for this specific use case https://www.codenameone.com/blog/dynamic-autocomplete.html
//Creating a window of ten items
WindowedStream<ObservationEvent,Tuple,GlobalWindow> windowStream = inputStream.keyBy("rackId").countWindow(10);
// Applying a Window Function , adding some custom evaluating all the values in the window
DataStream<ObservationEvent> inactivityStream = windowStream.apply(new WindowFunction<ObservationEvent, ObservationEvent , Tuple , GlobalWindow>() {
#Override
public void apply(Tuple tuple, GlobalWindow timeWindow, Iterable<ObservationEvent> itr, Collector<ObservationEvent> out)
//custom evaluation logic
out.collect(new ObservationEvent(1,"temperature", "stable"));
}
});
//Defining Simple CEP Pattern
Pattern<ObservationEvent, ?> inactivityPattern = Pattern.ObservationEvent>begin("first")
.subtype(ObservationEvent.class)
.where(new FilterFunction<ObservationEvent>() {
#Override
public boolean filter(ObservationEvent arg0) throws Exception {
System.out.println( arg0 ); //This function is not at all called
return false;
}
});
PatternStream<ObservationEvent> inactivityCEP = CEP.pattern(inactivityStream.keyBy("rackId"), inactivityPattern);
When I run this code, the filter function inside the where clause is not at all getting called.
I have printed the inactivityStream.print() and I can see the matching value.
Now, when I plug in the inputStream directly without applying a window. The pattern is matching
I printed inputStream and WindowedStream and I can see they both send similar kind of data.
What am I missing
The FilterFunction should be getting called eventually but you are going to have to wait for 10 events for the SAME key before you see your FilterFunction called for the first time. Could it be that you are just not waiting long enough in your windowing test?
Keep in mind that if you have many unique keys this implies you will have to wait well more than 10 times as long in the window test before you'll see your filter function called.
I am seeing an error be reported quite frequently in my application. I know where it is happening, I just don't know why. The app plays sound effects (not background music), I am using Xna.Audio and I have a timer to call FrameworkDispatcher.Update() in regular intervals. I am using SoundEffect.CreateInstance so I can I have the sound effect loop. It looks like an error happens when changing the volume. I don't know WHY though. Are there known instances when this can fail?
Function ::: Offset
xxx_RaiseException ::: 32
WatsonUnhandledManagedException ::: 300
Dbg_NotifyManagedException ::: 136
FirstPassException ::: 1044
TransitionStub ::: 0
Microsoft.Xna.Framework.Helpers.ThrowExceptionFromErrorCode ::: 76
Microsoft.Xna.Framework.Audio.SoundEffectInstance.set_Volume ::: 232
Microsoft.Xna.Framework.Audio.SoundEffectInstance..ctor ::: 232
Microsoft.Xna.Framework.Audio.SoundEffect.CreateInstance ::: 192
AgiliTrain.PhoneyTools.Media.SoundEffectPlayer..ctor ::: 96
WildSafari.ClassicModeGame.animalVisibleTimer_Tick ::: 344
...snip....
For those asking, here is more code. The SoundEffectPlayer takes a SoundEffect, creates an instance, and kicks off the XNA FrameworkDispatcher (via the GameTimer). This code is taken from PhoneyTools, so go check out the codeplex project if you need more context. When I want to play an effect, I just new up a SoundEffectPlayer, and pass in what I want.
public class SoundEffectPlayer
{
GameTimer _xnaTimer = new GameTimer();
SoundEffectInstance _effect = null;
public float _duration;
public SoundEffectPlayer(SoundEffect effect, bool loop)
{
_effect = effect.CreateInstance();
_effect.IsLooped = loop;
_duration = (float)effect.Duration.TotalSeconds;
}
public void Play(float volume)
{
_xnaTimer.Start();
_effect.Play();
_effect.Volume = volume;
}
public void Stop()
{
_effect.Stop(true);
_xnaTimer.Stop();
}
}
public class GameTimer
{
DispatcherTimer _timer = new DispatcherTimer()
{
Interval = TimeSpan.FromMilliseconds(50),
};
public GameTimer()
{
_timer.Tick += new EventHandler(_timer_Tick);
}
void _timer_Tick(object sender, EventArgs e)
{
FrameworkDispatcher.Update();
}
public void Start()
{
if (!_timer.IsEnabled) _timer.Start();
}
public void Stop()
{
if (_timer.IsEnabled) _timer.Stop();
}
}
Trying to help out, although i don't have a valid fix...
The setter property for SoundEffectInstance is not as simple as one might think:
set
{
lock (this.voiceHandleLock)
{
if (this.IsDisposed)
{
throw new ObjectDisposedException(base.GetType().Name, FrameworkResources.ObjectDisposedException);
}
if ((value < 0f) || (value > 1f))
{
throw new ArgumentOutOfRangeException("value");
}
Helpers.ThrowExceptionFromErrorCode(SoundEffectUnsafeNativeMethods.SetVolume(this.voiceHandle, value));
this.currentVolume = value;
}
}
Basically the part we're after comes in the line before last.
If the call from SoundEffectUnsafeNativeMethods.SetVolume returns any code smaller than 0, the exception you're getting will be triggered.
Here's what's happening inside the unsafe method:
public static unsafe int modopt(IsLong) SetVolume(uint soundEffectInstanceHandle, float volume)
{
lock (SoundEffectSubsystemSyncObject)
{
if ((soundEffectInstanceHandle != 0) && (soundEffectInstanceHandle != uint.MaxValue))
{
CHandleTable* tablePtr = ?g_pTable#CHandleTable##0PAV1#A;
KernelSoundEffectInstance* instancePtr = CHandleTable.LookUp<class Microsoft::Xna::Framework::Audio::KernelSoundEffectInstance>((CHandleTable modopt(IsConst)* modopt(IsConst) modopt(IsConst)) ?g_pTable#CHandleTable##0PAV1#A, soundEffectInstanceHandle, 0);
if (instancePtr == null)
{
return -2147024809;
}
return **(((int*) instancePtr))[0x34](instancePtr, volume);
}
return -2147024809;
}
}
You can try to surround the creation of _effect.CreateInstance with try...catch and rethrow a more comprehensive and helpful exception that will contain for example, information about the SoundEffect object which you're trying to create (basically you can note down all of its field's values for inspection. (not sure that what you get from MS is only a stack trace or the actual exception messaage).
It seems that on WP7, the call to set the volume goes to some OS pointer table that does some magic behind the scenes and looks for the sound effect handle in this table.
If for some reason it is not found or something similar (don't know why, maybe we can take it with XNA developers), it will throw this exception you're getting.
I know this is not a solution but maybe this is a step in the right direction.
Generally, the exception should not be thrown for the situation you are describing, as long as you built the XNA compatibility layer as it should be (details here).
However, I noticed that you are using PhoneyTools.Media.SoundEffectPlayer - what for? It might be the cause of the problem in your situation.
Also, you seem to have ambiguous references - effect and _effect - which one are you working with when playing content?