I am new to Camel and trying to implement a requirement as below
from("seda://requestCamel").throttle(ExpressionBuilder.beanExpression(CamelHandler.getInstance(),"getTransactionsPerMillie")).timePeriodMillis(CamelHandler.getTimeInMillie()).bean(service, "callCamel(${body})");
From the above expression both the values (throttle and timePeriodMillis) required to change dynamically at run time. Using bean expression I could able to manage for throttle however timePeriodMillis does not accept real time values. Any help/work around would be very helpful.
Thanks
You cannot change the time period millis at runtime - its a fixed value in the current implementation of the throttler in Apache Camel.
Related
I understand in general that event time uses Watermarks to make progress in time. In the case of Flink Statefun which is more based on iteration it may be a problem. So my question is if I use the delayed message (https://nightlies.apache.org/flink/flink-statefun-docs-stable/docs/sdk/java/#sending-delayed-messages), then does it mean we can use only processing time notion in Stateful functions ?
I would like to change to Event time processing model but not sure how it will work with Stateful functions.
pipeline.time-characteristic: EventTime
pipeline.auto-watermark-interval: 200
Can someone explain if it is possible ?
Stateful Functions (statefun) doesn't support watermarks or event-time processing. But you could implement your own triggering logic based on the timestamps in arriving events.
I am trying to de-duplicate events in my Flink pipeline. I am trying to do that using guava cache.
My requirement is that, I want to de-duplicate over a 1 minute window. But at any given point I want to maintain not more than 10000 elements in the cache.
A small background on my experiment with Flink windowing:
Tumbling Windows: I was able to implement this using Tumbling windows + custom trigger. But the problem is, if an element occurs in the 59th minute and 61st minute, it is not recognized as a duplicate.
Sliding Windows: I also tried sliding window with 10 second overlap + custom trigger. But an element that came in the 55th second is part of 5 different windows and it is written to the sink 5 times.
Please let me know if I should not be seeing the above behavior with windowing.
Back to Guava:
I have Event which looks like this and a EventsWrapper for these events which looks like this. I will be getting a stream of EventsWrappers. I should remove duplicate Events across different EventsWrappers.
Example if I have 2 EventsWrappers like below:
[EventsWrapper{id='ew1', org='org1', events=[Event{id='e1',
name='event1'}, Event{id='e2', name='event2'}]},
EventsWrapper{id='ew2', org='org2', events=[Event{id='e1',
name='event1'}, Event{id='e3', name='event3'}]}
I should emit as output the following:
[EventsWrapper{id='ew1', org='org1', events=[Event{id='e1',
name='event1'}, Event{id='e2', name='event2'}]},
EventsWrapper{id='ew2', org='org2', events=[Event{id='e3', name='event3'}]}
i.e Making sure that e1 event is emitted only once assuming these two events are within the time and size requirements of the cache.
I created a RichFlatmap function where I initiate a guava cache and value state like this. And set the Guava cache in the value state like this. My overall pipeline looks like this.
But each time I try to update the guava cache inside the value state:
eventsState.value().put(eventId, true);
I get the following error:
java.lang.NullPointerException
at com.google.common.cache.LocalCache.hash(LocalCache.java:1696)
at com.google.common.cache.LocalCache.put(LocalCache.java:4180)
at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4888)
at events.piepline.DeduplicatingFlatmap.lambda$flatMap$0(DeduplicatingFlatmap.java:59)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
On further digging, I found out that the error is because the keyEquivalence inside the Guava cache is null.
I checked by directly setting on the Guava cache(not through state, but directly on the cache) and that works fine.
I felt this could be because, ValueState is not able to serialize GuavaCache. So I added a Serializer like this and registered it like this:
env.registerTypeWithKryoSerializer((Class<Cache<String,Boolean>>)(Class<?>)Cache.class, CacheSerializer.class);
But this din't help either.
I have the following questions:
Any idea what I might be doing wrong with the Guava cache in the above case.
Is what I am seeing with my Tumbling and Slinding windows implementation is what is expected or am I doing something wrong?
What will happen if I don't set the Guava Cache in ValueState, instead just use it as a plain object in the DeduplicatingFlatmap class and operate directly on the Guava Cache instead of operating through the ValueState? My understanding is, the Guava cache won't be part of the Checkpoint. So when the pipeline fails and restarts, the GuavaCahe would have lost all the values in it and it will be empty on restart. Is this understanding correct?
Thanks a lot in advance for the help.
See below.
These windows are behaving as expected.
Your understanding is correct.
Even if you do get it working, using a Guava cache as ValueState will perform very poorly, because RocksDB is going to deserialize the entire cache on every access, and re-serialize it on every update.
Moreover, it looks like you are trying to share a single cache instance across all of the orgs that happen to be multiplexed across a single flatmap instance. That's not going to work, because the RocksDB state backend will make a copy of the cache for each org (a side effect of the serialization involved).
Your requirements aren't entirely clear, but a deduplication query might help. But I'm thinking MapState in combination with timers in a KeyedProcessFunction is more likely to be the building block you need. Here's an example that might help you get started (but you'll be wanting to handle the timers differently).
My use case is that I want to apply different CEP patterns to the same datastream. the CEP patterns come dynamically & i want them to be added to flink without having to restart the job. While all conditions can be handled via custom classes that implement IterativeCondition, my main problem is that the temporal condition accepts only TimeWindow; which cannot be handled. Is there some way that the value passed to .within() be set based on the input elements?
Something similar was asked here: Flink and Dynamic templates recognition
Best Answer:
"What one could add is a co-flat map operator which receives on one input channel the events and on the other input channel patterns. For each newly received pattern one either updates the existing NFA (this functionality is missing) or compiles a new one. In the latter case, one would apply incoming events to all stored NFAs."
I am trying to implement this but I am facing some difficulty. Specifically, on the point of "In the latter case, one would apply incoming events to all stored NFAs"
Reason being that I apply stream to pattern using: PatternStream matchStream = CEP.pattern(tmatchStream, pattern);
But the stream "tmatchStream" would not be defined inside the co-flatMap. Am I missing something here??? Any help would be greatly appreciated.
Unfortunately the answer to the linked question is still valid. Flink CEP does not support dynamic patterns at that moment. There is already a JIRA ticket for that though: FLINK-7129
The earliest reasonable target version for that feature will be 1.6.0
Angular JS (1.x) has a directive for input[time].
I think to map time to a datetime causes more new problems than solving some (when mapping time to a standard javascript time I think to map to number of seconds or milliseconds would be a much better approach)
I want get back to the original html5 implementation. Is there a way to do so?
The original implementation does/should not use Date! See
https://www.w3.org/TR/html-markup/input.time.html
Used Apache axis to consume a WSDL which has a column of type="xsd:dateTime".
In SOAP UI with plain vanilla request, response has -
<UpdateDateTime>2012-05-08T04:58:00</UpdateDateTime>
However when using axis consumer, for the same value - listOfValues[pos].getUpdateDateTime().getTime() returns a different time -
2012-05-07 21:58:00.
getUpdateDateTime() in the above returns instance of java.util.Calendar.
Is it a timezone problem or the 'T' in between is parsed incorrect?
How can I resolve this?
How are you displaying the "2012-05-07 21:58:00"? It's almost certainly just a time zone issue. I suspect it's treating 2012-05-08T04:58:00 as a universal time, and applying your local time zone to that. It's hard to say without seeing any code or where your diagnostics have come from, but I'd be surprised if it weren't just a time zone issue.
Of course, if you can use Joda Time instead of java.util.Date/Calendar, you can use LocalDateTime which is what I suspect is represented here (given the lack of time zone information in the response). I don't know if Axis supports that, but it would be worth looking into.
Here is the Eclipse debug with date time value in Inspect -
http://i45.tinypic.com/157zpy1.jpg
& the plain request in soap UI gives -
2012-05-08T04:58:00
Bombay,India
Sorry, i meant UTC in my first reply.