I'm trying to setup a route that process files, here is my route configuration:
#Component
public class MyRoute extends RouteBuilder {
public static final String FIRST_ROUTE = "firstRoute";
public static final String SECOND_ROUTE = "secondRoute";
private final Processor myProcessor;
#Autowired
public MyRoute (#Qualifier("my.processor") Processor myProcessor) {
this.myProcessor= myProcessor;
}
#Override
public void configure() throws Exception {
from("file://{{data.input.dir}}?moveFailed=errors&delete=true&doneFileName=ACK-${file:name}&include=MY-FILE-FIRST.*")
.routeId(FIRST_ROUTE)
.process(myProcessor);
from("file://{{data.input.dir}}?moveFailed=errors&delete=true&doneFileName=ACK-${file:name}&include=MY-FILE-SECOND.*")
.routeId(SECOND_ROUTE)
.process(myProcessor);
}
}
As you can see I have two route for two files that have different name (one with FIRST, one with SECOND).
I want to trigger the process with variable to avoid to check the filename inside the process.
Currently my process function look like that:
public void process(Exchange exchange) throws Exception
What I want is something like that:
public void process(Exchange exchange, String identifier) throws Exception
And have the identifier set in the route definition (it's static, not depending on the filename, I need to have "FIRST" for the first route, and "SECOND" for the second one).
Is this possible ?
Because a process interface signature only takes an Exchange parameter, we cannot do process(Exchange exchange, String identifier).
Instead we can create a bean and call the method from the route like below.
from("file://{{data.input.dir}}?moveFailed=errors&delete=true&doneFileName=ACK-${file:name}&include=MY-FILE-SECOND.*")
.routeId(SECOND_ROUTE)
.bean(myBean,"process(*, " + SECOND + ")");
Related
In camel , Processor interface has the following api:
public interface Processor{
public void process(Exchange exchange) throws Exception;
}
In camel, is there any component that returns value similar to that given below:
public interface ComponentThatReturns<T>{
public T result(Exchange exchange) throws Exception;
}
Processors in Camel can store any needed values as either message headers or exchange properties.
If you had a component that returned a different type as you describe, something would still need to get that value into the Exchange to pass it to the next step in the route.
Inside Flink task instance I need to access remote web service to get some data when the event coming ,however I don't want to access remote web service every time when event coming, so I need to cache the data in local memory and can be accessed by all task of the process , how to do it ? storing the data in the static private variable at the class level ?
Such as the following example ,if set the local variable localCache at class Splitter, it cached at operator level instead of process level .
public class WindowWordCount {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Tuple2<String, Integer>> dataStream = env
.socketTextStream("localhost", 9999)
.flatMap(new Splitter())
.keyBy(0)
.timeWindow(Time.seconds(5))
.sum(1);
dataStream.print();
env.execute("Window WordCount");
}
public static class Splitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
***private object localCache ;***
#Override
public void flatMap(String sentence, Collector<Tuple2<String, Integer>> out) throws Exception {
for (String word: sentence.split(" ")) {
out.collect(new Tuple2<String, Integer>(word, 1));
}
}
}
}
Exactly like you said. You'd use a static variable in a RichFlatMapFunction and initialize it in open. open will be called on each TaskManager before feeding in any record. Note that there is an instance of Splitter being created for each different slot, so in most cases there are several Splitter instances on one TaskManager. Thus, you need to guard against double creation.
public static class Splitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
private transient Object localCache;
#Override
public void open(Configuration parameters) throws Exception {
if (localCache == null)
localCache = ... ;
}
#Override
public void flatMap(String sentence, Collector<Tuple2<String, Integer>> out) throws Exception {
for (String word: sentence.split(" ")) {
out.collect(new Tuple2<String, Integer>(word, 1));
}
}
}
A scalable approach might use a Source operator to actually perform the call to the web service and then write the result to a stream. You can then access that stream as a broadcast stream to your operator resulting in the one object (web call result) emitted to the broadcast stream being sent to each instance of the receiving operator. This will share the result of that single web call across all machines and JVM's in your cluster. You can also persist broadcast state and share it with new instances of your operator as the cluster scales up.
I have a route with a custom aggregator, it looks like this:
I would like to wrap the details so that developers can just add a single line to their routes to get the functionality. Can I wrap those lines so that I have a class that extends ProcessorDefinition, and then add that ProcessorDefinition to routes that need it, so that it looks like an extension of the DSL? If yes, is the addOutput() method the way to do this?
Something like this:
from("file:" + FILE_PATH + "?noop=true")
.log("Detected file")
.split().tokenize("\n")
.streaming()
.unmarshal(bindy)
.addProcessorDefinition(new MyCustomAggregation())
.to("direct:handleAggregatedRecords");
Where
MyCustomerAggegation extends ProcessorDefinition
I have a similar use-case and I don't think Camel has anything to offer out of the box to extend routes at a certain point with route fragments. However, you can break apart one route into multiple fragments
public abstract class ExtensibleRouterBuilder extends RouteBuilder {
#Override
public void configure() {
ProcessorDefinition<?> routeFragmentToExtend= from("file:" + FILE_PATH + "?noop=true")
.log("Detected file")
.split().tokenize("\n")
.streaming()
.unmarshal(bindy);
configure(routeFragmentToExtend);
routeFragmentToExtend.to("direct:handleAggregatedRecords");
}
//"Users" of your API can implement and extend your route
public abstract void configure(ProcessorDefinition<?> from);
}
But this can get messy, especially if you have several choice() calls, so I wish Camel enhances it's API one day.
To get a similar outcome, I use Lombok #ExtensionMethod in the following way:
#ExtensionMethod(MyRouteBuilder.Extensions.class)
public class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() {
from("direct:in")
.debugLog("Incoming message ${body}");
}
#UtilityClass
public static class Extensions {
public static ProcessorDefinition<?> debugLog(
#NonNull ProcessorDefinition<?> route, String logMessage) {
return route
.log(LoggingLevel.DEBUG, "debug-logger", logMessage);
}
}
}
It is possible to pass parameters to a Camel route?, for instance, in the next code snippet:
public class MyRoute extends RouteBuilder {
public void configure() throws Exception {
from("direct:start")
.to("cxf:bean:inventoryEndpoint?dataFormat=PAYLOAD");
}
}
The value for dataFormat is in hard code, but, what if I want set it dynamically?, passing a value from the code where route is called. I know this is possible adding a constructor and passing parameters in it, like this:
public class MyRoute extends RouteBuilder {
private String type;
public MyRoute(String type){
this.type = type;
}
public void configure() throws Exception {
from("direct:start")
.to("cxf:bean:inventoryEndpoint?dataFormat=" + type);
}
}
There is another way?
Thanks so much!
As you mentioned, you can use a constructor (or setters or any other Java/Framework instruments) if the parameters are static from a Camel point of view.
The parameters are configurable in the application, but after the application is started they do no more change. So, every message processed by the Camel route uses the same value.
In contrast, when the parameters are dynamic - i.e. they can change for every processed message, you can use the dynamic endpoint toD() of Camel. These endpoint addresses can contain expressions that are computed on runtime. For example the route
from("direct:start")
.toD("${header.foo}");
sends messages to a dynamic endpoint and takes the value from the message header named foo.
Or to use your example
.toD("cxf:bean:inventoryEndpoint?dataFormat=${header.dataFormat}");
This way you can set the dataformat for every message individually through a header.
You can find more about dynamic endpoints on this Camel documentation page
In my application I have a generic Camel Route such as the following
from("direct:something").to("direct:outgoing")
and then dynamically in my code I deploy another route:
from("direct:outgoing").process(processor)
When flowing from route 1 to route 2 a new Exchange will be created. Is there an idiomatic way to correlate both? Should I set EXCHANGE.Correlation_ID header on the first route before sending it out?
This should definitely all be processed on the one exchange. Run this test and you'll see the same camel Exchange, with the same properties, etc.
public class CamelExchangeTest {
public static void main(String[] args) throws Exception {
final Processor showExchangeIdProcessor = new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println(exchange.getExchangeId());
}
};
Main camelMain = new Main();
camelMain.addRouteBuilder(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("timer:foo?period=1s&repeatCount=1")
.log("excgabge created!")
.process(showExchangeIdProcessor)
.to("direct:outgoing")
;
from("direct:outgoing")
.log("outgoing!")
.process(showExchangeIdProcessor)
;
}
});
camelMain.run();
}
}
Output:
ID-MYPC-55760-1411129552791-0-2
ID-MYPC-55760-1411129552791-0-2
So something else is going on. When you say "direct:outgoing", do you mean exactly that or is it something different - a different component perhaps?
When you say the route is created dynamically, how exactly is that done, and when (and why?)
From the Camel doc:
Some EIP patterns will spin off a sub message, and in those cases, Camel will add a correlation id on the Exchange as a property with they key Exchange.CORRELATION_ID, which links back to the source Exchange. For example the Splitter, Multicast, Recipient List, and Wire Tap EIP does this.
Thus, Exchange.CORRELATION_ID is set by Camel and should not be set by your application. But feel free to set a custom header or property if you need to such as:
exchange.getIn().setProperty("myProperty", myIdentifier);