I have a route that, when lauched, requests messages from two datasources (routeA and routeB) and aggregates them into a single message. Each aggregated message MUST contain exactly one routeA message and one routeB message, if not, then drop it.
This process must be launched at specific intervals (i.e: every 5 min).
My question is, how can I let the aggregator know that all messages from routeA and routeB where processed and the messages that didn't find their pair, must be droped?
I'm currently using completionTimeout feature, but I don't like this solution for obvious reasons.
I know camel has a completionFromBatchConsumer feature, but I don't know how to use it with with multiple datasets.
Am greatefull for any advice.
Here's what I have right now:
<!-- main route -->
<route id="main">
<camel:from uri="timer://timer1?period=20000"/>
<multicast>
<to uri="direct:startA"/>
<to uri="direct:startB"/>
</multicast>
</route>
<!-- messages from route A -->
<route id="routeA" />
<from uri="direct:startA" />
<to uri="sql:select * from sampleDB?dataSource=ds"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectAJsonConverter"/>
<unmarshal ref="ObjectAJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectACorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- messages from route B -->
<route id="routeB" />
<from uri="direct:startB"/>
<to uri="ldap:ldapcontext?base=DC=company,DC=net"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectBJsonConverter"/>
<unmarshal ref="ObjectBJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectBCorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- aggregate the messages, create new ObjectC that contains ObjectA and ObjectB -->
<!-- wait 200000 ms for all messages from routeA and routeB to enter the aggregator -->
<route id="aggretatorRoute">
<from uri="seda:myAggregator"/>
<aggregate ref="myEntityAggregator" completionSize="2" completionTimeout="200000" discartOnCompletionTimeout="true" ignoreInvalidCorrelationKeys="true">
<correlationExpression><simple>${in.header.objectid}</simple></correlationExpression>
<to uri="bean:myProcessor?method=doSomethingWithObjectC"/>
</aggregate>
You can just in your AggregationStrategy only aggregate one of ObjectA and one of ObjectB. So if you see a 2nd of either of them, then just not aggregate it. And if you then want to drop what you have done so far, then you can mark the exchange to stop, by setting
exchange.setProperty("CamelRouteStop", true);
And if you then want to drop this immediately, then add a completionPredicate, that checks if that stop has been set.
<completionPredicate><simple>${property.CamelRouteStop} == true</simple></completionPredicate>
And for the correlationExpression, you can likely just use <constant>true</constant> as it seems you only work on one group.
Thanks Claus. Actually I took a different approach. Instead using an aggregator to join three different object, I now query a list of ids and use those to progressively build my complex object.
<route id="composeObject">
<from uri="sql:select id from people?oneSource">
<split><simple>${body}</simple>
<to uri="direct:getobjectOne"/>;
<to uri="bean:addToComplexObject"/>
<to uri="direct:getObjectTwo/>
<to uri="bean:addToComplexObject"/>
<to uri="direct:getobjectThree/>
<to uri="bean:addToComplexObject"/>
<to uri="seda:outChannel"/>
</split>
</route>
Related
I'm processing files with a Camel route like this:
<route>
<from uri="file:inbox?delete=true"/>
<recipientList>
<simple>exec://process.sh?args=inbox/${file:name}</simple>
</recipientList>
<log message="processed ${file:name}: ${body.stdout} ${body.stderr}"/>
</route>
Now I'd like the route to fail when process.sh finishes with nonzero exit-code. I found ${headers.CamelExecExitValue} but don't really know what to do with it.
In the example above, the file should not get deleted when process.sh fails. In my actual use-case, the route consumes files from a JMS queue and I want the file to stay in the queue. I think this can be done with <transacted/> but need to know how to fail the route.
I found How to define exception to be thrown through ref in Apache Camel which in combination with CamelExecExitValue lets me abort this way:
<route>
<from uri="file:inbox?delete=true"/>
<to uri="exec://process.sh"/>
<choice>
<when>
<simple>
${headers.CamelExecExitValue} != 0
</simple>
<throwException exceptionType="java.lang.RuntimeException" message="failed importing ${file:name}: ${body.stdout} ${body.stderr}"/>
</when>
</choice>
<log message="processed ${file:name}"/>
</route>
A bit verbose for my taste but works fine.
I am trying to process a zip file, which contains several files inside (all small, so working in memory is not a problem) that need to be transformed and zipped together again.
I managed to unzip, transform the files, but for some reason, the splitter is not completing, only using completionTimeout makes the aggregator to create the final zip archive.
Here's the route:
<route id="ZipFile">
<from uri="file:{{file.path.in}}?move=.done&moveFailed=.error&readLock=rename"/>
<setProperty propertyName="OriginalZipName">
<simple>${header.CamelFileName}</simple>
</setProperty>
<unmarshal>
<zipFile usingIterator="true"/>
</unmarshal>
<split streaming="true">
<simple>${body}</simple>
<log message="************ CamelSplitComplete = ${property.CamelSplitComplete}"/>
<to uri="direct:ProcessUnzippedFile"/>
<setHeader headerName="CamelFileName">
<simple>${property.OriginalZipName}</simple>
</setHeader>
<!-- Aggregate to zip -->
<aggregate strategyRef="zipAggregationStrategy" eagerCheckCompletion="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<completionPredicate>
<simple>${property.CamelSplitComplete}</simple>
</completionPredicate>
<setHeader headerName="CamelFileName">
<simple>${property.OriginalZipName}</simple>
</setHeader>
<to uri="file://{{file.path.out}}"/>
</aggregate>
</split>
</route>
Any idea what could be the issue?
The Camel Splitter provides a built- in aggregator, which makes it even easier to
aggregate split messages back into single outgoing message
You have to define the aggregation strategy in the split definition
<split strategyRef="zipAggregationStrategy">
Haven't tried it in XML but in Java DSL this is one of my examples that does exactly that.
.unmarshal(zipFile)
.split(bodyAs(Iterator.class),new ZipAggregationStrategy(true,true))
.streaming()
.stopOnException()
.to("direct:transform-ticket")
.end();
I have a requirement where i want to use mutlicast in Apache Camel for than single time in a single route. i.e Multicast within a multicast.
<routeContext id="myRoute" xmlns="http://camel.apache.org/schema/spring">
<route id="myRouteId">
<from uri="activemq:queue:{{XXXX.queue}}" />
....
<multicast parallelProcessing="true">
<pipeline>
##everything working fine here
</pipeline>
<pipeline>
<multicast>
<pipeline>
<log message="Inserting in database now"></log>
<transform>
<method ref="insertBean" method="myBatchInsertion"></method>
</transform>
<choice>
<when>
<simple>${in.header.myCount} == ${properties:batch.size} </simple>
<to uri="sql:{{sql.core.insertMyQuery}}?batch=true"></to>
<log message="Inserted rows ${body}"></log>
</when>
</choice>
</pipeline>
</multicast>
</pipeline>
</multicast>
</route>
</routeContext>
Is it possible to do that?
When i am trying to do that, my program is not getting executed successfully.
Is the unsuccessful execution is a result of mulitple multicast?
Can anybody help?
I got the reference from following link:
http://camel.apache.org/multicast.html
Why do you use pipeline? It "is" pipeline by default.
Also all the log and transform and choice statements can be put outside of the multicast. And since you are generating your endpoints dynamically, put the values in a header and use recipientlist for dynamic endpoints. Multicast is for hard-coded endpoints. Here is an example:
<route>
<from uri="direct:a" />
<!-- use comma as a delimiter for String based values -->
<recipientList delimiter=",">
<header>myHeader</header>
</recipientList>
</route>
http://camel.apache.org/recipient-list.html
I have a route where I'd like to compute if "from" is activemq or something else. I found replaceFromWith, but it seems to be used for testing only. I tried using camel's choice when/otherwise to switch between "from activemq" and "from seda", but that errors out as invalid syntax. I'm looking for something that would do the following:
<route id="doPost">
<choice>
<when>
<groovy>exchange.getIn().getHeader("sometest") != null</groovy>
<from uri="activemq:doPost?..."/>
</when>
<otherwise>
<from uri="seda:doPost?....."/>
</otherwise>
</choice>
Thanks in advance, appreciate the help.
You cannot dynamically select a consumer based on the exchange, the exchange wont exist before consuming it using the <from .../> tag.
Instead, you can construct a route for each consumer, normalize the exchange you receive from different types of consumers, and forward them to a common route for processing.
<route id="activemqConsumerRoute">
<from uri="activemq:doPost?..."/>
<!-- normalize the exchange to be understandable by the next route -->
<to uri="direct:commonProcessingRoute" />
</route>
<route id="sedaConsumerRoute">
<from uri="seda:doPost?....."/>
<!-- normalize the exchange to be understandable by the next route -->
<to uri="direct:commonProcessingRoute" />
</route>
<route id="commonProccessingRoute">
<from uri="direct:commonProcessingRoute" />
<!-- whatever business logic you have -->
</route>
I wrote the following route and expected that the bean 'teaserService' should be called only one time, at the end of processing of all files, but ... it's called after processing of each file:
<route id="teaserInterface">
<from
uri="file://{{teaser.dropInDir}}?readLock=changed&delete=true&delay=60000" />
<choice>
<when>
<simple>${file:ext} == 'properties'</simple>
<to uri="file://{{teaser.config.directory}}" />
</when>
<when>
<simple>${file:ext} == 'jpg' || ${file:ext} == 'JPG'</simple>
<to uri="sftp://{{apache.ftp.user}}#{{apache.ftp.host}}/{{apache.teaser.ftp.targetDir}}?password={{apache.ftp.password}}&binary=true&separator=UNIX" />
</when>
<otherwise>
<transform>
<simple>Dear user,\n\n the Teaser interface only accept *.jpg and *.properties files, but we found the file ${file.name}.\n\n Have a nice day,\nYour lovely Teaser interface</simple>
</transform>
<to
uri="smtp://smtp.blabla.com?contentType=text/plain&from=blabla#blabla.com&to=chica#chicas.com&subject=A problem occured while setting up new teaser!" />
</otherwise>
</choice>
<bean ref="teaserService" method="updateTeaser" />
</route>
How to achieve such a behavior?
Thanks
The Camel file compoment is a batch consumer and adds properties to the exchange regarding the batch it is processing. You can test for the property CamelBatchComplete and if that is set to true, then call your bean.
If you want to proceed only after all files have been read, you must sample them somehow. This can be achieved using the aggregator pattern:
<route>
<from uri="file://src/data/aggregate-and-process?readLock=changed&delete=true&delay=60000" />
<aggregate strategyRef="aggregationStrategy" completionFromBatchConsumer="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="direct:sub" />
</aggregate>
</route>
<route>
<from uri="direct:sub" />
<!-- processing aggregated body -->
</route>
Please note that I set completionFromBatchConsumer="true". From the Camel documentation:
This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. [...] This can be used to aggregate all files consumed from a File endpoint in that given poll.