Apache Camel unzip, process and aggregate not completing - apache-camel

I am trying to process a zip file, which contains several files inside (all small, so working in memory is not a problem) that need to be transformed and zipped together again.
I managed to unzip, transform the files, but for some reason, the splitter is not completing, only using completionTimeout makes the aggregator to create the final zip archive.
Here's the route:
<route id="ZipFile">
<from uri="file:{{file.path.in}}?move=.done&moveFailed=.error&readLock=rename"/>
<setProperty propertyName="OriginalZipName">
<simple>${header.CamelFileName}</simple>
</setProperty>
<unmarshal>
<zipFile usingIterator="true"/>
</unmarshal>
<split streaming="true">
<simple>${body}</simple>
<log message="************ CamelSplitComplete = ${property.CamelSplitComplete}"/>
<to uri="direct:ProcessUnzippedFile"/>
<setHeader headerName="CamelFileName">
<simple>${property.OriginalZipName}</simple>
</setHeader>
<!-- Aggregate to zip -->
<aggregate strategyRef="zipAggregationStrategy" eagerCheckCompletion="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<completionPredicate>
<simple>${property.CamelSplitComplete}</simple>
</completionPredicate>
<setHeader headerName="CamelFileName">
<simple>${property.OriginalZipName}</simple>
</setHeader>
<to uri="file://{{file.path.out}}"/>
</aggregate>
</split>
</route>
Any idea what could be the issue?

The Camel Splitter provides a built- in aggregator, which makes it even easier to
aggregate split messages back into single outgoing message
You have to define the aggregation strategy in the split definition
<split strategyRef="zipAggregationStrategy">
Haven't tried it in XML but in Java DSL this is one of my examples that does exactly that.
.unmarshal(zipFile)
.split(bodyAs(Iterator.class),new ZipAggregationStrategy(true,true))
.streaming()
.stopOnException()
.to("direct:transform-ticket")
.end();

Related

Camel blueprint <setHeader headerName="CamelHttpPath" id="_setHeader1"><el>${header.uriPattern}</el></setHeader> waiting for dependencies

I have the following Camel Context.
<camelContext id="_camuatomicservicecontext" xmlns="http://camel.apache.org/schema/blueprint">
<route id="_camuatomicserviceroute1">
<from id="_from1" uri="direct-vm:camuatomicservice">
<description>accepts vm messages directly </description>
</from>
<log id="_log1" message="Camu Atomic Service body = ${body}, header= ${header.uriPattern}"/>
<!-- <to id="_to1" uri="restlet:protocol:localhost:8189/"/> -->
<setHeader headerName="api.key" id="_setHeader1">
<constant>replace later with properties api.key Does not matter for this poc</constant>
</setHeader>
<setHeader headerName="CamelHttpPath" id="_setOutHeader1">
<el>${header.uriPattern}</el>
</setHeader>
<to id="_to1" pattern="InOut" uri="netty4-http:http:localhost:8189/path"/>
<log id="_log2" message="CamuAtomicService Response body ${body}"/>
</route>
</camelContext>
From the documentation I expect the CamelHttpPath header to override the endpoint configuration "/path" such that calling Facade Services can pass the header.uriPattern in and dynamically change the resource they want to access. The bundle worked fine until I added the setHeader for CamelHttpPath and now getting "Waiting for dependencies." I assume I need to install a feature, but Simple EL in other bundles on that server work already so not sure what feature I need to install.
Instead of I used and it worked fine. The choice was farther down in the options.

How can I fail a camel-route when exec returns nonzero?

I'm processing files with a Camel route like this:
<route>
<from uri="file:inbox?delete=true"/>
<recipientList>
<simple>exec://process.sh?args=inbox/${file:name}</simple>
</recipientList>
<log message="processed ${file:name}: ${body.stdout} ${body.stderr}"/>
</route>
Now I'd like the route to fail when process.sh finishes with nonzero exit-code. I found ${headers.CamelExecExitValue} but don't really know what to do with it.
In the example above, the file should not get deleted when process.sh fails. In my actual use-case, the route consumes files from a JMS queue and I want the file to stay in the queue. I think this can be done with <transacted/> but need to know how to fail the route.
I found How to define exception to be thrown through ref in Apache Camel which in combination with CamelExecExitValue lets me abort this way:
<route>
<from uri="file:inbox?delete=true"/>
<to uri="exec://process.sh"/>
<choice>
<when>
<simple>
${headers.CamelExecExitValue} != 0
</simple>
<throwException exceptionType="java.lang.RuntimeException" message="failed importing ${file:name}: ${body.stdout} ${body.stderr}"/>
</when>
</choice>
<log message="processed ${file:name}"/>
</route>
A bit verbose for my taste but works fine.

Is it possible to read a file after receiving an event?

I'm using a ActiveMQ Broker with built-in Camel Routes. I want to read a file after an Event received.
<pseudo>
from Event A
read File XY
to Event B with Body from File XY
</pseuod>
I simple tried moving files from a temporary directory based on an event but only event B is written. In the Log file are no Exceptions or Error messages.
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<!-- You can use Spring XML syntax to define the routes here using the <route> element -->
<route>
<description>Example Camel Route</description>
<from uri="activemq:example.A"/>
<from uri="file://tmp/a?delete=true"/>
<to uri="file://tmp/b?overruleFile=copy-of-${file:name}"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
Update with working solution for single file:
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<!-- You can use Spring XML syntax to define the routes here using the <route> element -->
<route>
<description>Example Camel Route</description>
<from uri="activemq:example.A"/>
<pollEnrich>
<constant>file:///tmp/a?fileName=file1</constant>
</pollEnrich>
<log message="file content ${body}"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
You need to use Content Enrichers for this. This is exactly what you are looking for.
<route>
<from uri="activemq:example.A"/>
<pollEnrich>
<constant>file://tmp/a?delete=true</constant>
</pollEnrich>
<to uri="activemq:example.B"/>
</route>
Please be aware that for camel version 2.15 or older
pollEnrich does not access any data from the current Exchange which
means when polling it cannot use any of the existing headers you may
have set on the Exchange. For example you cannot set a filename in the
Exchange.FILE_NAME header and use pollEnrich to consume only that
file. For that you must set the filename in the endpoint URI.

Can we use multiple mutlicast in apache camel?

I have a requirement where i want to use mutlicast in Apache Camel for than single time in a single route. i.e Multicast within a multicast.
<routeContext id="myRoute" xmlns="http://camel.apache.org/schema/spring">
<route id="myRouteId">
<from uri="activemq:queue:{{XXXX.queue}}" />
....
<multicast parallelProcessing="true">
<pipeline>
##everything working fine here
</pipeline>
<pipeline>
<multicast>
<pipeline>
<log message="Inserting in database now"></log>
<transform>
<method ref="insertBean" method="myBatchInsertion"></method>
</transform>
<choice>
<when>
<simple>${in.header.myCount} == ${properties:batch.size} </simple>
<to uri="sql:{{sql.core.insertMyQuery}}?batch=true"></to>
<log message="Inserted rows ${body}"></log>
</when>
</choice>
</pipeline>
</multicast>
</pipeline>
</multicast>
</route>
</routeContext>
Is it possible to do that?
When i am trying to do that, my program is not getting executed successfully.
Is the unsuccessful execution is a result of mulitple multicast?
Can anybody help?
I got the reference from following link:
http://camel.apache.org/multicast.html
Why do you use pipeline? It "is" pipeline by default.
Also all the log and transform and choice statements can be put outside of the multicast. And since you are generating your endpoints dynamically, put the values in a header and use recipientlist for dynamic endpoints. Multicast is for hard-coded endpoints. Here is an example:
<route>
<from uri="direct:a" />
<!-- use comma as a delimiter for String based values -->
<recipientList delimiter=",">
<header>myHeader</header>
</recipientList>
</route>
http://camel.apache.org/recipient-list.html

Apache Camel - Aggregate different object types

I have a route that, when lauched, requests messages from two datasources (routeA and routeB) and aggregates them into a single message. Each aggregated message MUST contain exactly one routeA message and one routeB message, if not, then drop it.
This process must be launched at specific intervals (i.e: every 5 min).
My question is, how can I let the aggregator know that all messages from routeA and routeB where processed and the messages that didn't find their pair, must be droped?
I'm currently using completionTimeout feature, but I don't like this solution for obvious reasons.
I know camel has a completionFromBatchConsumer feature, but I don't know how to use it with with multiple datasets.
Am greatefull for any advice.
Here's what I have right now:
<!-- main route -->
<route id="main">
<camel:from uri="timer://timer1?period=20000"/>
<multicast>
<to uri="direct:startA"/>
<to uri="direct:startB"/>
</multicast>
</route>
<!-- messages from route A -->
<route id="routeA" />
<from uri="direct:startA" />
<to uri="sql:select * from sampleDB?dataSource=ds"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectAJsonConverter"/>
<unmarshal ref="ObjectAJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectACorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- messages from route B -->
<route id="routeB" />
<from uri="direct:startB"/>
<to uri="ldap:ldapcontext?base=DC=company,DC=net"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectBJsonConverter"/>
<unmarshal ref="ObjectBJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectBCorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- aggregate the messages, create new ObjectC that contains ObjectA and ObjectB -->
<!-- wait 200000 ms for all messages from routeA and routeB to enter the aggregator -->
<route id="aggretatorRoute">
<from uri="seda:myAggregator"/>
<aggregate ref="myEntityAggregator" completionSize="2" completionTimeout="200000" discartOnCompletionTimeout="true" ignoreInvalidCorrelationKeys="true">
<correlationExpression><simple>${in.header.objectid}</simple></correlationExpression>
<to uri="bean:myProcessor?method=doSomethingWithObjectC"/>
</aggregate>
You can just in your AggregationStrategy only aggregate one of ObjectA and one of ObjectB. So if you see a 2nd of either of them, then just not aggregate it. And if you then want to drop what you have done so far, then you can mark the exchange to stop, by setting
exchange.setProperty("CamelRouteStop", true);
And if you then want to drop this immediately, then add a completionPredicate, that checks if that stop has been set.
<completionPredicate><simple>${property.CamelRouteStop} == true</simple></completionPredicate>
And for the correlationExpression, you can likely just use <constant>true</constant> as it seems you only work on one group.
Thanks Claus. Actually I took a different approach. Instead using an aggregator to join three different object, I now query a list of ids and use those to progressively build my complex object.
<route id="composeObject">
<from uri="sql:select id from people?oneSource">
<split><simple>${body}</simple>
<to uri="direct:getobjectOne"/>;
<to uri="bean:addToComplexObject"/>
<to uri="direct:getObjectTwo/>
<to uri="bean:addToComplexObject"/>
<to uri="direct:getobjectThree/>
<to uri="bean:addToComplexObject"/>
<to uri="seda:outChannel"/>
</split>
</route>

Resources