I'm trying to pick a file from a directory, split a file and add each splitted lines to activemq. I'm facing a problem with exception handling during this process. Lets say a file in the directory is a binary file (executable), then splitter thows org.apache.camel.RuntimeCamelException and java.nio.charset.MalformedInputException exceptions. If this occurs, then I need to catch these exceptions, nothing should add in activemq and that just particular thread should exit after logging exception. Referred online and wrote the following code but don't know how to stop adding to activemq and quit specific thread.
<route id="msg_producer">
<from uri="input.file.from" />
<doTry>
<split parallelProcessing="true" executorServiceRef="msgProducer"
streaming="true">
<tokenize token="\n"></tokenize>
<to uri="input.activemq.to" />
</split>
<doCatch>
<exception>org.apache.camel.RuntimeCamelException</exception>
<exception>java.nio.charset.MalformedInputException</exception>
<handled> <constant>true</constant></handled>
<setBody>
<simple>${exception.stacktrace}</simple>
</setBody>
<setHeader headerName="CamelFileName">
<simple>${file:onlyname.noext}_error.log</simple>
</setHeader>
</doCatch>
</doTry>
</route>
as #claus Ibsen said to filter the files you can use a filefilter property so that you pick only files based on extension and some standard pattern something like this
<bean id="FileFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
<!-- ? matches one character
* matches zero or more characters
** matches zero or more directories in a path -->
<property name="includes" value="#{databaseProperties.getProperties().getProperty('file.name.pattern')}"/>
<!-- if you wan to exclude specific files say bad in name or .exe files. Use comma to separate multiple excludes -->
<!-- <property name="excludes" value="**/*bad*,**/*.exe"/> -->
</bean>
and your file.name.pattern can be something like this **/contract.csv
Related
I'm using a ActiveMQ Broker with built-in Camel Routes. I want to read a file after an Event received.
<pseudo>
from Event A
read File XY
to Event B with Body from File XY
</pseuod>
I simple tried moving files from a temporary directory based on an event but only event B is written. In the Log file are no Exceptions or Error messages.
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<!-- You can use Spring XML syntax to define the routes here using the <route> element -->
<route>
<description>Example Camel Route</description>
<from uri="activemq:example.A"/>
<from uri="file://tmp/a?delete=true"/>
<to uri="file://tmp/b?overruleFile=copy-of-${file:name}"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
Update with working solution for single file:
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<!-- You can use Spring XML syntax to define the routes here using the <route> element -->
<route>
<description>Example Camel Route</description>
<from uri="activemq:example.A"/>
<pollEnrich>
<constant>file:///tmp/a?fileName=file1</constant>
</pollEnrich>
<log message="file content ${body}"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
You need to use Content Enrichers for this. This is exactly what you are looking for.
<route>
<from uri="activemq:example.A"/>
<pollEnrich>
<constant>file://tmp/a?delete=true</constant>
</pollEnrich>
<to uri="activemq:example.B"/>
</route>
Please be aware that for camel version 2.15 or older
pollEnrich does not access any data from the current Exchange which
means when polling it cannot use any of the existing headers you may
have set on the Exchange. For example you cannot set a filename in the
Exchange.FILE_NAME header and use pollEnrich to consume only that
file. For that you must set the filename in the endpoint URI.
I am trying to process a zip file, which contains several files inside (all small, so working in memory is not a problem) that need to be transformed and zipped together again.
I managed to unzip, transform the files, but for some reason, the splitter is not completing, only using completionTimeout makes the aggregator to create the final zip archive.
Here's the route:
<route id="ZipFile">
<from uri="file:{{file.path.in}}?move=.done&moveFailed=.error&readLock=rename"/>
<setProperty propertyName="OriginalZipName">
<simple>${header.CamelFileName}</simple>
</setProperty>
<unmarshal>
<zipFile usingIterator="true"/>
</unmarshal>
<split streaming="true">
<simple>${body}</simple>
<log message="************ CamelSplitComplete = ${property.CamelSplitComplete}"/>
<to uri="direct:ProcessUnzippedFile"/>
<setHeader headerName="CamelFileName">
<simple>${property.OriginalZipName}</simple>
</setHeader>
<!-- Aggregate to zip -->
<aggregate strategyRef="zipAggregationStrategy" eagerCheckCompletion="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<completionPredicate>
<simple>${property.CamelSplitComplete}</simple>
</completionPredicate>
<setHeader headerName="CamelFileName">
<simple>${property.OriginalZipName}</simple>
</setHeader>
<to uri="file://{{file.path.out}}"/>
</aggregate>
</split>
</route>
Any idea what could be the issue?
The Camel Splitter provides a built- in aggregator, which makes it even easier to
aggregate split messages back into single outgoing message
You have to define the aggregation strategy in the split definition
<split strategyRef="zipAggregationStrategy">
Haven't tried it in XML but in Java DSL this is one of my examples that does exactly that.
.unmarshal(zipFile)
.split(bodyAs(Iterator.class),new ZipAggregationStrategy(true,true))
.streaming()
.stopOnException()
.to("direct:transform-ticket")
.end();
I wrote the following route and expected that the bean 'teaserService' should be called only one time, at the end of processing of all files, but ... it's called after processing of each file:
<route id="teaserInterface">
<from
uri="file://{{teaser.dropInDir}}?readLock=changed&delete=true&delay=60000" />
<choice>
<when>
<simple>${file:ext} == 'properties'</simple>
<to uri="file://{{teaser.config.directory}}" />
</when>
<when>
<simple>${file:ext} == 'jpg' || ${file:ext} == 'JPG'</simple>
<to uri="sftp://{{apache.ftp.user}}#{{apache.ftp.host}}/{{apache.teaser.ftp.targetDir}}?password={{apache.ftp.password}}&binary=true&separator=UNIX" />
</when>
<otherwise>
<transform>
<simple>Dear user,\n\n the Teaser interface only accept *.jpg and *.properties files, but we found the file ${file.name}.\n\n Have a nice day,\nYour lovely Teaser interface</simple>
</transform>
<to
uri="smtp://smtp.blabla.com?contentType=text/plain&from=blabla#blabla.com&to=chica#chicas.com&subject=A problem occured while setting up new teaser!" />
</otherwise>
</choice>
<bean ref="teaserService" method="updateTeaser" />
</route>
How to achieve such a behavior?
Thanks
The Camel file compoment is a batch consumer and adds properties to the exchange regarding the batch it is processing. You can test for the property CamelBatchComplete and if that is set to true, then call your bean.
If you want to proceed only after all files have been read, you must sample them somehow. This can be achieved using the aggregator pattern:
<route>
<from uri="file://src/data/aggregate-and-process?readLock=changed&delete=true&delay=60000" />
<aggregate strategyRef="aggregationStrategy" completionFromBatchConsumer="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="direct:sub" />
</aggregate>
</route>
<route>
<from uri="direct:sub" />
<!-- processing aggregated body -->
</route>
Please note that I set completionFromBatchConsumer="true". From the Camel documentation:
This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. [...] This can be used to aggregate all files consumed from a File endpoint in that given poll.
I have a route that, when lauched, requests messages from two datasources (routeA and routeB) and aggregates them into a single message. Each aggregated message MUST contain exactly one routeA message and one routeB message, if not, then drop it.
This process must be launched at specific intervals (i.e: every 5 min).
My question is, how can I let the aggregator know that all messages from routeA and routeB where processed and the messages that didn't find their pair, must be droped?
I'm currently using completionTimeout feature, but I don't like this solution for obvious reasons.
I know camel has a completionFromBatchConsumer feature, but I don't know how to use it with with multiple datasets.
Am greatefull for any advice.
Here's what I have right now:
<!-- main route -->
<route id="main">
<camel:from uri="timer://timer1?period=20000"/>
<multicast>
<to uri="direct:startA"/>
<to uri="direct:startB"/>
</multicast>
</route>
<!-- messages from route A -->
<route id="routeA" />
<from uri="direct:startA" />
<to uri="sql:select * from sampleDB?dataSource=ds"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectAJsonConverter"/>
<unmarshal ref="ObjectAJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectACorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- messages from route B -->
<route id="routeB" />
<from uri="direct:startB"/>
<to uri="ldap:ldapcontext?base=DC=company,DC=net"/>
<split>
<simple>${body}<simple>
<marshal ref="ObjectBJsonConverter"/>
<unmarshal ref="ObjectBJsonConverter"/>
<to uri="bean:myProcessor?method=addObjectBCorrelationKey"/>
<to uri="seda:myAggregator"/>
</split>
<!-- aggregate the messages, create new ObjectC that contains ObjectA and ObjectB -->
<!-- wait 200000 ms for all messages from routeA and routeB to enter the aggregator -->
<route id="aggretatorRoute">
<from uri="seda:myAggregator"/>
<aggregate ref="myEntityAggregator" completionSize="2" completionTimeout="200000" discartOnCompletionTimeout="true" ignoreInvalidCorrelationKeys="true">
<correlationExpression><simple>${in.header.objectid}</simple></correlationExpression>
<to uri="bean:myProcessor?method=doSomethingWithObjectC"/>
</aggregate>
You can just in your AggregationStrategy only aggregate one of ObjectA and one of ObjectB. So if you see a 2nd of either of them, then just not aggregate it. And if you then want to drop what you have done so far, then you can mark the exchange to stop, by setting
exchange.setProperty("CamelRouteStop", true);
And if you then want to drop this immediately, then add a completionPredicate, that checks if that stop has been set.
<completionPredicate><simple>${property.CamelRouteStop} == true</simple></completionPredicate>
And for the correlationExpression, you can likely just use <constant>true</constant> as it seems you only work on one group.
Thanks Claus. Actually I took a different approach. Instead using an aggregator to join three different object, I now query a list of ids and use those to progressively build my complex object.
<route id="composeObject">
<from uri="sql:select id from people?oneSource">
<split><simple>${body}</simple>
<to uri="direct:getobjectOne"/>;
<to uri="bean:addToComplexObject"/>
<to uri="direct:getObjectTwo/>
<to uri="bean:addToComplexObject"/>
<to uri="direct:getobjectThree/>
<to uri="bean:addToComplexObject"/>
<to uri="seda:outChannel"/>
</split>
</route>
I'm trying to make a route that will process big xml files using camel-stax. A file content processing works fine, but at the end it fails with a following error:
Caused by: java.io.IOException: Renaming file from: C:\workdir\file.xml to: C:\workdir\.camel\file.xml failed due cannot delete from file: C:\workdir\file.xml after copy succeeded
It seems that camel doesn't close a file input stream, so after processing it cannot move a file to a target location. Of course, I can set noop=true, bit I wanted to remove processed files.
My route looks like following:
<route id="myRoute">
<from uri="file:{{working_dir}}?include=file.xml" />
<split streaming="true">
<ref>staxRecord</ref>
<to uri="log:test"/>
</split>
</route>
Initially it was a little bit more complex and I simplified it as possible. Now it looks just like a last sample from here http://camel.apache.org/stax.
Additional note: I execute the route on Windows. Camel version: 2.12.2.
So it looks like a bug in the camel-stax component.
I've found an alternative way of how to deal with big xml files. I've rewritten my route as following:
<route id="myRoute">
<from uri="file:{{working_dir}}?include=file.xml&delete=true" />
<split streaming="true">
<tokenize token="entry" xml="true"/>
<unmarshal ref="myJaxb"/>
<!-- ... -->
</split>
</route>