Camel Multicast issue - apache-camel

We are using the below configuration to send a file from single source to multiple remote destinations.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camel="http://camel.apache.org/schema/spring"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<routeContext id="gcgRatesOutbound" xmlns="http://camel.apache.org/schema/spring">
<route id="gcgRatesFileOut">
<from uri="file:{{nas.root}}/{{gcg.out.prices.dir}}?delay={{poll.delay}}&initialDelay={{initial.delay}}&readLock=rename&scheduledExecutorService=#scheduledExecutorService" />
<multicast stopOnException="true">
<to uri="scp://{{gcg.ste.Client1_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client1}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client2_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client2}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client3_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client3}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
</multicast>
</route>
</routeContext>
</beans>
Using the above confirguration the file reaches the remote destinations, which confirmed that the connection to all the remote destinations were successfull.
We need that the file should be moved to the archive folder after the file has been successfully transfered to all the remote destinations.
And should move to error folder incase of any error.
However, when I add the archival code ( element) as child element to the multicast element in the above configuration and use the and the tag to move the file to error folder incase of an error. The file does not reach the remote destinations.
<doTry>
<multicast stopOnException="true" parallelProcessing="true">
<to uri="scp://{{gcg.ste.Client1_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client1}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client2_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client2}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client3_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client3}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="file://{{nas.root}}/{{gcg.out.prices.dir}}?fileName={{archive.dir}}" />
</multicast>
<doCatch>
<exception>java.lang.Exception</exception>
<handled>
<constant>true</constant>
</handled>
<to uri="file://{{nas.root}}/{{gcg.out.prices.dir}}?fileName={{error.dir}}" />
</doCatch>
</doTry>
The file does not reach the remote destinations nor does it produce any log and the file is moved to archive folder.
I tried placing the remote destinations, the archival code and the move to error code in seperate routes and have its reference in a single multicast as below
<to uri="direct:Client1FileOut" />
<to uri="direct:Client2FileOut" />
<to uri="direct:Client3FileOut" />
<to uri="direct:MoveToArchive" />
<route id="gcgFileOut1">
<from uri="direct:Client1FileOut" />
<doTry>
<to uri="scp://{{gcg.ste.Client1_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client1}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<doCatch>
<exception>java.lang.Exception</exception>
<handled>
<constant>true</constant>
</handled>
<to uri="direct:gcgError" />
</doCatch>
</doTry>
</route>
However, The file does not reach the remote destinations nor does it produce any log and the file is moved to archive folder.
I am new to camel.
I tried using the shareUnitOfWork attribute as below
<multicast shareUnitOfWork="true">
However, the file moved to archive folder along with Test_2_19082013_3.txt.camelLock file
Not sure why this file is also moved to archive when the file dropped was Test_2_19082013_3.txt
Also there is folder named "ARCHIV~1" created in the drop location.

Something might went wrong when moving the file to archive. You have stopOnException="true" parallelProcessing="true" and the local file is probably done first as it should be the fastest.
You probably want to print the error somewhere. Now, you catch the exception and mark the error as handled. Still you expect logs. Use the log component and output some log statements manually. Not only in case or error but also in case of success. You can log in the debug level so that you can manually enable log outprint in case you need it - like now.
Another option for you to figure out what's going on is to enable trace which will make you less "blind".
That said about debugging on your own -
You are archiving and storeing error messages in the input directory. The lock file you see is when Camel is reading. This is likely what's casuing your application to malfunction.
{{nas.root}}/{{gcg.out.prices.dir}}
The fileName=... is the filename, not another sub directory.
So: file://{{nas.root}}/{{gcg.out.prices.dir}}/{{archive.dir}} should do it (likewise for the error path).

Related

File not created in copying data to file in Apache Camel

I want to copy contents from one file in input folder to a new file in output folder using Apache camel xml. When this didn't work I tried to simply set the body and copy it to a new file. The log message is displayed correctly and there are no errors but no output file is created.
<route id="route1">
<from uri="timer://tenSecondsTimer?repeatCount=1"/>
<setBody>
<simple>Today is thursday</simple>
</setBody>
<log message="This is the message: ${body}"/>
<to uri="file:/output"/>
</route>
Also tried this syntax -
<to uri="file:///c:/Users/<route_to_output_folder>/output"/>
Code I tried for file to file transfer -
<route id="route1">
<from uri="file:/input/?fileName=file1.txt"/>
<log message="This is the message: ${body}"/>
<to uri="file:/output"/>
</route>

Terminate the current camel exchange

I am processing file in cluster environment. The cluster works fine. It is being processed on Only one server.
But on the second server It identifies as duplicates but still execute the form route delete=true
ERROR:
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file:
I am setting header CamelRouteStop to true but the exchange still try's to delete a file, instead of stop executing the route.
All I need is to end the route if it is duplicate.
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="smb:url?delete=true"/>
<idempotentConsumer messageIdRepositoryRef="myRepo">
<header>messageId</header>
<setHeader headerName="fileExist">
<simple>true</simple>
</setHeader>
</idempotentConsumer>
<when>
<simple>${header.fileExist} == null</simple>
<log message="File ${header.CamelFileName} processing/processed by other Nodes - DUPLICATE" loggingLevel="INFO" />
<setHeader headerName="CamelRouteStop">
<simple <simple resultType="java.lang.Boolean">true</simple>>true</simple>
</setHeader>
</when>
</route>
</camelContext>
For CamelRouteStop you need to use setProperty, not setHeader.

Exception handling Camel

I'm trying to pick a file from a directory, split a file and add each splitted lines to activemq. I'm facing a problem with exception handling during this process. Lets say a file in the directory is a binary file (executable), then splitter thows org.apache.camel.RuntimeCamelException and java.nio.charset.MalformedInputException exceptions. If this occurs, then I need to catch these exceptions, nothing should add in activemq and that just particular thread should exit after logging exception. Referred online and wrote the following code but don't know how to stop adding to activemq and quit specific thread.
<route id="msg_producer">
<from uri="input.file.from" />
<doTry>
<split parallelProcessing="true" executorServiceRef="msgProducer"
streaming="true">
<tokenize token="\n"></tokenize>
<to uri="input.activemq.to" />
</split>
<doCatch>
<exception>org.apache.camel.RuntimeCamelException</exception>
<exception>java.nio.charset.MalformedInputException</exception>
<handled> <constant>true</constant></handled>
<setBody>
<simple>${exception.stacktrace}</simple>
</setBody>
<setHeader headerName="CamelFileName">
<simple>${file:onlyname.noext}_error.log</simple>
</setHeader>
</doCatch>
</doTry>
</route>
as #claus Ibsen said to filter the files you can use a filefilter property so that you pick only files based on extension and some standard pattern something like this
<bean id="FileFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
<!-- ? matches one character
* matches zero or more characters
** matches zero or more directories in a path -->
<property name="includes" value="#{databaseProperties.getProperties().getProperty('file.name.pattern')}"/>
<!-- if you wan to exclude specific files say bad in name or .exe files. Use comma to separate multiple excludes -->
<!-- <property name="excludes" value="**/*bad*,**/*.exe"/> -->
</bean>
and your file.name.pattern can be something like this **/contract.csv

How to call a bean after all files has been processed?

I wrote the following route and expected that the bean 'teaserService' should be called only one time, at the end of processing of all files, but ... it's called after processing of each file:
<route id="teaserInterface">
<from
uri="file://{{teaser.dropInDir}}?readLock=changed&delete=true&delay=60000" />
<choice>
<when>
<simple>${file:ext} == 'properties'</simple>
<to uri="file://{{teaser.config.directory}}" />
</when>
<when>
<simple>${file:ext} == 'jpg' || ${file:ext} == 'JPG'</simple>
<to uri="sftp://{{apache.ftp.user}}#{{apache.ftp.host}}/{{apache.teaser.ftp.targetDir}}?password={{apache.ftp.password}}&binary=true&separator=UNIX" />
</when>
<otherwise>
<transform>
<simple>Dear user,\n\n the Teaser interface only accept *.jpg and *.properties files, but we found the file ${file.name}.\n\n Have a nice day,\nYour lovely Teaser interface</simple>
</transform>
<to
uri="smtp://smtp.blabla.com?contentType=text/plain&from=blabla#blabla.com&to=chica#chicas.com&subject=A problem occured while setting up new teaser!" />
</otherwise>
</choice>
<bean ref="teaserService" method="updateTeaser" />
</route>
How to achieve such a behavior?
Thanks
The Camel file compoment is a batch consumer and adds properties to the exchange regarding the batch it is processing. You can test for the property CamelBatchComplete and if that is set to true, then call your bean.
If you want to proceed only after all files have been read, you must sample them somehow. This can be achieved using the aggregator pattern:
<route>
<from uri="file://src/data/aggregate-and-process?readLock=changed&delete=true&delay=60000" />
<aggregate strategyRef="aggregationStrategy" completionFromBatchConsumer="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="direct:sub" />
</aggregate>
</route>
<route>
<from uri="direct:sub" />
<!-- processing aggregated body -->
</route>
Please note that I set completionFromBatchConsumer="true". From the Camel documentation:
This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. [...] This can be used to aggregate all files consumed from a File endpoint in that given poll.

Camel File component should not rename file when an exception is thrown

I have camel route configured as below:
<route id="text-file-route">
<from
uri="file:files/merchant?antInclude={{include-file-type}}&initialDelay=1000&delay=1000&move=${file:name.noext}.processed" />
<split streaming="true">
<tokenize token="\n" />
<process ref="splitBatchAdapterProcessor" />
<process ref="merchantStreamProcessor" />
<process ref="merchantTableProcessor" />
<to uri="mock:dummy" />
</split>
</route>
With current configuration the file gets renamed even if an exception is thrown while processing the file.
What i want is, the file should only be rename if no exception is thrown while the file.
I an using camel 2.12.
You need to turn on shareUnitOfWork so the splitter returns back the exception so the file consumer can rollback. You can read more about this at: http://camel.apache.org/splitter
<split streaming="true" shareUnitOfWork="true">

Resources