How to call a bean after all files has been processed? - apache-camel

I wrote the following route and expected that the bean 'teaserService' should be called only one time, at the end of processing of all files, but ... it's called after processing of each file:
<route id="teaserInterface">
<from
uri="file://{{teaser.dropInDir}}?readLock=changed&delete=true&delay=60000" />
<choice>
<when>
<simple>${file:ext} == 'properties'</simple>
<to uri="file://{{teaser.config.directory}}" />
</when>
<when>
<simple>${file:ext} == 'jpg' || ${file:ext} == 'JPG'</simple>
<to uri="sftp://{{apache.ftp.user}}#{{apache.ftp.host}}/{{apache.teaser.ftp.targetDir}}?password={{apache.ftp.password}}&binary=true&separator=UNIX" />
</when>
<otherwise>
<transform>
<simple>Dear user,\n\n the Teaser interface only accept *.jpg and *.properties files, but we found the file ${file.name}.\n\n Have a nice day,\nYour lovely Teaser interface</simple>
</transform>
<to
uri="smtp://smtp.blabla.com?contentType=text/plain&from=blabla#blabla.com&to=chica#chicas.com&subject=A problem occured while setting up new teaser!" />
</otherwise>
</choice>
<bean ref="teaserService" method="updateTeaser" />
</route>
How to achieve such a behavior?
Thanks

The Camel file compoment is a batch consumer and adds properties to the exchange regarding the batch it is processing. You can test for the property CamelBatchComplete and if that is set to true, then call your bean.

If you want to proceed only after all files have been read, you must sample them somehow. This can be achieved using the aggregator pattern:
<route>
<from uri="file://src/data/aggregate-and-process?readLock=changed&delete=true&delay=60000" />
<aggregate strategyRef="aggregationStrategy" completionFromBatchConsumer="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="direct:sub" />
</aggregate>
</route>
<route>
<from uri="direct:sub" />
<!-- processing aggregated body -->
</route>
Please note that I set completionFromBatchConsumer="true". From the Camel documentation:
This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. [...] This can be used to aggregate all files consumed from a File endpoint in that given poll.

Related

Multi File Download Not Working in Apache Camel

I am using quartz and poll enrich to download multiple files at once , one of the configuration used is localWorkDirectory=/tmp/sftp_tmp/ . I see in logs that says
org.apache.camel.component.file.remote.SftpConsumer.processExchange - About to process file: RemoteFile[filename] . But it randomly sometimes places the file in /tmp/sftp_tmp/ instead of the at final destination c:/cameldata at times. I am using camel version 2.24.1
<route id="sftp-read">
<from uri="quartz2://sftp?stateful=true&trigger.repeatInterval=3s"/>
<pollEnrich><spel>file:c:/cameldata?include=SFTPRECFILE&noop=true&idempotent=false&readLock=markerFile</spel></pollEnrich>
<to uri="seda:startSftpExecutor?waitForTaskToComplete=Always&timeout=-1"/>
</route>
<route id="sftp-executor">
<from uri="seda:startSftpExecutor" />
<pollEnrich><spel>{{sftpUrl}}&sendEmptyMessageWhenIdle=true&binary=false&include=BR-.*&maximumReconnectAttempts=3&noop=true&maxMessagesPerPoll=1&localWorkDirectory=/tmp/sftp_tmp/&stepwise=false</spel></pollEnrich>
<choice>
<when>
<simple>${in.body} != null</simple>
<to uri="file:c:/cameldata?fileName=${file:onlyname}&tempPrefix=.tmp"/>
</when>
<otherwise>
<setBody><constant></constant></setBody>
<log loggingLevel="INFO" message="file(s) downloaded successfully" />
<to uri="file:c:/cameldata?fileName=LOADFILE" />
</otherwise>
</choice>
</route>

Terminate the current camel exchange

I am processing file in cluster environment. The cluster works fine. It is being processed on Only one server.
But on the second server It identifies as duplicates but still execute the form route delete=true
ERROR:
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file:
I am setting header CamelRouteStop to true but the exchange still try's to delete a file, instead of stop executing the route.
All I need is to end the route if it is duplicate.
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="smb:url?delete=true"/>
<idempotentConsumer messageIdRepositoryRef="myRepo">
<header>messageId</header>
<setHeader headerName="fileExist">
<simple>true</simple>
</setHeader>
</idempotentConsumer>
<when>
<simple>${header.fileExist} == null</simple>
<log message="File ${header.CamelFileName} processing/processed by other Nodes - DUPLICATE" loggingLevel="INFO" />
<setHeader headerName="CamelRouteStop">
<simple <simple resultType="java.lang.Boolean">true</simple>>true</simple>
</setHeader>
</when>
</route>
</camelContext>
For CamelRouteStop you need to use setProperty, not setHeader.

How to add scheduler and custome checks around camel file processing

Before explaining my problem I would state here that I am completely new to camel file processing. I have a requirement to read the file from a directory do some processing and delete them. This was a very high level requirement and I am able to achieve this using camel. But now I've got some new requirements as stated below. Need help on that.
Create this application as a job and trigger it by reading another directory where some specific files would be dropped other wise it should kicked of by its own every 15-20 minutes.
Before triggering the actual application make sure that the directory has some specific number of files present (say 25 files)
If all files are present - execute a method to create a unique tracking ID for all these 25 files. If I have a unique ID how can I make it available through multiple routes?
As of now I have tried implementing routepolicy but since I have never used it earlier I need some guidance so that I can go ahead with this.
1. Separe your route logic from route triggering
<route id="TriggerFromFile">
<from uri="file:triggerFolder" />
<log message="Triggered from file" />
<to uri="direct:startLogic" />
<route>
<route id="TriggerFromTimer">
<from uri="timer:triggerTimer?period=15m" />
<log message="Triggered from timer" />
<to uri="direct:startLogic" />
</route>
<route id="Logic>
<from uri="direct:startLogic" />
<to uri="..." />
</route>
2. Count the number of files and use that as a filter
Define a bean that counts the number of files in the dir, set that number
as body and validate using a filter.
<route id="TriggerFromFile">
<from uri="file:triggerFolder" />
<log message="Triggered from file" />
<to uri="direct:countFile" />
<route>
<route id="TriggerFromTimer">
<from uri="timer:triggerTimer?period=15m" />
<log message="Triggered from timer" />
<to uri="direct:countFile" />
</route>
<route id="FileCount">
<from uri="direct:countFile" />
<to uri="bean:countFilesInDir" />
<log message="There are ${body} files the directory" />
<filter>
<simple>${body} >= 25</simple>
<to uri="direct:startLogic" />
</filter>
</route>
<route id="Logic">
<from uri="direct:startLogic" />
<to uri="..." />
</route>
3. Set a Header to your Exchange's message before sending it to other routes
When Camel sends an Exchange between routes, headers and properties are copied.
Calculate a unique id in some way (concatenate file names, md5 of content, file modification timestamp....) and set it in a Header.
An header can hold any java object.

Can we use multiple mutlicast in apache camel?

I have a requirement where i want to use mutlicast in Apache Camel for than single time in a single route. i.e Multicast within a multicast.
<routeContext id="myRoute" xmlns="http://camel.apache.org/schema/spring">
<route id="myRouteId">
<from uri="activemq:queue:{{XXXX.queue}}" />
....
<multicast parallelProcessing="true">
<pipeline>
##everything working fine here
</pipeline>
<pipeline>
<multicast>
<pipeline>
<log message="Inserting in database now"></log>
<transform>
<method ref="insertBean" method="myBatchInsertion"></method>
</transform>
<choice>
<when>
<simple>${in.header.myCount} == ${properties:batch.size} </simple>
<to uri="sql:{{sql.core.insertMyQuery}}?batch=true"></to>
<log message="Inserted rows ${body}"></log>
</when>
</choice>
</pipeline>
</multicast>
</pipeline>
</multicast>
</route>
</routeContext>
Is it possible to do that?
When i am trying to do that, my program is not getting executed successfully.
Is the unsuccessful execution is a result of mulitple multicast?
Can anybody help?
I got the reference from following link:
http://camel.apache.org/multicast.html
Why do you use pipeline? It "is" pipeline by default.
Also all the log and transform and choice statements can be put outside of the multicast. And since you are generating your endpoints dynamically, put the values in a header and use recipientlist for dynamic endpoints. Multicast is for hard-coded endpoints. Here is an example:
<route>
<from uri="direct:a" />
<!-- use comma as a delimiter for String based values -->
<recipientList delimiter=",">
<header>myHeader</header>
</recipientList>
</route>
http://camel.apache.org/recipient-list.html

How to process the same message sequentially in Camel?

I am new to Camel and I am trying to do the following.
I want to process the same message twice. I have to transform both messages and process one message first and then second only if the first one is sucessfully executed (I have a condition).
I tried to use a multicast first. Then I transform the messages in each route. The first one (the operation_DC) only sends a successfull message to the second one (the operation_AC) only if it is sucessfull. The second operation contains an aggregation that waits until it times out for both messages. I only want to process the messaged from the multicast and discard the other.
<route id="t_operation_ME">
<from uri="direct:operation_ME" id="t.direct.aggregatedService"></from>
<setHeader headerName="id">
<simple>exchangeId</simple>
</setHeader>
<multicast parallelProcessing="true" strategyRef="defaultAggregationStrategy" stopOnException="true" onPrepareRef="cleanHeader" parallelAggregate="false" completionPredicate="">
<to uri="direct:operation_DC"></to>
<to uri="direct:operation_AC"></to>
</multicast>
</route>
<route id="direct:operation_DC">
<from uri="direct:operation_DC" />
<log message="ENTER DC"></log>
<to uri="xslt:{{depasse:core.transformation.xml.path}}client/t/Operation_toDC_request.xsl" id="t.dc.transform.productos" />
<to uri="activemq:QCIn" id="t.dc.qcin.queue.send"></to>
<log message="EXIT DC ${body}"></log>
<choice>
<when>
<xpath>//Data/Status[. = 'OK']</xpath>
<log message="SEND TO AC"></log>
<to uri="direct:operation_AC"></to>
</when>
</choice>
</route>
<route id="direct:operation_AC">
<from uri="direct:operation_AC" />
<log message="ENTER AC"></log>
<aggregate completionTimeout="20000" completionSize="2" discardOnCompletionTimeout="true" strategyRef="tAggregationStrategy">
<correlationExpression>
<simple>header.id</simple>
</correlationExpression>
<log message="ENTER AGG AC ${body}"></log>
<to uri="xslt:{{depasse:core.transformation.xml.path}}client/t/Operation_toAC_request.xsl" id="t.ac.transform.productos" />
<to uri="activemq:QCIn" id="t.ac.qcin.queue.send"></to>
<log message="EXIT AC ${body}"></log>
</aggregate>
<log message="END AC\n${body}"></log>
</route>
The thing is that when I log "EXIT AC" and "END AC" the message is different. This means that while in the server I am watching the process correctly, the AC client receives an incorrect message.
Multicast send copy of messages always to every endpoint between tags. If you want to call route operation_AC only after operation_DC run successfully, you shouldn't use multicast, but you should save your body at the beginning of operation_DC, for example using property and Camel Simple:
<from uri="direct:operation_DC" />
<setProperty propertyName="body">
<simple>${body}</simple>
</setProperty>
and before calling operation_AC set body to bring message body from the beginning:
<setBody>
<simple>${property.body}</simple>
</setBody>
<to uri="direct:operation_AC"/>

Resources