How to add scheduler and custome checks around camel file processing - file

Before explaining my problem I would state here that I am completely new to camel file processing. I have a requirement to read the file from a directory do some processing and delete them. This was a very high level requirement and I am able to achieve this using camel. But now I've got some new requirements as stated below. Need help on that.
Create this application as a job and trigger it by reading another directory where some specific files would be dropped other wise it should kicked of by its own every 15-20 minutes.
Before triggering the actual application make sure that the directory has some specific number of files present (say 25 files)
If all files are present - execute a method to create a unique tracking ID for all these 25 files. If I have a unique ID how can I make it available through multiple routes?
As of now I have tried implementing routepolicy but since I have never used it earlier I need some guidance so that I can go ahead with this.

1. Separe your route logic from route triggering
<route id="TriggerFromFile">
<from uri="file:triggerFolder" />
<log message="Triggered from file" />
<to uri="direct:startLogic" />
<route>
<route id="TriggerFromTimer">
<from uri="timer:triggerTimer?period=15m" />
<log message="Triggered from timer" />
<to uri="direct:startLogic" />
</route>
<route id="Logic>
<from uri="direct:startLogic" />
<to uri="..." />
</route>
2. Count the number of files and use that as a filter
Define a bean that counts the number of files in the dir, set that number
as body and validate using a filter.
<route id="TriggerFromFile">
<from uri="file:triggerFolder" />
<log message="Triggered from file" />
<to uri="direct:countFile" />
<route>
<route id="TriggerFromTimer">
<from uri="timer:triggerTimer?period=15m" />
<log message="Triggered from timer" />
<to uri="direct:countFile" />
</route>
<route id="FileCount">
<from uri="direct:countFile" />
<to uri="bean:countFilesInDir" />
<log message="There are ${body} files the directory" />
<filter>
<simple>${body} >= 25</simple>
<to uri="direct:startLogic" />
</filter>
</route>
<route id="Logic">
<from uri="direct:startLogic" />
<to uri="..." />
</route>
3. Set a Header to your Exchange's message before sending it to other routes
When Camel sends an Exchange between routes, headers and properties are copied.
Calculate a unique id in some way (concatenate file names, md5 of content, file modification timestamp....) and set it in a Header.
An header can hold any java object.

Related

Multi File Download Not Working in Apache Camel

I am using quartz and poll enrich to download multiple files at once , one of the configuration used is localWorkDirectory=/tmp/sftp_tmp/ . I see in logs that says
org.apache.camel.component.file.remote.SftpConsumer.processExchange - About to process file: RemoteFile[filename] . But it randomly sometimes places the file in /tmp/sftp_tmp/ instead of the at final destination c:/cameldata at times. I am using camel version 2.24.1
<route id="sftp-read">
<from uri="quartz2://sftp?stateful=true&trigger.repeatInterval=3s"/>
<pollEnrich><spel>file:c:/cameldata?include=SFTPRECFILE&noop=true&idempotent=false&readLock=markerFile</spel></pollEnrich>
<to uri="seda:startSftpExecutor?waitForTaskToComplete=Always&timeout=-1"/>
</route>
<route id="sftp-executor">
<from uri="seda:startSftpExecutor" />
<pollEnrich><spel>{{sftpUrl}}&sendEmptyMessageWhenIdle=true&binary=false&include=BR-.*&maximumReconnectAttempts=3&noop=true&maxMessagesPerPoll=1&localWorkDirectory=/tmp/sftp_tmp/&stepwise=false</spel></pollEnrich>
<choice>
<when>
<simple>${in.body} != null</simple>
<to uri="file:c:/cameldata?fileName=${file:onlyname}&tempPrefix=.tmp"/>
</when>
<otherwise>
<setBody><constant></constant></setBody>
<log loggingLevel="INFO" message="file(s) downloaded successfully" />
<to uri="file:c:/cameldata?fileName=LOADFILE" />
</otherwise>
</choice>
</route>

camel when/otherwise for activemq and seda

I have a route where I'd like to compute if "from" is activemq or something else. I found replaceFromWith, but it seems to be used for testing only. I tried using camel's choice when/otherwise to switch between "from activemq" and "from seda", but that errors out as invalid syntax. I'm looking for something that would do the following:
<route id="doPost">
<choice>
<when>
<groovy>exchange.getIn().getHeader("sometest") != null</groovy>
<from uri="activemq:doPost?..."/>
</when>
<otherwise>
<from uri="seda:doPost?....."/>
</otherwise>
</choice>
Thanks in advance, appreciate the help.
You cannot dynamically select a consumer based on the exchange, the exchange wont exist before consuming it using the <from .../> tag.
Instead, you can construct a route for each consumer, normalize the exchange you receive from different types of consumers, and forward them to a common route for processing.
<route id="activemqConsumerRoute">
<from uri="activemq:doPost?..."/>
<!-- normalize the exchange to be understandable by the next route -->
<to uri="direct:commonProcessingRoute" />
</route>
<route id="sedaConsumerRoute">
<from uri="seda:doPost?....."/>
<!-- normalize the exchange to be understandable by the next route -->
<to uri="direct:commonProcessingRoute" />
</route>
<route id="commonProccessingRoute">
<from uri="direct:commonProcessingRoute" />
<!-- whatever business logic you have -->
</route>

How to call URL from Apache Camel in blueprint and return the data to a text file?

I am using Apache Camel blueprints, my route is triggered from a URL at port 8081 on localhost. This route generates a file but the problem is the file is a binary file without the HTML showing.
I then navigate my browser to http://localhost:8081/foo which triggers the URL.
What is it that I don't understand that is causing it to output a binary file rather than a text file? I guess that I must transform the body somehow?
<route id="url1">
<from uri="netty4-http:http://0.0.0.0:8081/foo" />
<to uri="http://www.google.com/?bridgeEndpoint=true" />
<to uri="file:/test"/>
</route>
Update: I think the problem is that the content coming back from www.google.com is gzip. When I look at the bridgeEndpoint parameter in the documentation it mentions something about gzip... now I am not sure, as I tried it on another website and it still doesn't work.
This line in my log might be relevant.
writing body: DefaultFullHttpResponse(decodeResult: success,
version: HTTP/1.1, content: UnpooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0))
Update: I discover if I do:
<from uri="timer:secondfoo?period=20s" />
Replacing the from, then it works. Hmm... Something flowing from the netty4-http causes problem.
Update: I found something which works! Obsession pays off.
<route id="url1">
<from uri="netty-http:http://0.0.0.0:8081/foo" />
<removeHeaders pattern="*" />
<setBody>
<simple></simple>
</setBody>
<setHeader headerName="CamelHttpMethod">
<constant>GET</constant>
</setHeader>
<to uri="http://www.google.com/?bridgeEndpoint=true" />
<to uri="file:/test"/>
</route>
You can convert the message to a String type then its text based
<route id="url1">
<from uri="netty4-http:http://0.0.0.0:8081/foo" />
<to uri="http://www.google.com/?bridgeEndpoint=true" />
<convertBodyTo type="String"/>
<to uri="file:/test"/>
</route>

Apache camel: Multiple executions with onCompletion mode="BeforeConsumer"

Sorry for my english, I'll make my best effort for write this.
I have a xml file like this:
<routes>
<route id="id1">
<from uri="timer://foo?fixedRate=true&period=60s"/>
<onCompletion mode="BeforeConsumer">
<log message="finish"/>
</onCompletion>
<setBody>
<simple>Probando 1</simple>
</setBody>
<log message="working1"/>
<to uri="direct:test1"/>
</route>
<route id="id2">
<from uri="direct:test1"/>
<setBody>
<simple>Test2</simple>
</setBody>
<log message="working2"/>
<to uri="direct:test2"/>
</route>
<route id="id3">
<from uri="direct:test2"/>
<setBody>
<simple>Test3</simple>
</setBody>
<log message="working3"/>
</route>
</routes>
if remove mode="BeforeConsumer" from the first route i get one "finish" in the log, but if i use the xml with mode="BeforeConsumer" i get the message 3 times (one per each route). Is that the behavior expected?
I need that the onCompletion code runs one time but i need too the mode="BeforeConsumer"

How to call a bean after all files has been processed?

I wrote the following route and expected that the bean 'teaserService' should be called only one time, at the end of processing of all files, but ... it's called after processing of each file:
<route id="teaserInterface">
<from
uri="file://{{teaser.dropInDir}}?readLock=changed&delete=true&delay=60000" />
<choice>
<when>
<simple>${file:ext} == 'properties'</simple>
<to uri="file://{{teaser.config.directory}}" />
</when>
<when>
<simple>${file:ext} == 'jpg' || ${file:ext} == 'JPG'</simple>
<to uri="sftp://{{apache.ftp.user}}#{{apache.ftp.host}}/{{apache.teaser.ftp.targetDir}}?password={{apache.ftp.password}}&binary=true&separator=UNIX" />
</when>
<otherwise>
<transform>
<simple>Dear user,\n\n the Teaser interface only accept *.jpg and *.properties files, but we found the file ${file.name}.\n\n Have a nice day,\nYour lovely Teaser interface</simple>
</transform>
<to
uri="smtp://smtp.blabla.com?contentType=text/plain&from=blabla#blabla.com&to=chica#chicas.com&subject=A problem occured while setting up new teaser!" />
</otherwise>
</choice>
<bean ref="teaserService" method="updateTeaser" />
</route>
How to achieve such a behavior?
Thanks
The Camel file compoment is a batch consumer and adds properties to the exchange regarding the batch it is processing. You can test for the property CamelBatchComplete and if that is set to true, then call your bean.
If you want to proceed only after all files have been read, you must sample them somehow. This can be achieved using the aggregator pattern:
<route>
<from uri="file://src/data/aggregate-and-process?readLock=changed&delete=true&delay=60000" />
<aggregate strategyRef="aggregationStrategy" completionFromBatchConsumer="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="direct:sub" />
</aggregate>
</route>
<route>
<from uri="direct:sub" />
<!-- processing aggregated body -->
</route>
Please note that I set completionFromBatchConsumer="true". From the Camel documentation:
This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. [...] This can be used to aggregate all files consumed from a File endpoint in that given poll.

Resources