I am writing 2 routes to process a files in a directory, those files could have any name, but I need 2 routes as I need some complex processing.
First route:
<route id="Init">
<from uri="file:{{file.path}}?move=.done&moveFailed=.error&readLock=changed&readLockCheckInterval=1500&charset=UTF-8"/>
<transacted/>
<split streaming="true" stopOnException="true" shareUnitOfWork="true" parallelProcessing="false">
<tokenize token="\r\n"/>
<choice>
<when>
<simple>${body.substring(0,4)} == 4000</simple>
[...]
<to uri="file:{{file.path}}/tmp?fileName=${date:now:yyyyMMddss}.txt&fileExist=append&charset=UTF-8"/>
</when>
<when>
<simple>${body.substring(0,4)} == 4002</simple>
[...]
<to uri="file:{{file.path}}/tmp?fileName=${date:now:yyyyMMddss}.txt&fileExist=append&charset=UTF-8"/>
</when>
</choice>
</split>
</route>
Second route, which consumes the file produced by the first route:
<route id="End">
<from uri="file:{{file.path}}/tmp?delete=true&moveFailed=.error&readLock=changed&readLockCheckInterval=1500&charset=UTF-8"/>
<transacted/>
<split streaming="true" stopOnException="true" shareUnitOfWork="true" parallelProcessing="false">
<tokenize token="\r\n4000"/>
[...]
<to uri="[...]"/>
</split>
</route>
I am trying to make sure file produced by route Init won't be consumed by route End until the Init has finished processing the first file.
I guessed using a temp file extension, and then using an exlude on the second route, but it doesn't work with fileExists.
Any ideas?
Thanks!
Use done file
You need a mechanism to make sure the second route only consume file that have been completely processed by first route.
A simple method is to let first route emit a done file as a signal to tell second route that the file has been processed completed and is ready to pickup.
To use done file, you could add doneFileName parameter in first route when process completed and also add in the second route using same filename pattern.
For more details, please read Section "Using 'done' Files" of Camel File Component
You cannot use readLock=changed with the file component as it's only available for FTP/SFTP from Camel 2.8 onwards.
changed is using file length/modification timestamp to detect whether the file is currently being copied or not. Will at least use 1 sec. to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency. This option is only avail for the FTP component from Camel 2.8 onward. Note: from Camel 2.10.1 onward the FTP option fastExistsCheck can be enabled to speedup this readLock strategy, if the FTP server support the LIST operation with a full file name (some servers may not).
Try one of the other mechanisms such as markerFile, fileLock, or rename
Related
I have created multiple routes(say department, Employee) which takes input from file system folders(say department, Empployee) and process those files.
Now, I want to make them dependent. So, if I upload both emp.csv and dept.csv in those folders then it will process department file first and once complete it will start processing file for employee.
is there any way in camel to achieve this.
I looked at Route startupOrdering and AutoStartup feature, but it will work only for the first time when starting routes. However, I need same behavior for entire route life.
Thanks.
<route id="b" xmlns="http://camel.apache.org/schema/spring">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Department?repeatCount=1&noop=true&delay=10000"/>
<log message="Department data is : ${body}"/>
</route>
<route id="employee" xmlns="http://camel.apache.org/schema/spring">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Employee?noop=true&delay=10000"/>
<log message="Employee data is : ${body}"/>
</route>
I suggest to use other logic to handle the task. Two simple ways to go:
Use pollEnrich
Use pollEnrich to collect extra resource (e.g. a file with known name in file system) once at the middle of a route
Flow: Collect department files (From Endpoint) --(for each department file from file system) -> collect single employee file (trigger pollEnrich once with known name) ----> do anything else (if any)
Use ControlBus
Use ControlBus component to control the status of routes (only one of the route in 'start' status)
Flow: Start route A --(when route A complete its goal)-> Suspend route A ---> Start route B --(when route B complete its goal)-> Suspend route B ---> Start route A [loop back to head]
Dependent route execution first can be achieved in Camel using "RouteContext".
Example: If route 'A' is executed before route 'B' then route 'A' should be defined as 'RouteContext' and route be is defined inside "camelContext" like below:
<routeContext id="A" xmlns="http://camel.apache.org/schema/spring">
<route id="A">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Department?repeatCount=1&noop=true&delay=10000"/>
<log message="Department data is : ${body}"/>
</route>
</routeContext>
Then regular "camelContext" should be defined with reference to this routeContext first.
<camelContext id="test" xmlns="http://camel.apache.org/schema/spring">
<routeContextRef ref="A"/>
<route id="B">
<from uri="file:/home/dev/code/Integration/RunCamleExample/src/main/resources/csv/Employee?noop=true&delay=10000"/>
<log message="Employee data is : ${body}"/>
</route>
</camelContext>
I'm using a ActiveMQ Broker with built-in Camel Routes. I want to read a file after an Event received.
<pseudo>
from Event A
read File XY
to Event B with Body from File XY
</pseuod>
I simple tried moving files from a temporary directory based on an event but only event B is written. In the Log file are no Exceptions or Error messages.
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<!-- You can use Spring XML syntax to define the routes here using the <route> element -->
<route>
<description>Example Camel Route</description>
<from uri="activemq:example.A"/>
<from uri="file://tmp/a?delete=true"/>
<to uri="file://tmp/b?overruleFile=copy-of-${file:name}"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
Update with working solution for single file:
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<!-- You can use Spring XML syntax to define the routes here using the <route> element -->
<route>
<description>Example Camel Route</description>
<from uri="activemq:example.A"/>
<pollEnrich>
<constant>file:///tmp/a?fileName=file1</constant>
</pollEnrich>
<log message="file content ${body}"/>
<to uri="activemq:example.B"/>
</route>
</camelContext>
You need to use Content Enrichers for this. This is exactly what you are looking for.
<route>
<from uri="activemq:example.A"/>
<pollEnrich>
<constant>file://tmp/a?delete=true</constant>
</pollEnrich>
<to uri="activemq:example.B"/>
</route>
Please be aware that for camel version 2.15 or older
pollEnrich does not access any data from the current Exchange which
means when polling it cannot use any of the existing headers you may
have set on the Exchange. For example you cannot set a filename in the
Exchange.FILE_NAME header and use pollEnrich to consume only that
file. For that you must set the filename in the endpoint URI.
I'm using activeMQ 5.9.
I'm trying to implement an interception type route in my activemq.xml, where I check if a particular header equals some value then send it to a different queue, otherwise allow it to continue.
I'm following the info here: http://activemq.apache.org/broker-camel-component.html
My camel.xml file looks like this:
<camelContext id="camel" trace="false" xmlns="http://camel.apache.org/schema/spring">
<route id="routeAboveQueueLimitTest">
<from uri="activemq:queue:do.something"/>
<choice>
<when>
<simple>${header.scope} == 'test'</simple>
<to uri="activemq:queue:test.do.something"/>
</when>
<otherwise>
<to uri="activemq:queue:do.something"/>
</otherwise>
</choice>
</route>
</camelContext>
Then when I put a message on "activemq:queue:do.something" with header called scope = "test" it correctly routes to the "activemq:queue:test.do.something" queue. However, when it doesn't have that header, it puts it back on the "activemq:queue:do.something" queue and processes it again and again and again!
That kind of seems logical, but the above page clearly says that you have to explicitly send it back to the broker component, and the 2nd example on the page shows exactly that.
I realise this could be worked around by sending it to a different queue if it doesn't have the header but that is undesirable at this stage.
I think the intercept pattern would be much better suited for what you are looking.
<intercept>
<when><simple>${header.scope} == 'test'</simple></when>
<to uri="activemq:queue:test.do.something"/>
</intercept>
More info here: http://camel.apache.org/intercept.html
This will allow messages without the scope header set to 'test' to continue, but will redirect messages that do have the test header.
InterceptSendToEndpoint is a better option here...
<interceptSendToEndpoint uri="activemq:queue:do.something">
<when><simple>${header.scope} == 'test'</simple></when>
<to uri="activemq:queue:test.do.something"/>
<stop/>
</interceptSendToEndpoint>
I'm trying to make a route that will process big xml files using camel-stax. A file content processing works fine, but at the end it fails with a following error:
Caused by: java.io.IOException: Renaming file from: C:\workdir\file.xml to: C:\workdir\.camel\file.xml failed due cannot delete from file: C:\workdir\file.xml after copy succeeded
It seems that camel doesn't close a file input stream, so after processing it cannot move a file to a target location. Of course, I can set noop=true, bit I wanted to remove processed files.
My route looks like following:
<route id="myRoute">
<from uri="file:{{working_dir}}?include=file.xml" />
<split streaming="true">
<ref>staxRecord</ref>
<to uri="log:test"/>
</split>
</route>
Initially it was a little bit more complex and I simplified it as possible. Now it looks just like a last sample from here http://camel.apache.org/stax.
Additional note: I execute the route on Windows. Camel version: 2.12.2.
So it looks like a bug in the camel-stax component.
I've found an alternative way of how to deal with big xml files. I've rewritten my route as following:
<route id="myRoute">
<from uri="file:{{working_dir}}?include=file.xml&delete=true" />
<split streaming="true">
<tokenize token="entry" xml="true"/>
<unmarshal ref="myJaxb"/>
<!-- ... -->
</split>
</route>
I want to set the endpoint specific header value in Multicast component.
XML DSL as below:
<route>
<from uri="direct:testRoute"/>
<multicast strategyRef="MyAggregator" parallelProcessing="true">
<to uri="direct:call1"/> <!-- set the header MY_HEADER = "call_1" -->
<to uri="direct:call2/> <!-- set the header MY_HEADER = "call_2" -->
</multicast>
</route>
Basically in the response aggregation I want to know, to which service request this response belongs to.
I tried by doing this, but its not the correct way (parse exception):
<to uri="direct:call1">
<setHeader headerName="MY_HEADER"><simple>call1</simple></setHeader>
</to>
What I see from reading the documentation is that, multicast will copy the source Exchange and multicast each copy. So its a shallow copy of the Exchange and kind of reference shared between all the multicast recipient.
But here I am looking for specific header value for individual recipient.
How to do this? Any pointers?
You can't do that in the multicast route. But it should be simple in the direct route afterwards.
<route>
<from uri="direct:call1"/>
<setHeader headerName="MY_HEADER"><simple>call1</simple></setHeader>
.. do whatever
</from>
</route>
otherwise, if call1 is used for other things and you cannot know when to put the header once in that route, make a simple prep-route:
<route>
<from uri="direct:prepCall1"/>
<setHeader headerName="MY_HEADER"><simple>call1</simple></setHeader>
<to uri="direct:call1"/>
</from>
</route>
As a third option, even though you cannot place DSL (xml or java) in the multicast list, you can supply an "onPrepareRef" processor bean that adds the headers to your exchange. But one processor will handle all multicast endpoints.
There is a header with the key Exchange.TO_ENDPOINT that you can see which of the 2 endpoints the response is from.