I am trying move few files from location A to location B on same server based on the name and number of files present in that folder. To achieve this I am using processStrategy option and extended GenericFileDeleteProcessStrategy to check for the batchsize. The files get moved to the destination but I also wanted to move the processed files to the .done folder and any failed files to the .failed location.
If I noticed that move, premove, moveFailed options are not working when used along with processStrategy
below is the code snippet I have used.
Please let me know your suggestions. I need to have the files available in the backup.
public class CustomTestProcessStrategy extends GenericFileDeleteProcessStrategy<File>{
static Logger logger = Logger.getLogger(CustomTestProcessStrategy.class);
#Override
public boolean begin(GenericFileOperations<File> operations,
GenericFileEndpoint<File> endpoint, Exchange exchange,
GenericFile<File> file) throws Exception {
boolean begin = super.begin(operations, endpoint, exchange, file);
if(!begin){
return false;
}
int batchSize = (int) exchange.getProperty("CamelBatchSize");
if(batchSize != 2){
logger.info("[OUT] CustomTestProcessStrategy Stopped. batch size is not met.");
return false;
}
return true;
}
beans.xml
<bean id="testStrategy" class="com.camel.CustomTestProcessStrategy">
cameljob.properties
file.testmove.properties=?idempotent=true&processStrategy=#testStrategy&readLock=rename&readLockMinLength=0&initialDelay=10000&delay=5000&moveFailed=.failed&preMove=order&move=.done&scheduler=quartz2&scheduler.cron={{testmove.Schedule}}
folder.source=C:/readfiles/source
folder.destination =C:/readfiles/destination
camel-context.xml
<route id="FileTestMoveRoute">
<from uri="file:{{folder.source}}{{file.testmove.properties}}"/>
<bean ref="cifProcessor" method="startFinalProcess"/>
<log message="*****************Moving file ${header.CamelFileName} to the final desitnation folder *********" />
<choice>
<when>
<simple>${header.CamelFileName} regex '^.*(fileA).*(CSV|csv)'</simple>
<to uri="file:{{folder.destination}/FileA}"/>
</when>
<when>
<simple>${header.CamelFileName} regex '^.*(fileB).*(CSV|csv)'</simple>
<to uri="file:{{folder.destination}}/FileB"/>
</when>
</choice>
</when>
</route>
Related
I'm trying to pick a file from a directory, split a file and add each splitted lines to activemq. I'm facing a problem with exception handling during this process. Lets say a file in the directory is a binary file (executable), then splitter thows org.apache.camel.RuntimeCamelException and java.nio.charset.MalformedInputException exceptions. If this occurs, then I need to catch these exceptions, nothing should add in activemq and that just particular thread should exit after logging exception. Referred online and wrote the following code but don't know how to stop adding to activemq and quit specific thread.
<route id="msg_producer">
<from uri="input.file.from" />
<doTry>
<split parallelProcessing="true" executorServiceRef="msgProducer"
streaming="true">
<tokenize token="\n"></tokenize>
<to uri="input.activemq.to" />
</split>
<doCatch>
<exception>org.apache.camel.RuntimeCamelException</exception>
<exception>java.nio.charset.MalformedInputException</exception>
<handled> <constant>true</constant></handled>
<setBody>
<simple>${exception.stacktrace}</simple>
</setBody>
<setHeader headerName="CamelFileName">
<simple>${file:onlyname.noext}_error.log</simple>
</setHeader>
</doCatch>
</doTry>
</route>
as #claus Ibsen said to filter the files you can use a filefilter property so that you pick only files based on extension and some standard pattern something like this
<bean id="FileFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
<!-- ? matches one character
* matches zero or more characters
** matches zero or more directories in a path -->
<property name="includes" value="#{databaseProperties.getProperties().getProperty('file.name.pattern')}"/>
<!-- if you wan to exclude specific files say bad in name or .exe files. Use comma to separate multiple excludes -->
<!-- <property name="excludes" value="**/*bad*,**/*.exe"/> -->
</bean>
and your file.name.pattern can be something like this **/contract.csv
I wrote the following route and expected that the bean 'teaserService' should be called only one time, at the end of processing of all files, but ... it's called after processing of each file:
<route id="teaserInterface">
<from
uri="file://{{teaser.dropInDir}}?readLock=changed&delete=true&delay=60000" />
<choice>
<when>
<simple>${file:ext} == 'properties'</simple>
<to uri="file://{{teaser.config.directory}}" />
</when>
<when>
<simple>${file:ext} == 'jpg' || ${file:ext} == 'JPG'</simple>
<to uri="sftp://{{apache.ftp.user}}#{{apache.ftp.host}}/{{apache.teaser.ftp.targetDir}}?password={{apache.ftp.password}}&binary=true&separator=UNIX" />
</when>
<otherwise>
<transform>
<simple>Dear user,\n\n the Teaser interface only accept *.jpg and *.properties files, but we found the file ${file.name}.\n\n Have a nice day,\nYour lovely Teaser interface</simple>
</transform>
<to
uri="smtp://smtp.blabla.com?contentType=text/plain&from=blabla#blabla.com&to=chica#chicas.com&subject=A problem occured while setting up new teaser!" />
</otherwise>
</choice>
<bean ref="teaserService" method="updateTeaser" />
</route>
How to achieve such a behavior?
Thanks
The Camel file compoment is a batch consumer and adds properties to the exchange regarding the batch it is processing. You can test for the property CamelBatchComplete and if that is set to true, then call your bean.
If you want to proceed only after all files have been read, you must sample them somehow. This can be achieved using the aggregator pattern:
<route>
<from uri="file://src/data/aggregate-and-process?readLock=changed&delete=true&delay=60000" />
<aggregate strategyRef="aggregationStrategy" completionFromBatchConsumer="true">
<correlationExpression>
<constant>true</constant>
</correlationExpression>
<to uri="direct:sub" />
</aggregate>
</route>
<route>
<from uri="direct:sub" />
<!-- processing aggregated body -->
</route>
Please note that I set completionFromBatchConsumer="true". From the Camel documentation:
This option is if the exchanges are coming from a Batch Consumer. Then when enabled the Aggregator2 will use the batch size determined by the Batch Consumer in the message header CamelBatchSize. [...] This can be used to aggregate all files consumed from a File endpoint in that given poll.
I'd wish to create a general restful service to perform crud operations using Camel, but without using Restlet, just plain Servlet and SQL components.
I am familiar with Camel for 2 days only, and I am not able to get URL parts in order to use them in SQL query.
The idea is to use only XML configuring. I tried many doferent ways, came to using javascript. Here's my code so far (below).
I can't get how to check the url path, parse out it's parts, put into "message body" which should be a source for parameters for SQL query.
In this particular scenario it says, "ReferenceError: "response" is not defined".
<route>
<from uri="servlet:///crud?matchOnUriPrefix=true"/>
<setBody>
<javaScript><![CDATA[
request.headers.get("CamelHttpPath").match(/\/(\w+)\/(\w*)/)
]]></javaScript>
</setBody>
<choice>
<when>
<javaScript><![CDATA[
request.headers.get("CamelHttpMethod") == "GET" &&
response.body != null
]]></javaScript>
<to uri="sql:select * from person where id=2?dataSource=dataSource"></to>
</when>
<when>
<xpath>$CamelHttpMethod = 'POST'</xpath>
<transform>
<simple>Update!!! path - ${header.CamelHttpPath}, url - ${header.CamelHttpUrl}, uri = ${header.CamelHttpUri}, base uri = ${header.CamelHttpBaeUri}. And request is ${header.CamelHttpServletRequest}</simple>
</transform>
</when>
<when>
<xpath>$CamelHttpMethod = 'PUT'</xpath>
<transform>
<simple>Insert</simple>
</transform>
</when>
<when>
<xpath>$CamelHttpMethod = 'DELETE'</xpath>
<transform>
<simple>Delete...</simple>
</transform>
</when>
<otherwise>
<transform>
<simple>Unsupported method ${header.CamelHttpMethod} (${header.CamelHttpPath})</simple>
</transform>
</otherwise>
</choice>
</route>
I am doing data format conversion between POJO to CSV and vice versa. In this while converting CSV to Object file(Unmarshalling) i am getting illegal argument exception for int data type. Only for string its working fine.
Below is my POJO
#CsvRecord(separator="//|",crlf="UNIX",generateHeaderColumns=false)
public class EmployeeVO implements Serializable{
private static final long serialVersionUID = -663135747565879908L;
#DataField(pos=1)
private String name;
#DataField(pos=3)
private Integer age;
#DataField(pos=2)
private String grade;
// getter setter
}
csv data
sumit|4th standrad|22
the above csv is generated from the above POJO. But at the time of converting CSV to POJO i am getting folloing exception
java.lang.IllegalArgumentException: Parsing error detected for field defined at the position: 3, line: 1
following are my camel - context file for your reference
marshal
<route>
<from uri="cxf:bean:rtoemplyeeService"/>
<convertBodyTo type="java.lang.String" id="stringInput"/>
<bean ref="govtEmpBean" method="getEmployeeCSV" beanType="govtEmpBean" id="govtEmp"/>
<log message="before marshalling ================== ${body}"/>
<marshal ref="bindyDataformat">
<bindy type="Csv" packages="com.mycompany.converter.vo"/>
</marshal>
<log message="after marshalling ================== ${body}"/>
<to uri="file://D:/JATO_WORK/repo_bkp/csv/"/>
<setBody>
<simple>CSV output is generated at file system </simple>
</setBody>
</route>
un marshal
<route id="csvtoobject">
<from uri="file://D:/JATO_WORK/repo_bkp/csv?delay=10000&initialDelay=10"/>
<log message="csv string ============= ${body}"/>
<unmarshal ref="bindyDataformat"/>
<log message="${body}"/>
<bean ref="govtEmpBean" method="printCSVObject" beanType="govtEmpBean" id="govtEmp"/>
</route>
You need to specify the csv record as follows
#CsvRecord(separator = "\\|")
We are using the below configuration to send a file from single source to multiple remote destinations.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camel="http://camel.apache.org/schema/spring"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<routeContext id="gcgRatesOutbound" xmlns="http://camel.apache.org/schema/spring">
<route id="gcgRatesFileOut">
<from uri="file:{{nas.root}}/{{gcg.out.prices.dir}}?delay={{poll.delay}}&initialDelay={{initial.delay}}&readLock=rename&scheduledExecutorService=#scheduledExecutorService" />
<multicast stopOnException="true">
<to uri="scp://{{gcg.ste.Client1_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client1}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client2_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client2}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client3_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client3}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
</multicast>
</route>
</routeContext>
</beans>
Using the above confirguration the file reaches the remote destinations, which confirmed that the connection to all the remote destinations were successfull.
We need that the file should be moved to the archive folder after the file has been successfully transfered to all the remote destinations.
And should move to error folder incase of any error.
However, when I add the archival code ( element) as child element to the multicast element in the above configuration and use the and the tag to move the file to error folder incase of an error. The file does not reach the remote destinations.
<doTry>
<multicast stopOnException="true" parallelProcessing="true">
<to uri="scp://{{gcg.ste.Client1_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client1}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client2_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client2}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="scp://{{gcg.ste.Client3_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client3}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<to uri="file://{{nas.root}}/{{gcg.out.prices.dir}}?fileName={{archive.dir}}" />
</multicast>
<doCatch>
<exception>java.lang.Exception</exception>
<handled>
<constant>true</constant>
</handled>
<to uri="file://{{nas.root}}/{{gcg.out.prices.dir}}?fileName={{error.dir}}" />
</doCatch>
</doTry>
The file does not reach the remote destinations nor does it produce any log and the file is moved to archive folder.
I tried placing the remote destinations, the archival code and the move to error code in seperate routes and have its reference in a single multicast as below
<to uri="direct:Client1FileOut" />
<to uri="direct:Client2FileOut" />
<to uri="direct:Client3FileOut" />
<to uri="direct:MoveToArchive" />
<route id="gcgFileOut1">
<from uri="direct:Client1FileOut" />
<doTry>
<to uri="scp://{{gcg.ste.Client1_User_Name}}#{{gcg.ste.Host_Name}}:{{gcg.ste.Port_Number}}/{{gcg.ste.Destination_Client1}}?knownHostsFile={{ssh.knownHosts}}&privateKeyFile={{ssh.privateKey}}" />
<doCatch>
<exception>java.lang.Exception</exception>
<handled>
<constant>true</constant>
</handled>
<to uri="direct:gcgError" />
</doCatch>
</doTry>
</route>
However, The file does not reach the remote destinations nor does it produce any log and the file is moved to archive folder.
I am new to camel.
I tried using the shareUnitOfWork attribute as below
<multicast shareUnitOfWork="true">
However, the file moved to archive folder along with Test_2_19082013_3.txt.camelLock file
Not sure why this file is also moved to archive when the file dropped was Test_2_19082013_3.txt
Also there is folder named "ARCHIV~1" created in the drop location.
Something might went wrong when moving the file to archive. You have stopOnException="true" parallelProcessing="true" and the local file is probably done first as it should be the fastest.
You probably want to print the error somewhere. Now, you catch the exception and mark the error as handled. Still you expect logs. Use the log component and output some log statements manually. Not only in case or error but also in case of success. You can log in the debug level so that you can manually enable log outprint in case you need it - like now.
Another option for you to figure out what's going on is to enable trace which will make you less "blind".
That said about debugging on your own -
You are archiving and storeing error messages in the input directory. The lock file you see is when Camel is reading. This is likely what's casuing your application to malfunction.
{{nas.root}}/{{gcg.out.prices.dir}}
The fileName=... is the filename, not another sub directory.
So: file://{{nas.root}}/{{gcg.out.prices.dir}}/{{archive.dir}} should do it (likewise for the error path).