Batch insertion issue while shutting down camel - apache-camel

I am doing batch insertion in mysql using mybatis.I keep adding messages to an ArrayList and fire the query when ArrayList size has reached the batch size.
<route id="billingRoute">
<from uri="activemq:queue:{{billing.queue}}" />
<log message="Message received from billing queue is ${body}"></log>
<unmarshal ref="gsonBilling"></unmarshal>
<bean beanType="com.bng.upload.processors.GetCdr" method="process(com.bng.upload.beans.BillingEvent,${exchange})" />
log message="Multicasting data ${body} to file system and database" />
<multicast>
<pipeline>
<transform>
<method ref="insertionBean" method="billingBatchInsertion"></method>
</transform>
<choice>
<when>
<simple> ${body.size()} == ${properties:batch.size}</simple>
<to uri="mybatis:batchInsertBilling?statementType=InsertList"></to>
<log message="Inserted in billing table : ${in.header.CamelMyBatisResult}"></log>
</when>
</choice>
</pipeline>
<pipeline>
<choice>
<when>
<simple>${properties:billing.write.file} == true</simple>
<setHeader headerName="path">
<simple>${properties:billing.cdr.folder}</simple>
</setHeader>
<log message="Going to write to file : ${body}"></log>
<bean beanType="com.bng.upload.processors.ToFile"
method="process(${exchange},com.bng.upload.beans.BillingCdr)" />
<to uri="file://?fileExist=Append"></to>
</when>
</choice>
</pipeline>
</multicast>
</route>
</routeContext>
We are not using transactions as there are multiple components included in route -Files,ActiveMq and Database. The issue is that when the tomcat is restarted,if the ArrayList size has not reached the batch size,those messages get lost.Is there any solution to it ?

Related

Multi File Download Not Working in Apache Camel

I am using quartz and poll enrich to download multiple files at once , one of the configuration used is localWorkDirectory=/tmp/sftp_tmp/ . I see in logs that says
org.apache.camel.component.file.remote.SftpConsumer.processExchange - About to process file: RemoteFile[filename] . But it randomly sometimes places the file in /tmp/sftp_tmp/ instead of the at final destination c:/cameldata at times. I am using camel version 2.24.1
<route id="sftp-read">
<from uri="quartz2://sftp?stateful=true&trigger.repeatInterval=3s"/>
<pollEnrich><spel>file:c:/cameldata?include=SFTPRECFILE&noop=true&idempotent=false&readLock=markerFile</spel></pollEnrich>
<to uri="seda:startSftpExecutor?waitForTaskToComplete=Always&timeout=-1"/>
</route>
<route id="sftp-executor">
<from uri="seda:startSftpExecutor" />
<pollEnrich><spel>{{sftpUrl}}&sendEmptyMessageWhenIdle=true&binary=false&include=BR-.*&maximumReconnectAttempts=3&noop=true&maxMessagesPerPoll=1&localWorkDirectory=/tmp/sftp_tmp/&stepwise=false</spel></pollEnrich>
<choice>
<when>
<simple>${in.body} != null</simple>
<to uri="file:c:/cameldata?fileName=${file:onlyname}&tempPrefix=.tmp"/>
</when>
<otherwise>
<setBody><constant></constant></setBody>
<log loggingLevel="INFO" message="file(s) downloaded successfully" />
<to uri="file:c:/cameldata?fileName=LOADFILE" />
</otherwise>
</choice>
</route>

Apache Camel Java dsl tool

Is there any tool available that can convert java DSL to XML route or vice versa.
I want to convert the following XML route into Java DSL
<route id="test">
<from uri="file://{{VAR_DATA_PATH}}/test/xml"/>
<multicast>
<choice>
<when>
<xpath>/bookinfo</xpath>
<doTry>
<to uri="downloadBook"/>
<marshal ref="xstream-utf8"/>
<to uri="another URI"/>
<doCatch>
<exception>java.lang.Exception</exception>
<handled>
<constant>false</constant>
</handled>
<to uri="3rd URI"/>
</doCatch>
</doTry>
</when>
<when>
<xpath>somePath</xpath>
<to uri="4th URI" />
<to uri="5th URI"/>
</when>
</choice>
<to uri="6th URI" />
</multicast>
</route>
From Java DSL to XML, it's fairly easy. You can use Hawtio or karaf's "route-info" command. Even though the routes are in Java DSL, when you view them it would be XML.
I'm not aware of the other way around (from XML to Java) but it's not difficult at all to do it yourself.

How to process the same message sequentially in Camel?

I am new to Camel and I am trying to do the following.
I want to process the same message twice. I have to transform both messages and process one message first and then second only if the first one is sucessfully executed (I have a condition).
I tried to use a multicast first. Then I transform the messages in each route. The first one (the operation_DC) only sends a successfull message to the second one (the operation_AC) only if it is sucessfull. The second operation contains an aggregation that waits until it times out for both messages. I only want to process the messaged from the multicast and discard the other.
<route id="t_operation_ME">
<from uri="direct:operation_ME" id="t.direct.aggregatedService"></from>
<setHeader headerName="id">
<simple>exchangeId</simple>
</setHeader>
<multicast parallelProcessing="true" strategyRef="defaultAggregationStrategy" stopOnException="true" onPrepareRef="cleanHeader" parallelAggregate="false" completionPredicate="">
<to uri="direct:operation_DC"></to>
<to uri="direct:operation_AC"></to>
</multicast>
</route>
<route id="direct:operation_DC">
<from uri="direct:operation_DC" />
<log message="ENTER DC"></log>
<to uri="xslt:{{depasse:core.transformation.xml.path}}client/t/Operation_toDC_request.xsl" id="t.dc.transform.productos" />
<to uri="activemq:QCIn" id="t.dc.qcin.queue.send"></to>
<log message="EXIT DC ${body}"></log>
<choice>
<when>
<xpath>//Data/Status[. = 'OK']</xpath>
<log message="SEND TO AC"></log>
<to uri="direct:operation_AC"></to>
</when>
</choice>
</route>
<route id="direct:operation_AC">
<from uri="direct:operation_AC" />
<log message="ENTER AC"></log>
<aggregate completionTimeout="20000" completionSize="2" discardOnCompletionTimeout="true" strategyRef="tAggregationStrategy">
<correlationExpression>
<simple>header.id</simple>
</correlationExpression>
<log message="ENTER AGG AC ${body}"></log>
<to uri="xslt:{{depasse:core.transformation.xml.path}}client/t/Operation_toAC_request.xsl" id="t.ac.transform.productos" />
<to uri="activemq:QCIn" id="t.ac.qcin.queue.send"></to>
<log message="EXIT AC ${body}"></log>
</aggregate>
<log message="END AC\n${body}"></log>
</route>
The thing is that when I log "EXIT AC" and "END AC" the message is different. This means that while in the server I am watching the process correctly, the AC client receives an incorrect message.
Multicast send copy of messages always to every endpoint between tags. If you want to call route operation_AC only after operation_DC run successfully, you shouldn't use multicast, but you should save your body at the beginning of operation_DC, for example using property and Camel Simple:
<from uri="direct:operation_DC" />
<setProperty propertyName="body">
<simple>${body}</simple>
</setProperty>
and before calling operation_AC set body to bring message body from the beginning:
<setBody>
<simple>${property.body}</simple>
</setBody>
<to uri="direct:operation_AC"/>

Can I pipeline within content based router?

Can I pipeline within a content based router?
I have to pipeline beans within a content-based router. For that, I adopted the following configuration. I hope the configuration itself explains my requirements. Is it correct?
Do I have to add the end() tag also?
<route>
<from uri="activemq:queue:injob"/>
<choice>
<when>
<simple>${header.type} == 'heartbeat'</simple>
<to uri="bean:heartBeatHandler"/>
<to uri="activemq:queue:outjob"/>
</when>
<when>
<simple>${header.type} == 'dnsrequest'</simple>
<to uri="bean:dnsRequestHandler"/>
<to uri="bean:parser"/>
<to uri="activemq:queue:outjob"/>
</when>
<when>
<simple>${header.type} == 'whoisrequest'</simple>
<to uri="bean:whoisRequestHandler"/>
<to uri="bean:parser"/>
<to uri="activemq:queue:outjob"/>
</when>
<otherwise>
<to uri="bean:errorHandler"/>
</otherwise>
</choice>
</route>
Yes this is correct what you do.
Camel runs in pipeline mode by default (eg the pipes and filters EIP - http://camel.apache.org/pipes-and-filters.html), but if you want you can make that explicit using < pipeline >. eg
<when>
<simple>${header.type == 'heartbeat'}</simple>
<pipeline>
<to uri="bean:heartBeatHandler"/>
<to uri="activemq:queue:outjob"/>
</pipeline>
</when>
But very often you would omit the < pipeline > and do as your example.
Opposed to pipeline there is multicast (http://camel.apache.org/multicast.html), and when you combine these two, then you may need to use pipeline eg
<multicast>
<pipeline>
<to uri="bean:heartBeatHandler"/>
<to uri="activemq:queue:outjob"/>
</pipeline>
<pipeline>
<to uri="bean:somethingElse"/>
<to uri="activemq:queue:somethingElse"/>
</pipeline>
</multicast>

sftp using camel

I am trying to use sftp using camel, and getting jsch esception.
The route that I created for the SFTP -
<camelContext xmlns="http://activemq.apache.org/camel/schema/spring">
<package>myGroupId</package>
<route>
<from uri="file:src/srcData?noop=true"/>
<choice>
<when>
<xpath>/person/city = 'London'</xpath>
<to uri="file:src/targetData/UK"/>
</when>
<when>
<xpath>/person/city = 'Chicago'</xpath>
<to uri="file:src/targetData/US"/>
</when>
<when>
<xpath>/person/city = 'Tokyo'</xpath>
<to uri="sftp://XXXserverXXX:22/dir1/subdir?username=testUser?password=testPwd&binary=true"/>
</when>
<otherwise>
<to uri="file:src/targetData/OT"/>
</otherwise>
</choice>
</route>
</camelContext>
But with this configuration I am facing the following exception -
com.jcraft.jsch.JSchException: reject HostKey:
You should probably define a hostfile:
The "knownHostsFile" option should point to a ssh known hosts file with the public key of the host you are connecting to in it.
It's actually documented over here: http://camel.apache.org/ftp2.html

Resources