I am using infinispan9 with camel
infispan.xml
<distributed-cache name="myCache" mode="SYNC" start="EAGER">
<locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>
<!-- set lifespan to 1 hour in millisecond 3600000-->
<expiration lifespan="120000" />
</distributed-cache>
In my camel route I am using:
<idempotentConsumer messageIdRepositoryRef="infinispanRepo">
<header>CamelFileName</header>
<setHeader headerName="fileExist">
<simple>true</simple>
</setHeader>
</idempotentConsumer>
<when>
<simple>${header.fileExist} == null</simple>
<!-- log duplicate-->
</when>
Now my concern is I have processed a file at 6:30:00 it processed success. I reprocessed the file at 6:31:00, it loged duplicate. that's fine. I am expecting the file to be successfully reprocessed after 6:32:00 but Failed and logged duplicate. What I observed is when ever the idempotentConsumer is verifying, the time is updated and I need to wait 2 sec after the last processed duplicate log.
All I want is to expire after first entry. Please help.
Related
I am writing 2 routes to process a files in a directory, those files could have any name, but I need 2 routes as I need some complex processing.
First route:
<route id="Init">
<from uri="file:{{file.path}}?move=.done&moveFailed=.error&readLock=changed&readLockCheckInterval=1500&charset=UTF-8"/>
<transacted/>
<split streaming="true" stopOnException="true" shareUnitOfWork="true" parallelProcessing="false">
<tokenize token="\r\n"/>
<choice>
<when>
<simple>${body.substring(0,4)} == 4000</simple>
[...]
<to uri="file:{{file.path}}/tmp?fileName=${date:now:yyyyMMddss}.txt&fileExist=append&charset=UTF-8"/>
</when>
<when>
<simple>${body.substring(0,4)} == 4002</simple>
[...]
<to uri="file:{{file.path}}/tmp?fileName=${date:now:yyyyMMddss}.txt&fileExist=append&charset=UTF-8"/>
</when>
</choice>
</split>
</route>
Second route, which consumes the file produced by the first route:
<route id="End">
<from uri="file:{{file.path}}/tmp?delete=true&moveFailed=.error&readLock=changed&readLockCheckInterval=1500&charset=UTF-8"/>
<transacted/>
<split streaming="true" stopOnException="true" shareUnitOfWork="true" parallelProcessing="false">
<tokenize token="\r\n4000"/>
[...]
<to uri="[...]"/>
</split>
</route>
I am trying to make sure file produced by route Init won't be consumed by route End until the Init has finished processing the first file.
I guessed using a temp file extension, and then using an exlude on the second route, but it doesn't work with fileExists.
Any ideas?
Thanks!
Use done file
You need a mechanism to make sure the second route only consume file that have been completely processed by first route.
A simple method is to let first route emit a done file as a signal to tell second route that the file has been processed completed and is ready to pickup.
To use done file, you could add doneFileName parameter in first route when process completed and also add in the second route using same filename pattern.
For more details, please read Section "Using 'done' Files" of Camel File Component
You cannot use readLock=changed with the file component as it's only available for FTP/SFTP from Camel 2.8 onwards.
changed is using file length/modification timestamp to detect whether the file is currently being copied or not. Will at least use 1 sec. to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency. This option is only avail for the FTP component from Camel 2.8 onward. Note: from Camel 2.10.1 onward the FTP option fastExistsCheck can be enabled to speedup this readLock strategy, if the FTP server support the LIST operation with a full file name (some servers may not).
Try one of the other mechanisms such as markerFile, fileLock, or rename
I am reading camel-example-sql, it has two routes defined as below:
<!-- route that generate new orders and insert them in the database -->
<route id="generateOrder-route">
<from uri="timer:foo?period=5s"/>
<transform>
<method ref="orderBean" method="generateOrder"/>
</transform>
<to uri="sql:{{sql.insertOrder}}"/>
<log message="Inserted new order ${body[id]}"/>
</route>
<!-- route that process the orders by picking up new rows from the database
and when done processing then update the row to mark it as processed -->
<route id="processOrder-route">
<from uri="sql:{{sql.selectOrder}}?consumer.onConsume={{sql.markOrder}}"/>
<to uri="bean:orderBean?method=processOrder"/>
<log message="${body}"/>
</route>
I can understand first route, which fires every 5 second. But when does second route be triggered?
The 2nd route uses the SQL select statement from {{sql.selectOrder}} to query the database, and if there is a resultset, thean each row becomes a message that is routed. If the resultset is empty then no message is routed.
The 2nd route use a scheduler that runs every every 500 millis (eg the consumer.delay) option.
http://camel.apache.org/sql-component
We have a camel route that looks at a file and processes potentially hundreds of records on this file, almost like a batch routine (yet there will only be one message in camel). Thus the message will take potentially minutes or maybe hours to complete. We want to shut down the queue once this message (and any others waiting) are complete.
We have the following to consider:
The shutdown strategy defines the time to wait for a route to stop before a forced shutdown
<bean id="shutdown" class="org.apache.camel.impl.DefaultShutdownStrategy">
<property name="timeout" value="#[bpf.defaultShutdownStrategy.timeout]"/>
</bean>
The route has a parameter shutdownRunningTask="CompleteAllTasks" which should wait untill all messages processed.
Not sure which is going to take presidence as the timeout once exceeded is not graceful, it will force shutdown and for our scneario it is likely we will exceed a timeout, as we cannot predict how long processing will take.
Any ideas/considerations?
Thanks in advance.
You should look at the onCompletion functionality. It adds a new route in a separated thread when the Exchange is complete.
Here is some examples from the Camel documentation:
Java DSL
// define a global on completion that is invoked when the exchange is complete
onCompletion().to("log:global").to("mock:sync");
from("direct:start")
.process(new MyProcessor())
.to("mock:result");
XML DSL
<!-- this is a global onCompletion route that is invoke when any exchange is complete
as a kind of after callback -->
<onCompletion>
<to uri="log:global"/>
<to uri="mock:sync"/>
</onCompletion>
<route>
<from uri="direct:start"/>
<process ref="myProcessor"/>
<to uri="mock:result"/>
</route>
Then, here is documentation on how to stop a route in Camel.
Ashish here again.
I have developed a route which parses incoming SOAP request and logs some of information from the request.
I used headers and XPATH for fetching information from SOAP service request. The route successfully parse and logs appropriate information.
The route is as follows:
<route streamCache="true">
<from uri="<some URI>"/>
<setHeader headerName="SOAPAction">
<constant>http://www.something.com/constant>
</setHeader>
<setHeader headerName="actionId">
<xpath resultType="java.lang.String">//ws:actionId/text()</xpath>
</setHeader>
<choice>
<when>
<xpath>//ws:actionId = '1'</xpath>
<to ref="callService" />
</when>
<otherwise>
<log message="Wrong Action ID : ${in.headers.actionId}"
loggingLevel="ERROR"/>
</otherwise>
</choice>
</route>
As per logic the route must redirect to callService URI if actionId is 1, otherwise it should log wrong action Id without any response.
The logic works if actionId is 1, it redirects to callService and sends appropriate response to caller. If actionId is other than 1, it writes the log about wrong action Id with action Id but it sends the same request as response to the caller.
I checked it using soapUI and Java Socket Programming.But output is same, logs the actionId accurately but sends the same request as response to the caller.
I don't want same request as response. Response must be blank.
Can anybody help me to find the solution.
Regards,
Ashish
You need to set the response to empty or what empty means for you in that situation as well.
And the response needs to fit the WSDL contract what the response can be.
So in the otherwise add code to set the empty response
For example something a like:
<otherwise>
...
<transform>
<constant>NO DATA</constant>
</transform>
</constant>
Though also mind if you use camel-cxf then the dataFormat option can affect how an empty response should be created.
I would like to know if it's possible with Camel to do throttling based on the content of the exchange.
The situation is the following: I have to call a webservice via soap. Among, the parameters sent to that webservice there is a customerId. The problem is that the webservice send back an error if there are more than 1 request per minute for a given customerId.
I'm wondering if it would be possible to implement throttling per customerId with Camel. So the throttling should not be implemented for all messages but only for messages with the same customerId.
Let me know how I could implement this or if I need to clarify my question.
ActiveMQ Message Groups is designed to handle this case. So, if you can introduce a JMS queue hop in your route, then just set the JMSXGroupId header to the customerId. Then in another route, you can consume from this queue and send to your web service to get the behavior you described.
also see http://camel.apache.org/parallel-processing-and-ordering.html for more information...
While ActiveMQ Message Groups would definitely address the parallel processing of unique customer ID's, in my assessment Claus is correct that introducing a throttle for each unique group represents an unimplemented feature for Camel/ActiveMQ.
Message Groups alone will not meet the SLA described. While each group of messages (correlated by the customer ID) will be processed in order with one thread per group, as long as requests take less than a minute to receive a response, the requirement of one request per minute per customer would not be enforced.
That said, I would be very interested to know if it would be possible to combine Message Groups and a throttle strategy in a way that would simulate the feature request in JIRA. My attempts so far have failed. I was thinking something along these lines:
<route>
<from uri="activemq:pending?maxConcurrentConsumers=10"/>
<throttle timePeriodMillis="60000">
<constant>1</constant>
<to uri="mock:endpoint"/>
</throttle>
</route>
However, the throttle seems to be applied to the entire set of requests moving to the endpoint, and not to each individual consumer. I have to admit, I was a bit surprised to find that behavior. My expectation was that the throttle would apply to each consumer individually, which would satisfy the SLA in the original question, provided that the messages include the customer ID in the JMSXGroupId header.
I came across a similar problem and finally came up with the solution described here.
My assumptions are:
Order of messages is not important (though it can be solved by re-sequencer)
Total volume of messages per customer ID is not great so the runtime is not saturated.
The solution approach:
Run aggregator for 1 minute while using customerID to assemble messages with the same customer ID into a list
Use Splitter to split the list into individual messages
Send the first message from the splitter to the actual service
Re-route the rest of the list back into the aggregator.
Java DSL version is a bit easier to understand:
final AggregationStrategy aggregationStrategy = AggregationStrategies.flexible(Object.class)
.accumulateInCollection(ArrayList.class);
from("direct:start")
.log("Receiving ${body}")
.aggregate(header("customerID"), aggregationStrategy).completionTimeout(60000)
.log("Aggregate: releasing ${body}")
.split(body())
.choice()
.when(header(Exchange.SPLIT_INDEX).isEqualTo(0))
.log("*** Processing: ${body}")
.to("mock:result")
.otherwise()
.to("seda:delay")
.endChoice();
from("seda:delay")
.delay(0)
.to("direct:start");
Spring XML version looks like the following:
<!-- this is our aggregation strategy defined as a spring bean -->
<!-- see http://stackoverflow.com/questions/27404726/how-does-one-set-the-pick-expression-for-apache-camels-flexibleaggregationstr -->
<bean id="_flexible0" class="org.apache.camel.util.toolbox.FlexibleAggregationStrategy"/>
<bean id="_flexible2" factory-bean="_flexible0" factory-method="accumulateInCollection">
<constructor-arg value="java.util.ArrayList" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:start"/>
<log message="Receiving ${body}"/>
<aggregate strategyRef="_flexible2" completionTimeout="60000" >
<correlationExpression>
<xpath>/order/#customerID</xpath>
</correlationExpression>
<log message="Aggregate: releasing ${body}"/>
<split>
<simple>${body}</simple>
<choice>
<when>
<simple>${header.CamelSplitIndex} == 0</simple>
<log message="*** Processing: ${body}"/>
<to uri="mock:result"/>
</when>
<otherwise>
<log message="--- Delaying: ${body}"/>
<to uri="seda:delay" />
</otherwise>
</choice>
</split>
</aggregate>
</route>
<route>
<from uri="seda:delay"/>
<to uri="direct:start"/>
</route>
</camelContext>