I have a simple camel route that takes a list of items, splits them sending each element to a mq node for processing then joins them back together via an aggregator.
Very close to the Composed Message Processor: http://camel.apache.org/composed-message-processor.html
But we noticed that after the split, camel will create multiple concurrent consumers? or exchanges? Since the message is being sent to multiple consumers they never complete.
List: 1,2,3,4
Split: amq::process_each_item
Aggregate:
[Camel (camel-3) thread #41 - Aggregating 1 - Waiting on 3 more items
[Camel (camel-1) thread #16 - Aggregating 2 - Waiting on 3 more items
[Camel (camel-3) thread #49 - Aggregating 3 - Waiting on 2 more items
[Camel (camel-1) thread #15 - Aggregating 4 - Waiting on 2 more items
So, camel spawned 2 aggregators, and each is waiting on 4 items, but they only ever get two each.
Camel Route:
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route> <!-- This route splits the reg request into it's items. Adding needed info to the message header. -->
<from uri="activemq:registration.splitByItemQueue" /> <!-- pick up the reg req -->
<setHeader headerName="regReqId"> <!-- Need to store the Reg Req in the header -->
<simple>${body.registrationRequest.id}</simple>
</setHeader>
<split parallelProcessing="false" strategyRef="groupedExchangeAggregator"> <!-- Split the RegRequestInfo into it's individual requestItems (add, drop, etc) -->
<method ref="requestSplitter" method="split" /> <!-- does the actual splitting -->
<setHeader headerName="JMSXGroupID"> <!-- This is CRITICAL. It is how we ensure valid seat check counts without db locking -->
<simple>FOID=${body.formatOfferingId}</simple> <!-- grouping on the foid -->
</setHeader>
<to uri="activemq:registration.lprActionQueue"/> <!-- send to queue's for processing-->
</split>
</route>
<route> <!-- performs the registration + seat check -->
<from uri="activemq:registration.lprActionQueue" />
<bean ref="actionProcessor" method="process"/> <!-- go to the java code that makes all the decisions -->
<to uri="activemq:registration.regReqItemJoinQueue"/> <!-- send to join queue's for final processing-->
</route>
<route> <!-- This route joins items from the reg req item split. Once all items have completed, update state-->
<from uri="activemq:registration.regReqItemJoinQueue" /> <!-- Every Reg Req Item will come here-->
<aggregate strategyRef="groupedExchangeAggregator" ignoreInvalidCorrelationKeys="false" completionFromBatchConsumer="true"> <!-- take all the Reg Req Items an join them to their req -->
<correlationExpression>
<header>regReqId</header> <!-- correlate on the regReqId we stored in the header -->
</correlationExpression>
<bean ref="actionProcessor" method="updateRegistrationRequestStatus"/> <!-- update status -->
</aggregate>
</route>
</camelContext>
<bean id="groupedExchangeAggregator" class="org.apache.camel.processor.aggregate.GroupedExchangeAggregationStrategy" />
On my local machine the above works fine, but when we deploy to our test server half of the messages go to one camel aggregator, half to the other. causing none to ever finish. Notice in the config below that we've set concurrent consumers to 1 for camel.
Here's the camel / activemq config
<amq:broker useJmx="false" persistent="false">
<amq:plugins>
<amq:statisticsBrokerPlugin />
</amq:plugins>
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0" />
</amq:transportConnectors>
</amq:broker>
<!-- Basic AMQ connection factory -->
<amq:connectionFactory id="amqConnectionFactory" brokerURL="vm://localhost" />
<!-- Wraps the AMQ connection factory in Spring's caching (ie: pooled) factory
From the AMQ "Spring Support"-page: "You can use the PooledConnectionFactory for efficient pooling... or you
can use the Spring JMS CachingConnectionFactory to achieve the same effect."
See "Consuming JMS from inside Spring" at http://activemq.apache.org/spring-support.html
Also see http://codedependents.com/2010/07/14/connectionfactories-and-caching-with-spring-and-activemq/
Note: there are pros/cons to using Spring's caching factory vs Apache's PooledConnectionFactory; but, until
we have more explicit reasons to favor one over the other, Spring's is less tightly-coupled to a specific
AMQP-implementation.
See http://stackoverflow.com/a/19594974
-->
<bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<constructor-arg ref="amqConnectionFactory"/>
<property name="sessionCacheSize" value="1"/>
</bean>
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<constructor-arg ref="connectionFactory" />
</bean>
<bean id="jmsConfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="concurrentConsumers" value="1"/>
</bean>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig"/>
</bean>
Turns out we had another spring context / servlet importing our config. We believe this was the issue.
Related
Below is my camel context, but reading vlaue from property file doesnt work, it is not listening to Queue which is mentioned in queueName. Is using poll
enrich A good Idea, because it uses direct component?
<bean id="bridgePropertyPlaceholder"
class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
<property name="location" value="classpath:/config/queue.properties"/>
</bean>
<Route>
<from uri="activemq:queue:{{queueName}}/>
.......
</Route>
I'm trying to pick a file from a directory, split a file and add each splitted lines to activemq. I'm facing a problem with exception handling during this process. Lets say a file in the directory is a binary file (executable), then splitter thows org.apache.camel.RuntimeCamelException and java.nio.charset.MalformedInputException exceptions. If this occurs, then I need to catch these exceptions, nothing should add in activemq and that just particular thread should exit after logging exception. Referred online and wrote the following code but don't know how to stop adding to activemq and quit specific thread.
<route id="msg_producer">
<from uri="input.file.from" />
<doTry>
<split parallelProcessing="true" executorServiceRef="msgProducer"
streaming="true">
<tokenize token="\n"></tokenize>
<to uri="input.activemq.to" />
</split>
<doCatch>
<exception>org.apache.camel.RuntimeCamelException</exception>
<exception>java.nio.charset.MalformedInputException</exception>
<handled> <constant>true</constant></handled>
<setBody>
<simple>${exception.stacktrace}</simple>
</setBody>
<setHeader headerName="CamelFileName">
<simple>${file:onlyname.noext}_error.log</simple>
</setHeader>
</doCatch>
</doTry>
</route>
as #claus Ibsen said to filter the files you can use a filefilter property so that you pick only files based on extension and some standard pattern something like this
<bean id="FileFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
<!-- ? matches one character
* matches zero or more characters
** matches zero or more directories in a path -->
<property name="includes" value="#{databaseProperties.getProperties().getProperty('file.name.pattern')}"/>
<!-- if you wan to exclude specific files say bad in name or .exe files. Use comma to separate multiple excludes -->
<!-- <property name="excludes" value="**/*bad*,**/*.exe"/> -->
</bean>
and your file.name.pattern can be something like this **/contract.csv
I have written a camel route that polls a file folder, picks up request, checks for memory consumption on server (in a java file). If its below threshold it drops the request on a JMS queue otherwise it throws an exception and picks it again for processing.
What i need to do is that when exception is thrown i need to delay processing for a configurable amount of time say 15 mins. This will give some time for server to stabilize instead of keeping it polling unnecessarily.
I am using errorHandler mechanism of camel however it doesnt seem to work. Camle keeps on picking up the request without any delay. Please help with this issue.
Below is the bundle context snapshot:
<camel:onException>
<camel:exception>java.lang.Exception</camel:exception>
<camel:redeliveryPolicy backOffMultiplier="500" />
<camel:log message="Default error handler was called"></camel:log>
</camel:onException>
<camel:route>
<!-- Reading from REST url -->
<camel:from uri="<my url>" />
<!-- If else loop -->
<camel:choice>
<camel:when>
<camel:xpath>Some path</camel:xpath>
<!-- Parsing OrderNumber and OrderVersion-->
<camel:log message="Recieved request ${headers.OrderNumber}-${headers.OrderVersion}.xml"/>
<camel:setHeader headerName="OrderNumber">
<xpath>Some path</xpath>
</camel:setHeader>
<camel:setHeader headerName="OrderVersion">
<camel:xpath>Some path</camel:xpath>
</camel:setHeader>
<!-- Request being put in file folder -->
<to
uri="file:data/inbox?fileName=${header.OrderNumber}-${header.OrderVersion}.xml"
pattern="InOut" />
</camel:when>
<!-- For all other requests put on queue -->
<camel:otherwise>
<camel:log message="Request ${headers.OrderNumber}-${headers.OrderVersion}.xml directly sent to queue"/>
<to uri="my queue"
pattern="InOut" />
</camel:otherwise>
</camel:choice>
</camel:route>
<camel:route errorHandlerRef="errorHandler">
<!-- Route to put message from folder to JMS queue if memory consumption is below limit-->
<camel:from uri="file:data/inbox"/>
<camel:process ref="checkMemoryConsumption"/>
<camel:convertBodyTo type="String" />
<camel:log message="Sucessfully processing service order ${headers.OrderNumber}-${headers.OrderVersion}.xml"/>
<to uri="my queue"
pattern="InOut" />
</camel:route>
Did you try redeliveryDelay of redeliveryPolicyProfile?
Below policy profile will retry 3 times with the delay of 1000mS between each try.
<redeliveryPolicyProfile id="myRedeliveryProfile"
maximumRedeliveries="3"
redeliveryDelay="1000" allowRedeliveryWhileStopping="false"
retryAttemptedLogLevel="INFO" />
Read more on here
By making below modification in my bundle context (bundle context displayed above), re-delivery policy kicked in and seems to do the trick.
I have been playing around with camel-netty and camel-netty-http components, and trying to figure out what the setting maximumPoolSize does.
However, from what i observe based on this is that, the OrderPool always processes 16 concurrent requests. I am trying to change the maximumPoolSize to a value of 5 like the route as below,
<bean id="nettyBean" class="com.redhat.NettyTestImpl">
<property name="message" value="Netty maximumPoolSize test"/>
</bean>
<camelContext trace="false" xmlns="http://camel.apache.org/schema/blueprint" autoStartup="true">
<route>
<from uri="netty-http:http://localhost:8080/hello?maximumPoolSize=5&sync=true"/>
<log message="Forwarding to Netty component ....."/>
<setBody>
<method ref="nettyBean" method="sayHi"/>
</setBody>
<delay>
<constant>3000</constant>
</delay>
<log message="The body contains : ${body}"/>
</route>
</camelContext>
But it seems that i cannot get the maximumPoolSize to set to a value. So, what am i doing wrong ? How can i get the maximumPoolSize set ?
I check this by load testing with 20 concurrent requests and all are processed.
I just checked the code of camel-netty and camel-netty-http, maximumPoolSize is used for the OrderPool, you can use jconsoler to check the thread pool size.
As the OrderPool is used to handle the processor of camel, netty still has the work thread to handle the connection, which means it can still server 20 concurrent request, this time. But the TPS could be just about 5.
I am working with Apache Camel Bindy to process csv files of different data models.
e.g. file one is of data model on and file two is of data model two.
In the camel route, I associated two calls of the BindyCsvDataFormat with different data models as:
<bean id="bindyDataformat" class="org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat">
<constructor-arg name="type" value="com.barclays.creditit.cls.eoddata.model.risk.DataModel1" />
</bean>
<bean id="aBindyDataformat" class="org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat">
<constructor-arg name="type" value="DataModel2" />
</bean>
route looks like this:
<from uri="direct:start"/>
<bean ref="fileReader"/>
<unmarshal ref="bindyDataformat" />
<bean ref="flattener"/>
<bean ref="fileReader"/>
<unmarshal ref="aBindyDataformat" />
<bean ref="flattener"/>
When I run the code though, the factory has two models associated automatically, not one per run. And both the files are read into objects of the first data model and never the second data model. Any suggestions about how I could get this to work?
Thanks!
Create two different routes with different file filters and process them separately with one of the Bindy formaters.