I have been playing around with camel-netty and camel-netty-http components, and trying to figure out what the setting maximumPoolSize does.
However, from what i observe based on this is that, the OrderPool always processes 16 concurrent requests. I am trying to change the maximumPoolSize to a value of 5 like the route as below,
<bean id="nettyBean" class="com.redhat.NettyTestImpl">
<property name="message" value="Netty maximumPoolSize test"/>
</bean>
<camelContext trace="false" xmlns="http://camel.apache.org/schema/blueprint" autoStartup="true">
<route>
<from uri="netty-http:http://localhost:8080/hello?maximumPoolSize=5&sync=true"/>
<log message="Forwarding to Netty component ....."/>
<setBody>
<method ref="nettyBean" method="sayHi"/>
</setBody>
<delay>
<constant>3000</constant>
</delay>
<log message="The body contains : ${body}"/>
</route>
</camelContext>
But it seems that i cannot get the maximumPoolSize to set to a value. So, what am i doing wrong ? How can i get the maximumPoolSize set ?
I check this by load testing with 20 concurrent requests and all are processed.
I just checked the code of camel-netty and camel-netty-http, maximumPoolSize is used for the OrderPool, you can use jconsoler to check the thread pool size.
As the OrderPool is used to handle the processor of camel, netty still has the work thread to handle the connection, which means it can still server 20 concurrent request, this time. But the TPS could be just about 5.
Related
I have an Apache Camel route which invokes restlet component. To better understand what is happening at runtime I also keep the execution state in a database (number of retries, last execution state). I would like to use the redelivery mechanism from the exception handler but which performs some processing on every failure to update my database record.
<bean class="com.tcl.ExceptionOccurredRefProcessor" id="exProc" />
<camelContext errorHandlerRef="eh">
<errorHandler id="eh" onExceptionOccurredRef="exProc"> <redeliveryPolicy
maximumRedeliveries="3" maximumRedeliveryDelay="2000"
retryAttemptedLogLevel="WARN" />
</errorHandler>
<to uri="-------"/> //Invoke rest service
<onException id="_onException1" onExceptionOccurredRef="exProc" useOriginalMessage="true">
<exception>java.lang.Exception</exception>
<handled>
<simple>true</simple>
</handled>
...........
</onException>
</camelContext>
For this I tried to use onExceptionOccurred component with redelivery component and referred on camelContext as in above code but the message got delivered only once and stopped the execution.
Any suggestions please?
First of all: the code you pasted is a bit confusing because there is no Camel route, just a plain to element that cannot be alone in the Camel context.
However, I guess your problem is a "hidden" Camel handling. As documented in the Camel Docs of the Exception Clause, the onException clause overrides the maximumRedeliveries to 0 by default unless you explicitly set the option.
<onException>
<exception>java.lang.Exception</exception> // max. redelivery = 0
</onException>
<onException>
<exception>java.lang.Exception</exception>
<redeliveryPolicy maximumRedeliveries="3"/> // max. redelivery = 3
</onException>
I an not sure about the other redelivery options like maximumRedeliveryDelay, but at least the maximumRedeliveries is overridden if not explicitly set.
So I have the following route (Camel 2.20.0)
I was working on a global <onException> block for a new route. For some reason it wasn't firing, so I moved the items to a doTry/doCatch within one specific route just to play with the error handeling.
<camelContext xmlns="http://camel.apache.org/schema/spring" id="jobfeedCamelContext">
<route id="testError">
<from uri="timer://runOnce?repeatCount=1&delay=5000" />
<doTry>
<throwException exceptionType="java.lang.Throwable"/>
<to uri="errorBean"/> <!-- bean does nothing but explicitly throws java.lang.Throwable -->
<doCatch>
<exception>java.lang.Throwable</exception>
<log message="### exception" />
</doCatch>
</doTry>
<log message="### out of try" />
</route>
</camelContext>
For output I get the stack trace from the beans java.Lang.Throwable (but no stacktrace is generated for the<throwException exceptionType="java.lang.Throwable"/>. I do not get my "### exception" log entry in any scenario, but I do get the "### out of try" log entry.
Have used this functionality in other routes on older version of camel, so I can't really see where I am going wrong. Anyone have any ideas? I've turned on route tracing and there is nothing helpful.
<throwException exceptionType="java.lang.Throwable" message="some text"/>
I have the onException block below that worked before I upgraded to from camel 2.14 to camel 2.16.1. Before the update, I would get my error caught and printed in the log - "Error posting to MR". After I upgraded to camel 2.16.1, I still get my error in the log, but now, always, my handled error is followed by another timestamp with what appears to be camel default error handler for the error I think I already handled. It looks like this:
"2016-01-26 15:07:09,571 [Camel (mrPostContext) thread #56 - JmsConsumer[mrPost]] ERROR org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId"....
I don't know if I'm doing something wrong, I tried using java.lang.Throwable instead of java.lang.Exception but so far had no luck. Haven't found anything helpful in documentation yet. As I'm far from being good with camel, I will much appreciate some help.
<route id="mrPost">
<from uri="activemq:mrPost?concurrentConsumers=8" />
<onException>
<exception>java.lang.Exception</exception>
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="1000"/>
<handled>
<constant>true</constant>
</handled>
<to uri="activemq:mr-post-fail" />
<log loggingLevel="ERROR" message="Error posting to MR body:${out.body} exception message: ${exception.message} body:${exception.responseBody}" />
</onException>
<setHeader headerName="CamelHttpMethod">
<constant>PUT</constant>
</setHeader>
<recipientList>
<simple>{{mr.source}}/${headers.id}</simple>
</recipientList>
</route>
Here is a much cleaner solution in case someone is looking for it:
<errorHandler id="loggingErrorHandler" type="LoggingErrorHandler" logName="LoggingErrorHandler" level="OFF"/>
<camelContext id="mrPostContext" trace="false" errorHandlerRef="loggingErrorHandler" >
I think that camel is failing where it puts the message on to the error queue mr-post-fail and is throwing an exception when doing so, hence your exception is propogating to the DefaultExceptionHandler. Since the logging is after the put queue call it is not working as you expected.
If not the case can you give the whole line of the error thats being printed.
"2016-01-26 15:07:09,571 [Camel (mrPostContext) thread #56 - JmsConsumer[mrPost]] ERROR org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId"..
I got the defaultErrorHandler to collapse with these 2 adjustments:
- set messageHistory="false" on CONTEXT
- set logStackTrace="false" for onException block, like this:
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="1000" logStackTrace="false" />
The timestamp with default error handler still shows up, but much easier on the eye, it looks like this:
2016-02-01 11:26:27,792 [Camel (mrPostContext) thread #262 - JmsConsumer[mrPost]] ERROR org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId: ID-qn7nwscms01-com-44939-1454025367155-1569-4 on ExchangeId: ID-qn7nwscms01-com-44939-1454025367155-1569-5). Exhausted after delivery attempt: 3 caught: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://mr.cloud.dig.com/api/contents/news/30939510 with statusCode: 409. Processed by failure processor: FatalFallbackErrorHandler[Pipeline[[Channel[SetHeader(CamelExceptionCaught, Simple: Headers: ${headers} Message: ${exception.message} Body: ${exception.responseBody})], Channel[sendTo(Endpoint[activemq://mr-post-fail])], Channel[choice{when Filter[if: Simple: ${exception.message} not contains "statusCode: 409" do: Pipeline[[Channel[SetHeader(Content-Type, text/plain)], Channel[SetHeader(to, Simple: lw#ac.com)], Channel[SetHeader(subject, Simple: Error posting to MR)], Channel[SetBody(Simple: ${header.CamelExceptionCaught})]]]]}], Channel[Log(mrPost)[Error posting to MR ID: ${headers.contentId}, type:${headers.cmsContentType}, modified:${headers.lastModifiedDate}, getTime:${headers.lastModifiedGetTime}]], Channel[Log(merlinPost)[Error posting to MR body:${out.body} exception message: ${exception.message} body:${exception.responseBody}]]]]]
I hope there is a way to turn it off completely.
I also tried using defaultErrorHandler setup:
<errorHandler id="defaultErrorHandler" type="DefaultErrorHandler">
<redeliveryPolicy logStackTrace="false"/>
</errorHandler>
but it makes no impact.
You can use like the example in this page: http://camel.apache.org/exception-clause.html
<!-- setup our error handler as the deal letter channel -->
<bean id="errorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
<property name="deadLetterUri" value="activemq:mr-post-fail"/>
</bean>
<!-- this is the camel context where we define the routes -->
<!-- define our error handler as a global error handler -->
<camelContext errorHandlerRef="errorHandler" xmlns="http://camel.apache.org/schema/spring">
<route>
<!-- the route -->
<from uri="activemq:mrPost?concurrentConsumers=8" />
<setHeader headerName="CamelHttpMethod">
<constant>PUT</constant>
</setHeader>
<recipientList>
<simple>{{mr.source}}/${headers.id}</simple>
</recipientList>
</route>
</camelContext>
Or, set in rote, to the error handle work only with an specific route
<route errorHandlerRef="errorHandler">
...
</route>
And remove from camelContext
I'm using activeMQ 5.9.
I'm trying to implement an interception type route in my activemq.xml, where I check if a particular header equals some value then send it to a different queue, otherwise allow it to continue.
I'm following the info here: http://activemq.apache.org/broker-camel-component.html
My camel.xml file looks like this:
<camelContext id="camel" trace="false" xmlns="http://camel.apache.org/schema/spring">
<route id="routeAboveQueueLimitTest">
<from uri="activemq:queue:do.something"/>
<choice>
<when>
<simple>${header.scope} == 'test'</simple>
<to uri="activemq:queue:test.do.something"/>
</when>
<otherwise>
<to uri="activemq:queue:do.something"/>
</otherwise>
</choice>
</route>
</camelContext>
Then when I put a message on "activemq:queue:do.something" with header called scope = "test" it correctly routes to the "activemq:queue:test.do.something" queue. However, when it doesn't have that header, it puts it back on the "activemq:queue:do.something" queue and processes it again and again and again!
That kind of seems logical, but the above page clearly says that you have to explicitly send it back to the broker component, and the 2nd example on the page shows exactly that.
I realise this could be worked around by sending it to a different queue if it doesn't have the header but that is undesirable at this stage.
I think the intercept pattern would be much better suited for what you are looking.
<intercept>
<when><simple>${header.scope} == 'test'</simple></when>
<to uri="activemq:queue:test.do.something"/>
</intercept>
More info here: http://camel.apache.org/intercept.html
This will allow messages without the scope header set to 'test' to continue, but will redirect messages that do have the test header.
InterceptSendToEndpoint is a better option here...
<interceptSendToEndpoint uri="activemq:queue:do.something">
<when><simple>${header.scope} == 'test'</simple></when>
<to uri="activemq:queue:test.do.something"/>
<stop/>
</interceptSendToEndpoint>
I would like to know if it's possible with Camel to do throttling based on the content of the exchange.
The situation is the following: I have to call a webservice via soap. Among, the parameters sent to that webservice there is a customerId. The problem is that the webservice send back an error if there are more than 1 request per minute for a given customerId.
I'm wondering if it would be possible to implement throttling per customerId with Camel. So the throttling should not be implemented for all messages but only for messages with the same customerId.
Let me know how I could implement this or if I need to clarify my question.
ActiveMQ Message Groups is designed to handle this case. So, if you can introduce a JMS queue hop in your route, then just set the JMSXGroupId header to the customerId. Then in another route, you can consume from this queue and send to your web service to get the behavior you described.
also see http://camel.apache.org/parallel-processing-and-ordering.html for more information...
While ActiveMQ Message Groups would definitely address the parallel processing of unique customer ID's, in my assessment Claus is correct that introducing a throttle for each unique group represents an unimplemented feature for Camel/ActiveMQ.
Message Groups alone will not meet the SLA described. While each group of messages (correlated by the customer ID) will be processed in order with one thread per group, as long as requests take less than a minute to receive a response, the requirement of one request per minute per customer would not be enforced.
That said, I would be very interested to know if it would be possible to combine Message Groups and a throttle strategy in a way that would simulate the feature request in JIRA. My attempts so far have failed. I was thinking something along these lines:
<route>
<from uri="activemq:pending?maxConcurrentConsumers=10"/>
<throttle timePeriodMillis="60000">
<constant>1</constant>
<to uri="mock:endpoint"/>
</throttle>
</route>
However, the throttle seems to be applied to the entire set of requests moving to the endpoint, and not to each individual consumer. I have to admit, I was a bit surprised to find that behavior. My expectation was that the throttle would apply to each consumer individually, which would satisfy the SLA in the original question, provided that the messages include the customer ID in the JMSXGroupId header.
I came across a similar problem and finally came up with the solution described here.
My assumptions are:
Order of messages is not important (though it can be solved by re-sequencer)
Total volume of messages per customer ID is not great so the runtime is not saturated.
The solution approach:
Run aggregator for 1 minute while using customerID to assemble messages with the same customer ID into a list
Use Splitter to split the list into individual messages
Send the first message from the splitter to the actual service
Re-route the rest of the list back into the aggregator.
Java DSL version is a bit easier to understand:
final AggregationStrategy aggregationStrategy = AggregationStrategies.flexible(Object.class)
.accumulateInCollection(ArrayList.class);
from("direct:start")
.log("Receiving ${body}")
.aggregate(header("customerID"), aggregationStrategy).completionTimeout(60000)
.log("Aggregate: releasing ${body}")
.split(body())
.choice()
.when(header(Exchange.SPLIT_INDEX).isEqualTo(0))
.log("*** Processing: ${body}")
.to("mock:result")
.otherwise()
.to("seda:delay")
.endChoice();
from("seda:delay")
.delay(0)
.to("direct:start");
Spring XML version looks like the following:
<!-- this is our aggregation strategy defined as a spring bean -->
<!-- see http://stackoverflow.com/questions/27404726/how-does-one-set-the-pick-expression-for-apache-camels-flexibleaggregationstr -->
<bean id="_flexible0" class="org.apache.camel.util.toolbox.FlexibleAggregationStrategy"/>
<bean id="_flexible2" factory-bean="_flexible0" factory-method="accumulateInCollection">
<constructor-arg value="java.util.ArrayList" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:start"/>
<log message="Receiving ${body}"/>
<aggregate strategyRef="_flexible2" completionTimeout="60000" >
<correlationExpression>
<xpath>/order/#customerID</xpath>
</correlationExpression>
<log message="Aggregate: releasing ${body}"/>
<split>
<simple>${body}</simple>
<choice>
<when>
<simple>${header.CamelSplitIndex} == 0</simple>
<log message="*** Processing: ${body}"/>
<to uri="mock:result"/>
</when>
<otherwise>
<log message="--- Delaying: ${body}"/>
<to uri="seda:delay" />
</otherwise>
</choice>
</split>
</aggregate>
</route>
<route>
<from uri="seda:delay"/>
<to uri="direct:start"/>
</route>
</camelContext>