Apache Camel onExceptionOccurred component not redelivering as per the redelivery delay - apache-camel

I have Apache Camel route which invokes restlet component and used redelivery mechanism from the exception handler which performs some processing on every failure to update my database record but if I provide the redelivery delay of 2000, it is taking 24 seconds to retry the every preceding attempt. Below is the piece of code. Let me know why it is taking more delay than expected.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:cxf="http://camel.apache.org/schema/cxf"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd">
<!-- CXF Rest Endpoint Declaration -->
<cxf:rsServer address="http://localhost:9092/rest/corp"
id="FetchCDFRestRequest" serviceClass="com.tcl.Service.Service" />
<bean class="com.tcl.ExceptionOccurredRefProcessor" id="exProc" />
<bean class="org.apache.camel.builder.DeadLetterChannelBuilder"
id="DLCErrorHandler">
<property name="deadLetterUri"
value="activemq:queue:DMS.FAILURES.DLQ" />
<property name="redeliveryPolicy" ref="redeliveryPolicy" />
</bean>
<bean class="org.apache.camel.processor.RedeliveryPolicy"
id="redeliveryPolicy">
<property name="maximumRedeliveries"
value="3" />
<property name="maximumRedeliveryDelay" value="2000" />
<property name="retryAttemptedLogLevel" value="WARN" />
</bean>
<camelContext id="Corp"
xmlns="http://camel.apache.org/schema/spring">
<errorHandler id="eh" onExceptionOccurredRef="exProc">
<redeliveryPolicy id="redeliveryPolicy" />
</errorHandler>
<route errorHandlerRef="DLCErrorHandler"
id="MainRouteOppIDFolder" streamCache="true">
<from id="_CreateOppIDFolder"
uri="restlet:http://localhost:9092/rest/corp/createOppIDFolder?restletMethod=POST" />
----------
<to uri="restlet:http://localhost:9902/CreateOppIDFolder?restletMethod=POST" />
------------
<onException id="_onException1"
onExceptionOccurredRef="exProc"
redeliveryPolicyRef="redeliveryPolicy" useOriginalMessage="true">
<exception>java.lang.Exception</exception>
<handled>
<simple>true</simple>
</handled>
<log id="_log3" loggingLevel="INFO"
message="Handled ex >>>>> ${exception.message} " />
</onException>
</route>
</camelContext>
</beans>
Below are some of the logs
14:47:52.701 [Restlet-1879974483] WARN o.a.c.processor.DeadLetterChannel - Failed delivery for (MessageId: ID-DESKTOP-P2DBOO5-1580115331236-0-19 on ExchangeId: ID-DESKTOP-P2DBOO5-1580115331236-0-20). On delivery attempt: 0 caught: org.apache.camel.component.restlet.RestletOperationException: Restlet operation failed invoking http://localhost:9902/CreateOppIDFolder with statusCode: 500 /n responseBody:org.apache.cxf.interceptor.Fault: Could not send Message.
14:48:15.044 [Camel (MyCamel) thread #15 - ErrorHandlerRedeliveryTask] WARN o.a.c.processor.DeadLetterChannel - Failed delivery for (MessageId: ID-DESKTOP-P2DBOO5-1580115331236-0-19 on ExchangeId: ID-DESKTOP-P2DBOO5-1580115331236-0-20). On delivery attempt: 1 caught: org.apache.camel.component.restlet.RestletOperationException: Restlet operation failed invoking http://localhost:9902/CreateOppIDFolder with statusCode: 500 /n responseBody:org.apache.cxf.interceptor.Fault: Could not send Message.
14:48:37.252 [Camel (MyCamel) thread #17 - ErrorHandlerRedeliveryTask] WARN o.a.c.processor.DeadLetterChannel - Failed delivery for (MessageId: ID-DESKTOP-P2DBOO5-1580115331236-0-19 on ExchangeId: ID-DESKTOP-P2DBOO5-1580115331236-0-20). On delivery attempt: 2 caught: org.apache.camel.component.restlet.RestletOperationException: Restlet operation failed invoking http://localhost:9902/CreateOppIDFolder with statusCode: 500 /n responseBody:org.apache.cxf.interceptor.Fault: Could not send Message.
Any suggestions please?

Are you sure it is taking 24 seconds to (begin the) retry?
Couldn't it take around 24 seconds until the retry fails (end of retry)?
All three log statements are reporting an error 500 from the target endpoint. Is it possible that the retry starts after 2 seconds, makes the HTTP call, but it takes another 20 seconds until the endpoint answers with an error 500?
00:00 error occurs
00:02 Camel triggers retry after 2 seconds
00:02 HTTP call to http://localhost:9902...
00:xx Waiting for reponse from http://localhost:9902
00:2x Log output after 20-something seconds that retry is failed

Related

Camel errorHandler doesn't log before or after sending message to the dead letter channel

We have a simple Camel route like below. But we have noticed that errorHandler log is not working. After some investigation, we understood that errorHandler has different log parameters then other logs(like in route or onException).
We haven't succeeded to use errorHandler log. Yes, there are so many options to log but we want to learn how to use this one. Our aim is to log some texts before or after sending messages to deadletter queue.
How can we use errorHandler log?
apache-camel : 3.19.0
spring-boot : 2.7.5
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd">
<bean id="mdcEnricher" class="com.mycompany.MDCEnricher"/>
<camel:camelContext id="mycontext" xmlns="http://camel.apache.org/schema/spring" useMDCLogging="true">
<camel:errorHandler id="myErrorHandler" type="DeadLetterChannel"
useOriginalBody="true"
deadLetterUri="jms:queue:mybackoutq">
<redeliveryPolicy
maximumRedeliveries="1"
redeliveryDelay="1"
retryAttemptedLogLevel="WARN"
retriesExhaustedLogLevel="ERROR"/>
<log id="logIncomingMsg" logName="com.mypackage"
loggingLevel="ERROR" message="Pushing to backout queue"/>
</camel:errorHandler>
<camel:route id="myRoute" errorHandlerRef="myErrorHandler">
<from uri="jms:queue:myinputq"/>
<bean ref="mdcEnricher" method="enrich"/>
<log message="Received exchange with message id: [${headers.JMSMessageID}], starting processing"/>
<process ref="#class:com.mycompany.processor.MyProcessor"/>
<to uri="jms:queue:myoutputq"/>
<log message="Finished the processing exchange with message id: [${headers.JMSMessageID}]"/>
</camel:route>
Try to change log configuration and loggingLevel to DEBUG but nothing changed.
<springProfile name="local">
<logger name="org.apache.camel" level="DEBUG"/>
<logger name="com.mycompany" level="DEBUG"/>
<root level="DEBUG">
<appender-ref ref="LocalConsole"/>
</root>
</springProfile>

how to use custom error handler with camel rest dsl?

I am trying to use my own complex error handler for a rest dsl route. With myErrorHandler I define my own handler and its works. If occur an exception my handler is called and do his stuff. At the end of the handler I clear my exchange:
private void clearExchange() {
exchange.getIn().reset();
exchange.setException(null);
exchange.getMessage().setHeader(Exchange.HTTP_RESPONSE_CODE, 500);
exchange.getIn().setBody("Server Error");
}
but my main problem is, I don’t get as client a response with status code 500. My Application show me the exception stacktrace. If I use simple onException in my camelcontext, its works but my error handler are not called. I thought if an exception occur, the exchange go to my error handler and after this he go back to the consumer.
<bean id="myErrorHandler" class="org.apache.camel.builder.DefaultErrorHandlerBuilder">
<property name="onExceptionOccurred" ref="myErrorProcess"/>
</bean>
<bean id="myErrorProcess" class="de.myErrorProcess">
<property name="PropertiesFilePath" value="conf\general.properties"/>
</bean>
<camel:camelContext errorHandlerRef="myErrorHandler" xmlns="http://camel.apache.org/schema/spring">
<routeContextRef ref="RouteA"/>
<restConfiguration component="jetty" host="0.0.0.0" scheme="http" port="80"
apiContextPath="api-doc" enableCORS="false">
<componentProperty key="useContinuation" value="false"/>
<dataFormatProperty key="prettyPrint" value="true"/>
</restConfiguration>
<rest>
<get id="getPing" uri="/ping">
<responseMessage code="200" responseModel="de.model.Ping" message="Ok"/>
<to uri="direct:ping"/>
</get>
</rest>
</camel:camelContext>

OnException maximumRedeliveries ignored

In the following route, the maximumRedeliveries clausule from redeliveryPolicy is ignored when we get an exception. We get:
Failed delivery for (MessageId: ID-UW205584-58231-1527668174534-39-248 on ExchangeId: ID-UW205584-58231-1527668174534-39-24). On delivery attempt: 0
Failed delivery for (MessageId: ID-UW205584-58231-1527668174534-39-248 on ExchangeId: ID-UW205584-58231-1527668174534-39-24). On delivery attempt: 1
And then it remains in a infinity loop repeating the last retry. Any idea? Thank you very much community!
Our route looks like follows:
<?xml version="1.0" encoding="ASCII"?>
<routes xmlns="http://camel.apache.org/schema/spring">
<route handleFault="true">
<from uri="switchyard://ProcessTaskEx"/>
<log message="ProcessTaskEx - message received: ${body}" loggingLevel="DEBUG" logName="WebServiceQueues" />
<to uri="switchyard://RequestCapacity"/>
<onException>
<exception>java.lang.Exception</exception>
<exception>webservicequeues.utilities.WebServiceQueueException</exception>
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="6000" maximumRedeliveryDelay="90000" retriesExhaustedLogLevel="INFO" retryAttemptedLogLevel="INFO"/>
<handled>
<constant>true</constant>
</handled>
<log message="Failed after Retry.Sending ProcessTask Request to Error Queue" loggingLevel="ERROR" logName="WebServiceQueues" />
<to uri="switchyard://ErrorProcessTaskExQueue"/>
</onException>
</route>
</routes>
Because you get an infinite loop, it sounds like the message header CamelRedeliveryCounter gets overwritten every time and therefore it never reaches the maximumRedeliveries of 2.
Is it possible that the call to the endpoint where the error occurs deletes or resets message headers? Particularly CamelRedeliveryCounter?

Apache Camel: <onException> block doesn't work after upgrading to camel 2.16.1

I have the onException block below that worked before I upgraded to from camel 2.14 to camel 2.16.1. Before the update, I would get my error caught and printed in the log - "Error posting to MR". After I upgraded to camel 2.16.1, I still get my error in the log, but now, always, my handled error is followed by another timestamp with what appears to be camel default error handler for the error I think I already handled. It looks like this:
"2016-01-26 15:07:09,571 [Camel (mrPostContext) thread #56 - JmsConsumer[mrPost]] ERROR org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId"....
I don't know if I'm doing something wrong, I tried using java.lang.Throwable instead of java.lang.Exception but so far had no luck. Haven't found anything helpful in documentation yet. As I'm far from being good with camel, I will much appreciate some help.
<route id="mrPost">
<from uri="activemq:mrPost?concurrentConsumers=8" />
<onException>
<exception>java.lang.Exception</exception>
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="1000"/>
<handled>
<constant>true</constant>
</handled>
<to uri="activemq:mr-post-fail" />
<log loggingLevel="ERROR" message="Error posting to MR body:${out.body} exception message: ${exception.message} body:${exception.responseBody}" />
</onException>
<setHeader headerName="CamelHttpMethod">
<constant>PUT</constant>
</setHeader>
<recipientList>
<simple>{{mr.source}}/${headers.id}</simple>
</recipientList>
</route>
Here is a much cleaner solution in case someone is looking for it:
<errorHandler id="loggingErrorHandler" type="LoggingErrorHandler" logName="LoggingErrorHandler" level="OFF"/>
<camelContext id="mrPostContext" trace="false" errorHandlerRef="loggingErrorHandler" >
I think that camel is failing where it puts the message on to the error queue mr-post-fail and is throwing an exception when doing so, hence your exception is propogating to the DefaultExceptionHandler. Since the logging is after the put queue call it is not working as you expected.
If not the case can you give the whole line of the error thats being printed.
"2016-01-26 15:07:09,571 [Camel (mrPostContext) thread #56 - JmsConsumer[mrPost]] ERROR org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId"..
I got the defaultErrorHandler to collapse with these 2 adjustments:
- set messageHistory="false" on CONTEXT
- set logStackTrace="false" for onException block, like this:
<redeliveryPolicy maximumRedeliveries="2" redeliveryDelay="1000" logStackTrace="false" />
The timestamp with default error handler still shows up, but much easier on the eye, it looks like this:
2016-02-01 11:26:27,792 [Camel (mrPostContext) thread #262 - JmsConsumer[mrPost]] ERROR org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId: ID-qn7nwscms01-com-44939-1454025367155-1569-4 on ExchangeId: ID-qn7nwscms01-com-44939-1454025367155-1569-5). Exhausted after delivery attempt: 3 caught: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://mr.cloud.dig.com/api/contents/news/30939510 with statusCode: 409. Processed by failure processor: FatalFallbackErrorHandler[Pipeline[[Channel[SetHeader(CamelExceptionCaught, Simple: Headers: ${headers} Message: ${exception.message} Body: ${exception.responseBody})], Channel[sendTo(Endpoint[activemq://mr-post-fail])], Channel[choice{when Filter[if: Simple: ${exception.message} not contains "statusCode: 409" do: Pipeline[[Channel[SetHeader(Content-Type, text/plain)], Channel[SetHeader(to, Simple: lw#ac.com)], Channel[SetHeader(subject, Simple: Error posting to MR)], Channel[SetBody(Simple: ${header.CamelExceptionCaught})]]]]}], Channel[Log(mrPost)[Error posting to MR ID: ${headers.contentId}, type:${headers.cmsContentType}, modified:${headers.lastModifiedDate}, getTime:${headers.lastModifiedGetTime}]], Channel[Log(merlinPost)[Error posting to MR body:${out.body} exception message: ${exception.message} body:${exception.responseBody}]]]]]
I hope there is a way to turn it off completely.
I also tried using defaultErrorHandler setup:
<errorHandler id="defaultErrorHandler" type="DefaultErrorHandler">
<redeliveryPolicy logStackTrace="false"/>
</errorHandler>
but it makes no impact.
You can use like the example in this page: http://camel.apache.org/exception-clause.html
<!-- setup our error handler as the deal letter channel -->
<bean id="errorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
<property name="deadLetterUri" value="activemq:mr-post-fail"/>
</bean>
<!-- this is the camel context where we define the routes -->
<!-- define our error handler as a global error handler -->
<camelContext errorHandlerRef="errorHandler" xmlns="http://camel.apache.org/schema/spring">
<route>
<!-- the route -->
<from uri="activemq:mrPost?concurrentConsumers=8" />
<setHeader headerName="CamelHttpMethod">
<constant>PUT</constant>
</setHeader>
<recipientList>
<simple>{{mr.source}}/${headers.id}</simple>
</recipientList>
</route>
</camelContext>
Or, set in rote, to the error handle work only with an specific route
<route errorHandlerRef="errorHandler">
...
</route>
And remove from camelContext

Camel Split and Aggregate failing because messages going to multiple concurrent consumers

I have a simple camel route that takes a list of items, splits them sending each element to a mq node for processing then joins them back together via an aggregator.
Very close to the Composed Message Processor: http://camel.apache.org/composed-message-processor.html
But we noticed that after the split, camel will create multiple concurrent consumers? or exchanges? Since the message is being sent to multiple consumers they never complete.
List: 1,2,3,4
Split: amq::process_each_item
Aggregate:
[Camel (camel-3) thread #41 - Aggregating 1 - Waiting on 3 more items
[Camel (camel-1) thread #16 - Aggregating 2 - Waiting on 3 more items
[Camel (camel-3) thread #49 - Aggregating 3 - Waiting on 2 more items
[Camel (camel-1) thread #15 - Aggregating 4 - Waiting on 2 more items
So, camel spawned 2 aggregators, and each is waiting on 4 items, but they only ever get two each.
Camel Route:
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route> <!-- This route splits the reg request into it's items. Adding needed info to the message header. -->
<from uri="activemq:registration.splitByItemQueue" /> <!-- pick up the reg req -->
<setHeader headerName="regReqId"> <!-- Need to store the Reg Req in the header -->
<simple>${body.registrationRequest.id}</simple>
</setHeader>
<split parallelProcessing="false" strategyRef="groupedExchangeAggregator"> <!-- Split the RegRequestInfo into it's individual requestItems (add, drop, etc) -->
<method ref="requestSplitter" method="split" /> <!-- does the actual splitting -->
<setHeader headerName="JMSXGroupID"> <!-- This is CRITICAL. It is how we ensure valid seat check counts without db locking -->
<simple>FOID=${body.formatOfferingId}</simple> <!-- grouping on the foid -->
</setHeader>
<to uri="activemq:registration.lprActionQueue"/> <!-- send to queue's for processing-->
</split>
</route>
<route> <!-- performs the registration + seat check -->
<from uri="activemq:registration.lprActionQueue" />
<bean ref="actionProcessor" method="process"/> <!-- go to the java code that makes all the decisions -->
<to uri="activemq:registration.regReqItemJoinQueue"/> <!-- send to join queue's for final processing-->
</route>
<route> <!-- This route joins items from the reg req item split. Once all items have completed, update state-->
<from uri="activemq:registration.regReqItemJoinQueue" /> <!-- Every Reg Req Item will come here-->
<aggregate strategyRef="groupedExchangeAggregator" ignoreInvalidCorrelationKeys="false" completionFromBatchConsumer="true"> <!-- take all the Reg Req Items an join them to their req -->
<correlationExpression>
<header>regReqId</header> <!-- correlate on the regReqId we stored in the header -->
</correlationExpression>
<bean ref="actionProcessor" method="updateRegistrationRequestStatus"/> <!-- update status -->
</aggregate>
</route>
</camelContext>
<bean id="groupedExchangeAggregator" class="org.apache.camel.processor.aggregate.GroupedExchangeAggregationStrategy" />
On my local machine the above works fine, but when we deploy to our test server half of the messages go to one camel aggregator, half to the other. causing none to ever finish. Notice in the config below that we've set concurrent consumers to 1 for camel.
Here's the camel / activemq config
<amq:broker useJmx="false" persistent="false">
<amq:plugins>
<amq:statisticsBrokerPlugin />
</amq:plugins>
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0" />
</amq:transportConnectors>
</amq:broker>
<!-- Basic AMQ connection factory -->
<amq:connectionFactory id="amqConnectionFactory" brokerURL="vm://localhost" />
<!-- Wraps the AMQ connection factory in Spring's caching (ie: pooled) factory
From the AMQ "Spring Support"-page: "You can use the PooledConnectionFactory for efficient pooling... or you
can use the Spring JMS CachingConnectionFactory to achieve the same effect."
See "Consuming JMS from inside Spring" at http://activemq.apache.org/spring-support.html
Also see http://codedependents.com/2010/07/14/connectionfactories-and-caching-with-spring-and-activemq/
Note: there are pros/cons to using Spring's caching factory vs Apache's PooledConnectionFactory; but, until
we have more explicit reasons to favor one over the other, Spring's is less tightly-coupled to a specific
AMQP-implementation.
See http://stackoverflow.com/a/19594974
-->
<bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<constructor-arg ref="amqConnectionFactory"/>
<property name="sessionCacheSize" value="1"/>
</bean>
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<constructor-arg ref="connectionFactory" />
</bean>
<bean id="jmsConfig"
class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="concurrentConsumers" value="1"/>
</bean>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig"/>
</bean>
Turns out we had another spring context / servlet importing our config. We believe this was the issue.

Resources