I have a in/out producer in Camel that only hangs around for a limited time before getting back to the caller. Some times this naturally results in a dead letter item and an exception being caught by the caller when the response is late.
What I would like to do is have the caller receive a timeout message instead of an exception and the item to never end up in the DLQ. Naturally I could put a listener on the DLQ but as the item has a home to go to it shouldn't really ever get to the DLQ.
Does anyone have a pattern for this? How would it be done? There are redundant consumer patterns (see Camel in Action link) but this is kind of a combined producer/consumer problem generated by the in/out pattern.
Sounds like you are using the Dead Letter Channel error handler, try using the noErrorHandler - http://camel.apache.org/error-handler
Related
I'm having some issues trying to figure out the solution for this problem:
I need to implement a DLC on Apache Camel, though when message are dequeued from the dead letter queue I have on ActiveMQ, every single one of them has to be compared with the latest massage present on another AMQ queue.
So to be clear: when Camel is consuming from queue1 (dead letter queue) the message M1, before trying to resend it to a certain route, it has to compare M1 (for example header comparison) with the latest message present on queue2, M2. M2 is not to be removed from queue2 (it shall serve also for the next comparison) while M1 has to be removed from queue1.
I want to understand if this is possible and which EIP I'm missing in order to implement this.
What you need is a QueueBrowser to browse the messages of queue2 without consuming them.
Alternatively you could also consume from queue2 in a transaction and then force a rollback so that the message is not consumed. But when "latest message present on queue2" does not mean the first message, this will not work because you can only process the first message like this.
I have a queue system of which camel is just a small part of. In this queue system, for some queues, the broker returns FAIL when the queue is full. To handle this, I look at JMS Exception I get and from the message I can see whether or not the reason is that the queue is full.
What I want to achieve in Camel is that for the specific case of full queue, I want to retry deliveries, whereas for any other JMS Exception (or any other exception) I want to send it to DLQ.
I'm assuming that I have to use the onException(JMSException.class) and provide a custom processor which will look at the exception message, but after that, I'm not sure what to do. I tried raising a specific custom exception from the processor in that case (e.g. QueueFailedException) and have another onException(QueueFailedException) in the setup, but then I get the following error:
"(...) ERROR org.apache.camel.processor.FatalFallbackErrorHandler - Exception occurred while trying to handle previously thrown exception on exchangeId: (...) using: [Channel[DelegateSync[org.eumetsat.gems.bridge.JmsMessageReplicator$JMSExceptionHandler#67a93d5f]]]. The previous and the new exception will be logged in the following."
As far as I understood, you need to tell Camel it must consider an exception handled and also it should retry sending the message according to some logic check. The .handled() also accepts a Predicate or an Expression besides Boolean.
Don't the following sections help you achieve that?
Marking exceptions as handled
Using fine grained retry using retrywhile predicate.
I'm considering you're able to identify if the exception was really caused by a queue flooding through a caught JMSException.
onException(NullPointerException.class)
.handled(true)
.to("google-pubsub:some_topic");
In here, my design is such that for any unforseen error,I want to put the problem message to gcp pubsub error topic.
But I am saying "handled" as true.Hence for any error in publishing to pubsub error topic (say network error etc),the error will be silently ignored !! This is no good for me.Now I lost the message because since it was handled,the message has been acknowledged automatically and the message will not be redelivered by gcp pubsub !!
Please let me know my alternatives
I was just going through the Camel in Action Book.Below is the text
"Camel doesn’t allow further error
handling while already handling an error(onException handled is true) . In other words, when
Camel detects that another
exception was thrown during error handling, it prevents any further action
from taking place. This is done by the
org.apache.camel.processor.FataFallbackErrorHandler, which catches the
new exception, logs a warning, sets this as the exception on the Exchange, and
stops any further routing."
This mean if an exception is thrown while handling an exception,the exception will not be propagated,but a warning will be logged and sets an exception on the exchange.
Since an error has been marked in the exchange ,the pub sub message in the exchange will not be marked as Acked,and will be redelivered by pubsub.
Let me test it my self.
Request-reply pattern
If the caller knew operation has failed, they can retry so you will not mark the exception as handled
One-way pattern and can afford to lose message
Just like java catch block that catches but doesn't re-throw. May be you log a message. Since you can afford to lose the message, it is fine.
One-way pattern, you prefer moving the message somewhere on best of efforts
Exactly like your example. You prefer moving somewhere but if the error hapens on the onException route, you are ok to lose the message
One-way pattern and you cannot afford to lose message
In case of recoverable error, you want to retry a few times but after that you want to move it somewhere else. In the case of irrecoverable error, you want to move it somewhere else straight away. In both cases, if you don't move it, you will endup with infinite loop and your route will be busy repeatedly consuming the same message while others are ignored.
Since you cannot afford to lose the message if the error happens on the onException route, you cannot mark it as handled and at the same time, you cannot let it go back and start a infinite loop
So your option here is Dead Letter Channel error handler
DeadLetterChannel
When the DeadLetterChannel moves a message to the dead letter endpoint, any new Exception thrown is by default handled by the dead letter channel as well. This ensures that the DeadLetterChannel will always succeed.
Reference
https://camel.apache.org/components/latest/eips/dead-letter-channel.html
Note:
As you can see in the image, DeadLetterChannel is another error handler
Move the exception handling logic to new route and use queue/direct component to handle it. In case of failure you can write your failed message to an error/dlq and come up with a strategy to reprocess them.
Main Route:
onException(Exception.class)
.handled(true)
.to("queue:myapp.exception.handler");
from(direct:mainRoute")
.process("processTransaction")
.to("sql:***");
Exception Handler Route:
onException(Exception.class)
.handled(true)
.maximumRedeliveries(2)
.redeliveryDelay(30000)
.to("queue:myapp.exception.handler.failed") ;
from("queue:myapp.exception.handler")
.to("google-pubsub:some_topic");
I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.