Route1:
1. Send Message M1 to MQ Q1. // This message goes to a program that consumes from Q1 and once some processing is done, write message M2 to Q2.<br/>
2. Upon receiving Message M2 on Q2, //Q2 receives several messages (M2,N2,P2, etc.). ONLY when M2 is received, the Route1 should continue.<br/>
3. Send message M3 to Q3. // The step should be executed only after step 2 is complete<br/>
Step 2 implies that the message coming on Q2 needs to be inspected and only if it turns out to be M2, the Route1 should resume. Is there something we can do to pause a Route while it waits for a message and resume it once the message arrives?
Can we raise an event so that any Route that is waiting for M2 can be resumed? We know for sure that among all the parallely executing routes, only one may wait for M2. What do we need to do to resume the route?
Thanks,
Yash
there are a couple of options for controlling routes at runtime
use a RoutePolicy to add route lifecycle callbacks
use content based routing to apply specific logic based on message attributes/content
use the aggregator to group messages together based on various rules, etc.
use the ControlBus component to control routes status
also see this page for how to stop a route from another route
Related
I have requirement to poll two ftp folders say "output" and "error" for file. Actual file could come in either of two folders(not both). I tried multicast (even used completionSize of 1) but process keeps waiting as camel waits for both output and error endpoint routes to complete.
from("direct:got_filename_to_poll")
.multicast()
.parallelProcessing(true)
.to("ftp:output", "ftp:error")
.end()
.to("direct:process_extracted_file")
Is there any way to interrupt "ftp:error" sub-route if I get response from "ftp:output" sub-route(or vice versa) or is there any other option to solve this problem without compromising response time, example adding timeout on slow response will slow down overall response time.
I'm having some issues trying to figure out the solution for this problem:
I need to implement a DLC on Apache Camel, though when message are dequeued from the dead letter queue I have on ActiveMQ, every single one of them has to be compared with the latest massage present on another AMQ queue.
So to be clear: when Camel is consuming from queue1 (dead letter queue) the message M1, before trying to resend it to a certain route, it has to compare M1 (for example header comparison) with the latest message present on queue2, M2. M2 is not to be removed from queue2 (it shall serve also for the next comparison) while M1 has to be removed from queue1.
I want to understand if this is possible and which EIP I'm missing in order to implement this.
What you need is a QueueBrowser to browse the messages of queue2 without consuming them.
Alternatively you could also consume from queue2 in a transaction and then force a rollback so that the message is not consumed. But when "latest message present on queue2" does not mean the first message, this will not work because you can only process the first message like this.
I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.
I want the onCompletion to occur after all the aggregated exchanges triggered by both completion size followed by timeout are processed. But it occurs right after the completion size is triggered with some of the exchanges waiting to be triggered by the timeout criteria.
I have the route configured as
from(fromEndPoint)
.onCompletion()
.doSomething()
.split() // each Line
.streaming()
.parallelProcessing()
.unmarshal().bindy
.aggregate()
.completionSize(100)
.completionTimeout(5000)
.to(toEndpoint)
Assume if the split was done on 405 lines, the first 4 sets of aggregated exchanges go to the to endpoint completing 400 lines(exchanges) . And then, it immediately triggers the onCompletion. But there are still 5 more aggregated exchanges which would be triggered when the completionTimeout criteria is met. It didn't trigger the onCompletion after the 5 exchanges are routed to the to endpoint.
My question here is , either the onCompletion should be triggered for each exchange or once after all.
Note:- My from endpoint here is a File.
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.