I've configured two routes to consume messages from 2 ActiveMQ queues (queue1 and queue2) to move the messages to another destination (say to aws-sqs queue - sqsqueue1) and this setup is working as expected (queue1 --> sqsqueue1, queue2 --> sqsqueue1). Now my requirement is to modify this consumption such that the queue1 route must be consumed first and when there are no messages on queue1 then only the queue2 route should start consumption. I've explored Control Bus, RoutePolicy, and not sure whether that would fit my case. Moreover is it a valid use case in the EIP pattern. Please advise or any pointers would be helpful.
// pseudocode
from("activemq-queue:queue2")
.to("controlbus:route?routeId=queue1&action=suspend")
.onCompletion("controlbus:route?routeId=queue1&action=resume").end()
.to("aws-sqs:sqsqueue1")
Related
I'm having some issues trying to figure out the solution for this problem:
I need to implement a DLC on Apache Camel, though when message are dequeued from the dead letter queue I have on ActiveMQ, every single one of them has to be compared with the latest massage present on another AMQ queue.
So to be clear: when Camel is consuming from queue1 (dead letter queue) the message M1, before trying to resend it to a certain route, it has to compare M1 (for example header comparison) with the latest message present on queue2, M2. M2 is not to be removed from queue2 (it shall serve also for the next comparison) while M1 has to be removed from queue1.
I want to understand if this is possible and which EIP I'm missing in order to implement this.
What you need is a QueueBrowser to browse the messages of queue2 without consuming them.
Alternatively you could also consume from queue2 in a transaction and then force a rollback so that the message is not consumed. But when "latest message present on queue2" does not mean the first message, this will not work because you can only process the first message like this.
I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.
I have a EIP design related query.I have a requirement to process csv file by chunks and call a Rest API.After completion of processing of whole file i need to call another Rest API telling processing is complete.I wanted the route to be transacted so i have queue in between in case of end system not available the retry will happen at broker level.
My flow is as below.
First flow:
csv File->Split by chunk of 100 records->Place message in queue
the second flow(Transacted route):
Picks message from queue ->call the rest API
the second flow is transacted.Since iam breaking the flow and it is asynchronous iam not sure how to call to the completion call.I do not have a persistent store to status of each chunk processing.
is there anyway i can achive it using JMS functionality or Camel?
What you can use for your first flow is the Camel Splitter EIP:
http://camel.apache.org/splitter.html
And closely looking at the doc, you will find that there are three exchange properties available for each split exchange:
CamelSplitIndex: A split counter that increases for each Exchange being split. The counter starts from 0.
CamelSplitSize: The total number of Exchanges that was splitted. This header is not applied for stream based splitting. From Camel 2.9 onwards this header is also set in stream based splitting, but only on the completed Exchange.
CamelSplitComplete: Whether or not this Exchange is the last.
As they are exchange properties, you should put them to JMS headers before sending the messages to a queue. But then you should be able to make use of the information at the second flow, so you can know which is the last message.
Keep in mind, though, that it's all asynchronous so the CamelSplitComplete flag doesn't necessarily mean the last message at the second flow. You may create a stateful counter or utilise the Resequencer EIP http://camel.apache.org/resequencer.html to deal with the asynchronicity.
I have the following issue (simplified):
Two queue managers - QM1 and QM2 - form a cluster.
QMgr QM1
alias queue Q1 with base queue Q1.L, which is a local cluster queue (i.e. defined on QM1)
alias queue Q2 with base queue Q2.L, which is a non-local cluster queue (i.e. defined on QM2)
QMgr QM2
local cluster queue Q2.L
I can open the alias queues for inquiry, request MQCA_BASE_Q_NAME and I get the base queues in both cases.
I need to programatically find out, if this base queue is a local cluster queue or a remote (non-local) cluster queue. We are using C API (MQI).
I open the base queue for inquiry and, based on this documentation:
http://www-01.ibm.com/support/knowledgecenter/#!/SSFKSJ_7.5.0/com.ibm.mq.ref.dev.doc/q101840_.htm
(see Usage Notes - 4.)
I can request only the following attributes:
- MQCA_Q_DESC, MQCA_Q_NAME, MQIA_DEF_BIND, MQIA_DEF_PERSISTENCE, MQIA_DEF_PRIORITY, MQIA_INHIBIT_PUT, MQIA_Q_TYPE
This works, but MQIA_Q_TYPE returns for a cluster queue MQQT_CLUSTER (7). This is good - I know that I handle a cluster queue, but not enough - local or non-local?
Checking the cmqc.h header, I can see some other interesting attribute selectors, unfortunatelly not working.
For example: MQIA_CLUSTER_Q_TYPE, but when passing in the selector vector for inquiry, getting back
CompCode:2, Reason:2067 - Attribute selector not valid.
In the PCF documentation this seems to be possible:
http://www-01.ibm.com/support/knowledgecenter/#!/SSFKSJ_7.5.0/com.ibm.mq.ref.adm.doc/q087800_.htm
(Table 1, column Cluster queue)
Is this some limitation of C API? Any workaround?
Cheers, Miro
I know what you want to do, but why you would want to do it is an interesting question. I hope that what you are working on is instrumentation and monitoring rather than an application program. If a business application needs to know this information, then the design is almost certainly broken. The whole idea of async messaging was to decouple the sender from the receiver of the message, and thus the need for the app to know or care whether the destination is local or not. This is why the API doesn't address your question - to do so for business apps breaks the async model.
That said, the simplest way is to use MQIA_CURRENT_Q_DEPTH and inquire on the queue depth. If the queue is non-local, the call will fail.
(Deleted the previous answer about using PCF to DIS QL since this is much simpler and 100% accurate.)
Of the 60 queue attributes available for inquiry, why do you believe that you "can request only the following attributes: - MQCA_Q_DESC, MQCA_Q_NAME, MQIA_DEF_BIND, MQIA_DEF_PERSISTENCE, MQIA_DEF_PRIORITY, MQIA_INHIBIT_PUT, MQIA_Q_TYPE"? Is this a local shop standard?
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.