How does these two differ
from(endpoint).to(endpoint:a, endpoint:b)
from(endpoint).multicast().to(endpoint:a, endpoint:b)
couldn't find any documentation for the first
to(endpoint:a, endpoint:b) is equivalent to .to(endpoint:a).to(endpoint:b) This means that the output from endpoint:a is sent to endpoint:b, not the original Exchange. Also, each endpoint is executed one after the other.
.multicast() sends the original Exchange to each defined endpoint, allows for parallel processing, and allows you to define an AggregationStrategy to determine how to assemble the responses from each endpoint the original Exchange was sent to.
Yeah as jarrad writes the difference between the two are
The first is the pipes and filters EIP (default mode in Camel). Which is documented here: https://www.enterpriseintegrationpatterns.com/patterns/messaging/PipesAndFilters.html
The 2nd is the multicast EIP which is documented here:
http://camel.apache.org/multicast.html
All the Camel EIPs is listed here: http://camel.apache.org/eip
Related
I have a EIP design related query.I have a requirement to process csv file by chunks and call a Rest API.After completion of processing of whole file i need to call another Rest API telling processing is complete.I wanted the route to be transacted so i have queue in between in case of end system not available the retry will happen at broker level.
My flow is as below.
First flow:
csv File->Split by chunk of 100 records->Place message in queue
the second flow(Transacted route):
Picks message from queue ->call the rest API
the second flow is transacted.Since iam breaking the flow and it is asynchronous iam not sure how to call to the completion call.I do not have a persistent store to status of each chunk processing.
is there anyway i can achive it using JMS functionality or Camel?
What you can use for your first flow is the Camel Splitter EIP:
http://camel.apache.org/splitter.html
And closely looking at the doc, you will find that there are three exchange properties available for each split exchange:
CamelSplitIndex: A split counter that increases for each Exchange being split. The counter starts from 0.
CamelSplitSize: The total number of Exchanges that was splitted. This header is not applied for stream based splitting. From Camel 2.9 onwards this header is also set in stream based splitting, but only on the completed Exchange.
CamelSplitComplete: Whether or not this Exchange is the last.
As they are exchange properties, you should put them to JMS headers before sending the messages to a queue. But then you should be able to make use of the information at the second flow, so you can know which is the last message.
Keep in mind, though, that it's all asynchronous so the CamelSplitComplete flag doesn't necessarily mean the last message at the second flow. You may create a stateful counter or utilise the Resequencer EIP http://camel.apache.org/resequencer.html to deal with the asynchronicity.
I am getting as strange behavior from Camel Multicast. Looks like I am doing something wrong but can't point it.
I am creating a multicast route to two endpoints as below:
Start point : direct://start
endpoint : direct://route1
endpoint : direct://route2
Routes info :
1) from("direct://start").process(new DeepCloneRequest()).multicast(new Aggregator()).to("direct://route1","direct://route2")
2) from("direct://route1").setheader(<to use later in processor>).process(<do some preprocessing>).to("rest url1").processor(<to do post processing>).end
3) from("direct://route2").setheader(<to use later in processor>).process(<do some preprocessing>).to("rest url2").processor(<to do post processing>).end
What I notice is that in my route number 3 in preprocessing I got the exchange out of number 2 preprocessing instead of DeepCloneRequest from multicast.
Can someone please help explain this? I want to get same request that multicast receive in both route 2 and 3.
The Camel version I am using is 2.17.3.
You should use the onPrepare feature, see http://camel.apache.org/multicast.html
"The Multicast will copy the source Exchange and multicast each copy. However the copy is a shallow copy, so in case you have mutateable message bodies, then any changes will be visible by the other copied messages. If you want to use a deep clone copy then you need to use a custom onPrepare which allows you to do this using the Processor interface."
I have a scenario where I get as input Message A. Message A must then be split into 3 different types of message, and forwarded to other routes. It is important that the messages arrive in a precise order, Ie. A-1 must be sent before A-2, which must be sent before A-3.
To do this I have done the following (outline):
from("activemq:queue:somequeue-local")
.multicast().to("direct:a1","direct:a2","direct:a3");
from("direct:a1)
//split incoming message and prepare output document for A-1
.to("activemq:queue:otherqueue")
.from("direct:a2)
//split incoming message and prepare output document for A-2
.to("activemq:queue:otherqueue")
.from("direct:a3)
//split incoming message and prepare output document for A-3
.to("activemq:queue:otherqueue")
And in another context, responsible for sending out the info to the external system, I have
.from("activemq:queue:otherqueue?maxMessagesPerTask=1&concurrentConsumers=1&maxConcurrentConsumers=1")
// do different stuff based on which type we are called with then end with
.beanref("somebean","writeToFileAndCallImportbat");
Now, my problem is, that when I get to the receiver, I get the messages in random order. Sometimes A-1,A-3,A-2, sometimes right, A-1,A-2,A-3.
I have tried adding JMSXGroupID and JMSXGroupSeq to the messages, but without any luck.
I have also tried skipping the MQ part entirely, and use direct-vm: to call the shared receiver, but then it looks like I have three simultanious invocations of the receiver at once, and still in random execution order.
I was under the impression that multicast would run sequential, unless otherwise prompted to?
Is there something fundamentally wrong with the approach taken?
I am using Camel version 2.12.
Or, said more plainly:
I would like a route that creates three different output messages, and executes a batch file on them, in order. How do I go about that?
If you use the Splitter pattern, have you checked to see if the streaming property is set to false.
If enabled then Camel will split in a streaming fashion, which means it will split the input message in chunks. This reduces the memory overhead. For example if you split big messages its recommended to enable streaming. If streaming is enabled then the sub-message replies will be aggregated out-of-order, eg in the order they come back. If disabled, Camel will process sub-message replies in the same order as they where splitted.
So, it turned out to not be a problem with multicast after all.
Rather, in each of my sub-routes, I did this:
.split(..stax(SpecialClass)).streaming()
.beanRef("transformationBean","somefunction")
.aggregate(constant("1"), new MyAggregator())
.completionTimeout(5000)
.completionSize(1000)
.to(writeToFileAndRunBat)
Which, I assumed meant "Process all elements in the split, and if you aren't finished in 5 seconds or after 1000 elements, break out".
I changed it to
.split(..stax(SpecialClass), , new MyAggregator()).streaming()
.beanRef("transformationBean","somefunction")
.end()
.to(writeToFileAndRunBat)
Coming to think of it, it makes perfect sense, as the first version couldn't really know when we were done, while the last (I assume) just iterate over all elements in the split and calls the Aggregator for each.
Also, I had to .end() in the first version. So I guess the whole thing was just acting random.
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.
In apache camel, which of those is the best way to pass values from an exchange processor to another (and why) :
storing it in the exchange headers
using the setProperty method while building the route.
another way..
One distinction not mentioned by Ben and Petter is that properties are safely stored for the entire duration of the processing of the message in Camel. In contrast, headers are part of the message protocol, and may not be propagated during routing. For example, JMS has limitations what you can store as headers etc.
You may want to read the free chapter 1 of the Camel in Action book as it covers the Camel concepts with Exchange, Message, etc.
Properties and headers are pretty much the same. Headers are, however, converted to/from protocol specific headers on certain components, such as Jms. So,
Meta data inside a route: properties
Meta data to/from outside: headers
the Exchange is passed between Processors. It contains properties, IN message and optional OUT message. Each of these is capable of storing Object data, but in general:
use the Exchange Properties for general meta-data about the message (used less frequently)
use the IN message headers to configure endpoint properties or for meta-data about the message body (used often)
use the IN message body for the payload of the message (used most often)
create an OUT message only if necessary to maintain separate IN vs. OUT messages during processing (by default only IN is used)
That said, it really depends on the component called following your processor. Most have some headers and/or body values that are required to use the endpoint, etc. See the specific component page for these details.
Also, the Exchange/Message are explained in more detail on these pages:
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html
http://fusesource.com/docs/router/2.8/prog_guide/MsgFormats-Exchanges.html
Answer is here:
Properties: The properties is a Map and may look like message
headers. The main difference is their lifetime: the properties exist during the whole
exchange execution, whereas the headers are limited to the message duration (and a
message can change a lot during routing, so during the exchange execution). Camel
itself may add some properties for some use cases.