I'm trying to implement a mina service where the response to the final message should be based on the previous messages. Each message (header (1), data (n), end (1)) should receive a response, but the response to the "end" message should be based on the "header", and any "data" messages received as well as the "end" message. Currently, I'm routing the messages to an aggregator which completes when it finds a "header" and "end" message for a particular correlation id. Unfortunately, the response is being sent before (or at the same time?) the message is sent to the aggregator, so I don't have access to the aggregated message (which contains all the data I need to build the correct response) when building the response.
Is there a way to do this without manually storing and accessing the accumulated data (that is, without re-implementing camel's aggregator)?
Edit:
Route is something like:
<camelContext>
<route>
<from uri="mina:..."/>
<process ref="messageProcessor"/>
<aggregate>
<process ref="completeMessageProcessor"/>
</aggregate>
</route>
</camelContext>
I left out some tags and attributes (correlationExpression, completionPredicate, strategyRef, etc.) for clarity.
The messages were being aggregated properly, and they were being processed properly when "completed" (that is, when aggregated). But the response sent back through the mina endpoint to the client was the one generated by the messageProcessor, never the one generated by the completeMessageProcessor.
For example (and yes, it's a rather contrived example, but bear with me), let's say the protocol involves the client sending a header message which includes the total number of data messages it expects to send. Then it sends a number of data messages, which might be different in number to what it expected to send. Finally, it sends a footer, or end, message. The server should then respond back with the difference between the expected number of messages and the actual number of messages. With the route as written, that is impossible, since the number of messages is not known by the messageProcessor, which only processes individual messages. The completeMessageProcessor, having the aggregated message (consisting of header, all the data, and the end) does know this number, but the response generated at that point is not propagated back to the mina endpoint.
Changing the parsing of the messages to generate a message only when entire composed message is received is not an option, since the server must respond to the individual messages.
off the top, my guess is that the messageProcessor is setting up the OUT message, but the completeMessageProcessor is setting up the IN message. The mina consumer response is expecting/using the OUT message instead.
you can add some logging to verify this. if this is the case, then you might change your messageProcessor to use the IN body instead (or use exchanges headers) and add a transform after your completeMessageProcessor to set the OUT body based on the IN body
<transform>
<simple>${in.body}</simple>
</transform>
see this for more information: http://camel.apache.org/using-getin-or-getout-methods-on-exchange.html
UPDATE: after some discussion, the real issue is that the aggregator currently only handles "InOnly" exchanges
Related
In producer using librdkafka, is it possible to know the number of items produced but not sent yet. I have turned ack off, only interested in sending the message out from the producer. I wish to avoid producing more messages if there are certain number of messages on the send queue(ie the messages produced but not yet sent).
If you set statistics.interval.ms in the producer configuration file, it will start sending the internal metrics in the callback you register.
The callback you can set using the function rd_kafka_conf_set_stats_cb. The statistics are received in the callback in the form of json data.
One the fields from the json data, msg_cnt, tell you the queued messages that you are looking for.
More details are available at https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md
In ActiveMQ (and Hawtio) there is Admin UI functionality to move a messages from one queue to another. This process keeps the body and all the headers all the same as the original message.
Is there a way to do this in Camel? A simple from(queue1).to(queue2) always seems to change headers.
thanks
edit: tried
from(
"cMQConnectionFactory1:queue:queue1"
+ "?forceSendOriginalMessage=" + true
+ "&mapJmsMessage=" + false)
.routeId("testMove_cJMS_1")
.to("cMQConnectionFactory1:queue:queue2"
+ "?forceSendOriginalMessage=" + true
+ "&mapJmsMessage=" + false).id("testMove_cJMS_2");
}`
The Camel documentation for the JMS component describes the forceSendOriginalMessage option saying:
When using mapJmsMessage=false Camel will create a new JMS message to send to a new JMS destination if you touch the headers (get or set) during the route. Set this option to true to force Camel to send the original JMS message that was received.
forceSendOriginalMessage defaults to false.
The docs say this about mapJmsMessage:
Specifies whether Camel should auto map the received JMS message to a suited payload type, such as javax.jms.TextMessage to a String etc.
mapJmsMessage defaults to true.
Therefore it seems like you need to set mapJmsMessage=false & forceSendOriginalMessage=true.
Keep in mind that Camel will be using the JMS API to consume the message and then resend it. Even though the new message will have the same body and headers as the old message it will be slightly different because the JMS specification dictates that when a message is sent the broker must assign it a message ID and timestamp. Therefore the JMSMessageID and JMSTimestamp on the new message will be different from the old message, and there's really no way around that. If you need to identify the message uniquely you should set the correlation ID on the original message and use that to identify the message rather than the JMSMessageID. Also, if you need to preserve the original time the message was sent then set that in a custom property.
The reason the JMSMessageID and JMSTimestamp are not changed when moving the messages via the management console is because the broker moves the messages internally with a completely different mechanism than JMS.
I have two Camel routes, configured in XML and pasted below: -
Route 1:
<camel:route id="statementsArchivingPollRoute">
<camel:from uri="timer://tempQueue?fixedRate=true&period=30s&delay=30s"/>
<camel:transacted ref="PROPAGATION_REQUIRED">
<camel:process ref="statementsArchivingRequestZipProcessor"/>
<camel:choice>
<camel:when>
<camel:simple>${body.size} >= 1</camel:simple>
<camel:split>
<camel:simple>${body}</camel:simple>
<camel:marshal ref="archiveFileInterfaceMetadataMapper"/>
<camel:to pattern="InOnly"
uri="activemq:{{ccs.activemq.queue.prefix}}.sr.archive.bulk.ingestion.req?jmsMessageType=Text"/>
</camel:split>
<camel:log loggingLevel="INFO" message="Archiving content was processed"/>
</camel:when>
<camel:otherwise>
<camel:log loggingLevel="INFO" message="No archiving content to process"/>
</camel:otherwise>
</camel:choice>
</camel:transacted>
</camel:route>
Route 2:
<camel:route id="statementsArchivingBulkIngestionRequestRoute">
<camel:from uri="activemq:{{ccs.activemq.queue.prefix}}.sr.archive.bulk.ingestion.req"/>
<camel:throttle timePeriodMillis="4000">
<camel:constant>1</camel:constant>
<camel:setExchangePattern pattern="InOnly"/>
<camel:unmarshal ref="archiveFileInterfaceMetadataMapper"/>
<camel:bean ref="archiveFileEntryTransformer" method="transform"/>
<camel:setHeader headerName="CamelHttpMethod">
<camel:constant>POST</camel:constant>
</camel:setHeader>
<camel:toD uri="{{ccs.bulk.ingestion.service.ingest.archive.file}}"/>
</camel:throttle>
</camel:route>
The processor in the first route returns a list of request objects. The list is then split and each request is marshalled and placed on a queue.
The second route listens to this queue. When it de-queues a message, it unmarshals it, performs a transform and then uses it to send a post request to another service. I am throttling this route so that it only processes one message per second so as not to overwhelm the downstream service.
This all works fine when the list only contains a few requests and hence only a few messages enter the queue, but when there are many items in the list, route 2 times out and the log entry below appears:
Atomikos:12] c.a.icatch.imp.ActiveStateHandler : Timeout/setRollbackOnly of ACTIVE coordinator !
The timeout leads to the process repeating itself and the downstream service ends up getting called multiple times per message instead of just once.
I can't understand why the number of times route 2 is invoked should cause it to timeout. I thought that a single instance of the route would be launched for every message de-queued from the activemq. If a single message takes a long time to complete, then I would understand, but clearly the timeout is based on the cumulative times of all messages being de-queued.
I am fairly new to Camel and I have clearly misunderstood something from an architectural point of view. I would be extremely grateful for any guidance into how to stop these timeouts occurring. Thank you for reading.
Jon, you may want to try following to investigate
disable/comment the 2nd route. Purpose of using activemq would be to make process async which means 1st route should not have any impact due to 2nd route. If this works without route 2 then problem is elsewhere.
If you find 1st route is working fine without 2nd route then next would be to try setting fewer number of threads in 2nd route. May be put 1 or 2 threads and see if that helps. I am thinking it is contention on activemq rather than these route configurations.
Check the payload size that you are pushing to activemq. If you are publishing very large message to activemq that may also have an impact as large number of items and each item is large causing contention in activemq and transaction is taking longer than the timeout setting.
If you are pushing large data set to activemq, you may want to revisit the design. Save payload in some persistence area (db/file/cache) and sent notification events containing only reference to the payload and some metadata. 2nd route can then take the reference from event and retrieve the payload from where it was saved by route 1.
I have a really simple route that GETs an URL and prints the content using Camel HTTP4 component:
from("timer://foo?fixedRate=true&delay=0&period=10000")
.to("http4://www.google.com")
.process(e -> System.out.println("Out body: " + e.getOut().getBody()));
Note that I'm using out.body because, as stated in Camel documentation:
Camel will store the HTTP response from the external server on the OUT
body. All headers from the IN message will be copied to the OUT
message, so headers are preserved during routing.
But I get null values from OUT (both body and headers). Everything is being filled only in the IN message.
Am I missing anything or is it a bug?
In Camel a route consists of nodes. Each node takes the Exchange. Exchange has an IN and OUT message. So in your case, node with http4 component took the Exchange, called google.com and wrote body and headers to OUT message. Next, node with your proccesor took the Exchange. Now IN message has the response from the previous node(http4), but you are printing OUT which is empty! So IN and OUT message are per node not per route!
You are getting the Out body from the processor without setting it up first. That’s why you get null. To make this work you first need explicitly copy the incoming message, headers and attachments to Out Body and then print it. Or more easily take the In message as you mentioned.
Next part is from “Camel in Action” book which is a great book and I think it is very helpful.
in practice there’s a common pitfall when using getOut: the incoming
message headers and attachments will be lost. This is often not what
you want, so you must copy the headers and attachments from the
incoming message to the outgoing message, which can be tedious.
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.