I'm using Apache Camel, and trying to create a log of the history of the processing of each message in a workflow.
For simple straight-through workflows, where a message comes in, is processed by a few steps, and then leaves, this could be as simple as just keeping a sequential log of the exchanges. I can do this by writing a custom TraceEventHandler, which is called at each exchange and allows me to do logging.
However, if a splitter is involved, I don't know how to calculate the provenance of any given exchange. I could maintain my own log of exchanges, but in the case of a splitter, not all previous activity would be an ancestor of the current exchange. That is, if an incoming message is split into part1 and part2, which are then each processed separately, I don't want to consider the processing of part1 when calculating the history of part2.
A TraceEventHandler has this method:
#Override
public void traceExchange(ProcessorDefinition<?> node, Processor target,
TraceInterceptor traceInterceptor,Exchange exchange) throws Exception {
}
and I expected that there would be an Exchange method like Exchange getPreviousExchange() that I could call inside traceExchange, but I can find no such thing.
Any advice? I'm not married to using a custom TraceEventHandler if there's a better way to do this.
Thanks.
You can find the previous Exchange id by looking up the exchange property with the key "CamelCorrelationId".
If you want to track the post-split processing as separate branches, then you need to consider the Camel property "CamelSplitIndex". This property will indicate which iteration of the split you're processing and when combined with the CamelCorrelationId as William suggested, will provide the full picture.
Related
I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.
I have two queues which having same type of objects in them. I want to aggregate them into a single queue through java DSL. Could anyone tell me if this is possible? If so, any code references?
If I understand your question correctly, it is possible to do such a thing.
If you need just to drive them into a single route (without any aggregations, enrichments, etc.) you can just proceed with this piece of code:
from('direct:queue1')
.to('direct:start');
from('direct:queue2')
.to('direct:start');
from('direct:start')
//there goes your processing
If you need to aggregate them later on, use Aggregator. Or you can use example from java-addict301's answer if it solves your case.
I believe this may be doable in Camel using the Content Enricher pattern.
Specifically, the following paradigm can be used to retrieve a message from one queue (where direct:start is) and enrich it with a message from the second queue (where direct:resource is). The combined message can then be built in your AggregationStrategy implementation class.
AggregationStrategy aggregationStrategy = ...
from("direct:start")
.enrich("direct:resource", aggregationStrategy)
.to("direct:result");
from("direct:resource")
My question is very similar to this one but the solution there does not work for me here - I am trying to use the filter EIP to discard selected exchanges. My routes look like (edited down for clarity):
from("{{fromSource}}")
.convertBodyTo(RequestInterface.class)
.enrich(INVOKE_BACKEND_URI, combiner)
.to("{{toDestination}}");
from(INVOKE_BACKEND_URI)
.to(backendUri)
.filter().method(DiscardResponse.class).log(LoggingLevel.INFO, "Discarding undesired response").stop().end()
.convertBodyTo(BodyInterface.class);
When the filter does NOT select the message, all is well - the log() is not displayed and the message goes to the convertBodyTo() and then back to the main route.
However, when the filter DOES select the message, the log() text is displayed but the exchange still continues on to the convertBodyTo() where it throws an exception because it's a message that shouldn't be there. The stop() appears to either not be executed or has no affect.
Can anyone suggest a solution to this?
It is possible from within a Processor to do this in order to stop the exchange:
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
Since I'm not used to writing my routes using Java DSL I don't know if that option is available directly on the exchange within the route, but it probably is.
I guess one way could be:
from(INVOKE_BACKEND_URI)
.to(backendUri)
.filter().method(DiscardResponse.class).log(LoggingLevel.INFO, "Discarding undesired response")
.choice()
.when(simple("${property.Exchange.FILTER_MATCHED}=true")
.stop()
.end()
.convertBodyTo(BodyInterface.class);
Take a look at the bottom of the doc here:
http://camel.apache.org/message-filter.html
Edit: To be specific I think I am asking for an elegant way to specify 2 endpoints, which I want to send 2 different exchanges from an input of 1 exchange.
I know Camel 'can' do this - but I only have inelegant methods involving sending an object which contains both types to a multicast() and processors on each section removing
I expect there to be potentially multiple error messages with source objects attached. I could just throw them each as exceptions, but this feels incorrect. I'm wondering what the 'correct' approach might be. I almost just want to be able to specify an error endpoint as a target for my component
Currently I have
camel.addComponent( "my", new MyComponent() )
from( some source )
... processing // Lists of input objects as body of in
... onException()
.to( "my:endpoint" )
MyComponent <-- MyEndpoint <-- MyProducer
I want to process the items in each List object that arrives at MyProducer. I process the elements and send failing items out to one endpoint, and good items out to an endpoint
I do not see a good / elegant way of achieving this. If it was single elements (i.e. not collection) I can just throw an exception and catch it in an onException stream.
But I really want to be able to take items, and separate good items and send them one way, and bad items and send them another.
In other words, I want to simultaneously send 2 different messages to 2 different endpoints from the same input from an Endpoint. (The endpoint isn't actually so important here, it is juts I am writing one, it could be any Processor).
I know I could make a decomposable object with good and bad items on it, then multicast and process each good and bad section out on different pieces, but I really would like a succinct reusable mechanism (e.g. built into a Processor or endpoint)
In Camel the stuff between the from() and the to() are an Exchange which is treated as a single message. The component should just be a factory for Endpoint instances, which either creates or sends exchanges. It shouldn't be processing the message (which is what it sounds like here) so there's not really a concept of of errors or good objects, that's for the Route and the Processors/Beans to decide.
If you want to do it all within a single exchange then you can simply have your processor add 2 lists to exchange properties and route them based on the contents.
from("direct:somesource")
.process(new Processor() {
public void process(Exchange exchange) {
// get contents,
// turn it into a list
//
List<Object> worked = new ArrayList<>();
List<Object> failed = new ArrayList<>();
for (Object o : listFromBody) {
try {
// do work
worked.add(o);
} catch (Exception e) {
failed.add(o)
}
}
exchange.getIn().setProperty("worked", worked);
exchange.getIn().setProperty("failed", failed);
}
};)
.choice()
.when(header("failed").isNotEqualTo(null)) // might be a better way to do this, I've not got the IDE open
.process(turnFailedHeaderIntoMessageBody)
.to("direct:errorQueue")
.otherwise()
.process(turnWorkedHeaderIntoMessageBody)
.to("direct:workedQueue");
However this is not a good camel pattern. It's messy and tries to treat the properties as different messages which is contrary to how camel works. From the route's perspective the exchange is an atomic unit, so if you need to break the message up, it's usual to route the contents of the Exchange to be processed as an Exchange by a different route.
I personally would split the list into separate exchanges and process them individually like this:
from("direct:somesource")
.split(body())
.to("direct:processIndividualMessage");
from("direct:direct:processIndividualMessage")
.doTry()
.process(myProcessor)
.to("direct:goodQueue")
.doCatch(Exception.class)
.to("direct:errorQueue")
.end()
It's all depends on your data model. But Camel has no limitations in this regards, you certainly achieve this. I can clarify if you have any specific question.
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.