I am trying to understand the unit of work concept in Camel. I have a simple question hopefully someone here can help.
If there are multiple routes involved in routing the Excachange for example
from("aws-sqs:Q1").to("direct:processMe");//route1
from("direct:processMe").to("direct:aws-post");//route2
from("direct:aws-post").to("htt4:myservice");//route3
Is the unit of work invoked at the end of each routes? Or only at the end of route3? In my example, will the SQS message be deleted off of SQS once route1 completes? Or will it wait until my message reaches "myservice"?
Thanks.
FOLLOW UP:
I've modified the route slighty:
from("aws-sqs:Q1").to("direct:processMe");//route1
from("direct:processMe").process(
new Processor(){
public void process(Exchange exchange) throws Exception {
throw new RuntimeException("fail on purpose");
}
}
).to("direct:aws-post");//route2
from("direct:aws-post").to("http4:myservice");//route3
The thought process is this:
If the unit of work is invoked at end of each route then once the message is read from SQS Queue it will be acknowledged as read by the SQS component. On the other hand if unit of work is only invoked once the Exchange is done routing through all routes, then the exception in route 2 will result in the message not being acknowledged and will be available for redelivery once the visibility period expires.
The test showed that the message remains on the Q despite being read by first route. It is picked up again and again (until it ends up in dead letter q). As such I strongly believe that the unit boundary is defined by the end of Exchange being routed.
A unit of work is basically a transaction.
By default a message for a route (a camel Exchange) runs inside a single UnitOfWork context.
In your example there are 3 UnitOfWorks set up, starting at each from and finishing at the final to in each route.
I would expect the message from SQS to be consumed after the 1st route finishes. To test, you could add in a sleep to allow you to check the queue.
from("direct:processMe").process(new Processor()
{ void process() { try { Thread.sleep(60000L) } catch (Exception e) { } }
}).to("direct:aws-post")
If you want the message to remain on the queue until myservice gets the message then you need to put the processing in a single route.
I have found a great explanation as to the unit of work boundary:
"The OnCompletion DSL name is used to define an action that is to take place when a Unit of Work is completed.
A Unit of Work is a Camel concept that encompasses an entire exchange. See Section 43.1, “Exchanges”. The onCompletion command has the following features:
The scope of the OnCompletion command can be global or per route. A route scope overrides global scope.
OnCompletion can be configured to be triggered on success for failure.
The onWhen predicate can be used to only trigger the onCompletion in certain situations.
You can define whether or not to use a thread pool, though the default is no thread pool."
In case of SQS processing, the consumer defines onCompletion on the exchange. So it is invoked only after the exchange is done routing.
The whole answer can be found here: Apache Camel Development Guide2.11. OnCompletion
Related
I have this simple route in my RouteBuilder.
from("amq:MyQueue").routeId(routeId).log(LoggingLevel.DEBUG, "Log: ${in.headers} - ${in.body}")
As stated in the doc for HTTP-component:
Camel will store the HTTP response from the external server on the OUT body. All headers from the IN message will be copied to the OUT message, ...
I would like to know if this concept also applies to amq-component, routeId, and log? Is it the default behaviour, that IN always gets copied to OUT?
Thank you,
Hadi
First of all: The concept of IN and OUT messages is deprecated in Camel 3.x.
This is mentioned in the Camel 3 migration guide and also annotated on the getOut method of the Camel Exchange.
However, it is not (yet) removed, but what you can take from it: don't care about the OUT message. Use the getMessage method and don't use getIn and getOut anymore.
To answer your question:
Yes, most components behave like this
Every step in the route takes the (IN) message and processes it
The body is typically overwritten with the new processing result
The headers typically stay, new headers can be added
So while the Camel Exchange traverses the route, typically the body is continuously updated and the header list grows.
However, some components like aggregator create new messages based on an AggregationStrategy. In such cases nothing is copied automatically and you have to implement the strategy to your needs.
Edit: To be specific I think I am asking for an elegant way to specify 2 endpoints, which I want to send 2 different exchanges from an input of 1 exchange.
I know Camel 'can' do this - but I only have inelegant methods involving sending an object which contains both types to a multicast() and processors on each section removing
I expect there to be potentially multiple error messages with source objects attached. I could just throw them each as exceptions, but this feels incorrect. I'm wondering what the 'correct' approach might be. I almost just want to be able to specify an error endpoint as a target for my component
Currently I have
camel.addComponent( "my", new MyComponent() )
from( some source )
... processing // Lists of input objects as body of in
... onException()
.to( "my:endpoint" )
MyComponent <-- MyEndpoint <-- MyProducer
I want to process the items in each List object that arrives at MyProducer. I process the elements and send failing items out to one endpoint, and good items out to an endpoint
I do not see a good / elegant way of achieving this. If it was single elements (i.e. not collection) I can just throw an exception and catch it in an onException stream.
But I really want to be able to take items, and separate good items and send them one way, and bad items and send them another.
In other words, I want to simultaneously send 2 different messages to 2 different endpoints from the same input from an Endpoint. (The endpoint isn't actually so important here, it is juts I am writing one, it could be any Processor).
I know I could make a decomposable object with good and bad items on it, then multicast and process each good and bad section out on different pieces, but I really would like a succinct reusable mechanism (e.g. built into a Processor or endpoint)
In Camel the stuff between the from() and the to() are an Exchange which is treated as a single message. The component should just be a factory for Endpoint instances, which either creates or sends exchanges. It shouldn't be processing the message (which is what it sounds like here) so there's not really a concept of of errors or good objects, that's for the Route and the Processors/Beans to decide.
If you want to do it all within a single exchange then you can simply have your processor add 2 lists to exchange properties and route them based on the contents.
from("direct:somesource")
.process(new Processor() {
public void process(Exchange exchange) {
// get contents,
// turn it into a list
//
List<Object> worked = new ArrayList<>();
List<Object> failed = new ArrayList<>();
for (Object o : listFromBody) {
try {
// do work
worked.add(o);
} catch (Exception e) {
failed.add(o)
}
}
exchange.getIn().setProperty("worked", worked);
exchange.getIn().setProperty("failed", failed);
}
};)
.choice()
.when(header("failed").isNotEqualTo(null)) // might be a better way to do this, I've not got the IDE open
.process(turnFailedHeaderIntoMessageBody)
.to("direct:errorQueue")
.otherwise()
.process(turnWorkedHeaderIntoMessageBody)
.to("direct:workedQueue");
However this is not a good camel pattern. It's messy and tries to treat the properties as different messages which is contrary to how camel works. From the route's perspective the exchange is an atomic unit, so if you need to break the message up, it's usual to route the contents of the Exchange to be processed as an Exchange by a different route.
I personally would split the list into separate exchanges and process them individually like this:
from("direct:somesource")
.split(body())
.to("direct:processIndividualMessage");
from("direct:direct:processIndividualMessage")
.doTry()
.process(myProcessor)
.to("direct:goodQueue")
.doCatch(Exception.class)
.to("direct:errorQueue")
.end()
It's all depends on your data model. But Camel has no limitations in this regards, you certainly achieve this. I can clarify if you have any specific question.
In Camel in Action book, it is written that default exception handling is for the life span of exchange object only. Suppose I am using following route -
from(sftp:{details}).to(file:{details})
1st query -> For the above route, I think the consumer is sftp and the producer is file. Am I right here?
2nd query -> How can we handle the exception at destination end point (which is outside the life span of exchange object).
1st You are correct.
2nd Any exception that will be raised before the file is successfully written to disk will trigger Camel's exception handling which will by default in file based components trigger rollback of the original file so that data won't be lost.
I'm trying to run an onCompletion() block on my route, which contains an aggregate definition with completionTimeout. It seems like onCompletion is called before the route is actually completed, since I get log entries from OnCompletion before AggregateTimeoutChecker log entries.
How can I make onComplete wait for aggregation timeout?
Of course I can add a delay greater than completionTimeout to onCompletion, but that will slow down my tests a lot.
My route looks like this:
from(fileEndpoint)
.bean(externalLogger, "start")
.onCompletion()
.bean(externalLogger, "end") // <-- Gets called too early
.end()
.split().tokenize("\n")
.bean(MyBean.class)
.aggregate(header("CamelFileName"), ...)
.completionSize(size)
.completionTimeout(500)
.bean(AggregatesProcessor.class); // <-- some changes here don't arrive
// at onCompletion
onCompletion() is triggered for each incoming exchange when it has completed the route. When using an aggregator, all exchanges not completing the aggregation finish the route at the aggregator, thus your externalLogger gets called for each file being aggregated.
If you want logging after the aggregation you could just call the logger after aggregate().
If you need to distinguish between timeout and completion of your aggregation it could be helpful to provide a custom AggregationStrategy and to also implement the interfaces CompletionAwareAggregationStrategy and TimeoutAwareAggregationStrategy.
I have a in/out producer in Camel that only hangs around for a limited time before getting back to the caller. Some times this naturally results in a dead letter item and an exception being caught by the caller when the response is late.
What I would like to do is have the caller receive a timeout message instead of an exception and the item to never end up in the DLQ. Naturally I could put a listener on the DLQ but as the item has a home to go to it shouldn't really ever get to the DLQ.
Does anyone have a pattern for this? How would it be done? There are redundant consumer patterns (see Camel in Action link) but this is kind of a combined producer/consumer problem generated by the in/out pattern.
Sounds like you are using the Dead Letter Channel error handler, try using the noErrorHandler - http://camel.apache.org/error-handler