In Camel in Action book, it is written that default exception handling is for the life span of exchange object only. Suppose I am using following route -
from(sftp:{details}).to(file:{details})
1st query -> For the above route, I think the consumer is sftp and the producer is file. Am I right here?
2nd query -> How can we handle the exception at destination end point (which is outside the life span of exchange object).
1st You are correct.
2nd Any exception that will be raised before the file is successfully written to disk will trigger Camel's exception handling which will by default in file based components trigger rollback of the original file so that data won't be lost.
Related
I am trying to understand the unit of work concept in Camel. I have a simple question hopefully someone here can help.
If there are multiple routes involved in routing the Excachange for example
from("aws-sqs:Q1").to("direct:processMe");//route1
from("direct:processMe").to("direct:aws-post");//route2
from("direct:aws-post").to("htt4:myservice");//route3
Is the unit of work invoked at the end of each routes? Or only at the end of route3? In my example, will the SQS message be deleted off of SQS once route1 completes? Or will it wait until my message reaches "myservice"?
Thanks.
FOLLOW UP:
I've modified the route slighty:
from("aws-sqs:Q1").to("direct:processMe");//route1
from("direct:processMe").process(
new Processor(){
public void process(Exchange exchange) throws Exception {
throw new RuntimeException("fail on purpose");
}
}
).to("direct:aws-post");//route2
from("direct:aws-post").to("http4:myservice");//route3
The thought process is this:
If the unit of work is invoked at end of each route then once the message is read from SQS Queue it will be acknowledged as read by the SQS component. On the other hand if unit of work is only invoked once the Exchange is done routing through all routes, then the exception in route 2 will result in the message not being acknowledged and will be available for redelivery once the visibility period expires.
The test showed that the message remains on the Q despite being read by first route. It is picked up again and again (until it ends up in dead letter q). As such I strongly believe that the unit boundary is defined by the end of Exchange being routed.
A unit of work is basically a transaction.
By default a message for a route (a camel Exchange) runs inside a single UnitOfWork context.
In your example there are 3 UnitOfWorks set up, starting at each from and finishing at the final to in each route.
I would expect the message from SQS to be consumed after the 1st route finishes. To test, you could add in a sleep to allow you to check the queue.
from("direct:processMe").process(new Processor()
{ void process() { try { Thread.sleep(60000L) } catch (Exception e) { } }
}).to("direct:aws-post")
If you want the message to remain on the queue until myservice gets the message then you need to put the processing in a single route.
I have found a great explanation as to the unit of work boundary:
"The OnCompletion DSL name is used to define an action that is to take place when a Unit of Work is completed.
A Unit of Work is a Camel concept that encompasses an entire exchange. See Section 43.1, “Exchanges”. The onCompletion command has the following features:
The scope of the OnCompletion command can be global or per route. A route scope overrides global scope.
OnCompletion can be configured to be triggered on success for failure.
The onWhen predicate can be used to only trigger the onCompletion in certain situations.
You can define whether or not to use a thread pool, though the default is no thread pool."
In case of SQS processing, the consumer defines onCompletion on the exchange. So it is invoked only after the exchange is done routing.
The whole answer can be found here: Apache Camel Development Guide2.11. OnCompletion
I have a EIP design related query.I have a requirement to process csv file by chunks and call a Rest API.After completion of processing of whole file i need to call another Rest API telling processing is complete.I wanted the route to be transacted so i have queue in between in case of end system not available the retry will happen at broker level.
My flow is as below.
First flow:
csv File->Split by chunk of 100 records->Place message in queue
the second flow(Transacted route):
Picks message from queue ->call the rest API
the second flow is transacted.Since iam breaking the flow and it is asynchronous iam not sure how to call to the completion call.I do not have a persistent store to status of each chunk processing.
is there anyway i can achive it using JMS functionality or Camel?
What you can use for your first flow is the Camel Splitter EIP:
http://camel.apache.org/splitter.html
And closely looking at the doc, you will find that there are three exchange properties available for each split exchange:
CamelSplitIndex: A split counter that increases for each Exchange being split. The counter starts from 0.
CamelSplitSize: The total number of Exchanges that was splitted. This header is not applied for stream based splitting. From Camel 2.9 onwards this header is also set in stream based splitting, but only on the completed Exchange.
CamelSplitComplete: Whether or not this Exchange is the last.
As they are exchange properties, you should put them to JMS headers before sending the messages to a queue. But then you should be able to make use of the information at the second flow, so you can know which is the last message.
Keep in mind, though, that it's all asynchronous so the CamelSplitComplete flag doesn't necessarily mean the last message at the second flow. You may create a stateful counter or utilise the Resequencer EIP http://camel.apache.org/resequencer.html to deal with the asynchronicity.
I have a couple parallel routes in camel. One is reading sql data. One is reading a file on disk and then comparing to the prior sql data. I need to run route one, and based on if anything is imported, run route 2.
fromF("quartz2://mio/%s?cron={{route_1_cron}}", order).
log("Running data import...").
to("sql:{{sql_select}}").
choice().
when(body().isNull()).
stop().
when(body().isNotNull()).
bean(Utility.class,"incomingSqlData").
choice().when(header("status").isEqualTo(true).
to("direct:start").stop();
So far I am good. Now on the second route how do I start with from(direct:start) and then read the file from it's directory? Since I cannot have from(direct).from("file:..), since that would create two from routes.
And using from("direct:start").to("file:...") will try to write to the file.
Tl:dr: How should I start a route with direct and then read a file?
To expand on #noMad17 comment, you can use a content enricher. So, your from("direct:start") route can look something like:
from("direct:start")
.pollEnrich("file:...", new MyAggregationStrategy())
....
This will prompt your route to read a file.
Note that AggregationStrategy"is used to combine the original exchange and the resource exchange" and is optional. If not provided, then the body of resource exchange (i.e. the exchange resulting from reading the file) will overwrite the original exchange.
I have 2 routes. The first route uses poll enrich to check if a file is present. The second route uses a poll enrich on the same uri to read and process the file. The first route invokes the second via a SEDA queue, like so:
public void configure() throws Exception {
String myFile = "file://myDir?fileName=MyFile.zip&delete=false&readLock=none";
from("direct:test")
.pollEnrich(myFile, 10000)
.to("seda:myQueue")
;
from("seda:myQueue")
.pollEnrich(myFile, 10000)
.log("Do something with the body")
;
}
As it stands, if I execute the first route, the poll enrich finds a file, but when the poll enrich in the second route executes, it returns a body of null. If I just execute the second route on its own, it retrieves the file correctly.
Why does the second poll enrich return null, is the file locked? (I was hoping using a combination of noop,readLock, and delete=false would prevent any locking)
Does camel consider the second poll enrich as a duplicate, therefore filtering it out? (I have tried implementing my own IdempotentRepository to return false on contains(), but the second pollEnrich still returns null)
You may wonder why I'm trying to enrich from 2 routes, the first route has to check if a number of files exist, only when all files are present (i.e., pollEnrich doesn't return null) can the second route start processing them.
Is there an alternative to pollEnrich that I can use? I'm thinking that perhaps I'll need to create a bean that retrieves a file by URI and returns it as the body.
I'm using camel 2.11.0
I realize this is now an old topic, but I just had a similar problem.
I suggest you try the options:
noop=true
which you already have, and
idempotent=false
To tell Camel it is OK to process the same file twice.
Update after testing:
I actually tested this with both settings as suggested above, it works some times, but under moderate load, it fails, i.e. returns null body for some exchanges, although not all.
The documentation indicates that setting noop=true automatically sets idempotent=true, so I am not sure the idempotent setting is being honoured in this case.
Is there any specific reason why you are not using just one route?
I don't understand why you are using two routes for this. File component can check if the file is there and if it is, pull it. If you are worried about remembering the files so you don't get duplicates, you can use an idempotent repository. At least, based on your question, I don't think you need to complicate the logic using two routes and the content enricher EIP.
the second route returns NULL because the file was already consumed in the first route...if you are just looking for a signal message when all files are present, then use a file consumer along with an aggregator and possibly a claim check to avoid carrying around large payloads in memory, etc...
As you've probably learned this does not work as one might expect
noop=true&idempotent=false
my guess is that Camel ignores idempotent=false and as documented uses instance of MemoryMessageIdRepository. To work around this, one can configure file endpoint to use custom idempotent repo:
noop=true&idempotentRepository=#myRepo
and register custom repository in the registry or spring context:
#Component("myRepo")
public class MyRepo implements IdempotentRepository {
#Override
public boolean contains(Object o) {
return false;
}
...
}
Try pollEnrich with strategyMethodAllowNull="true". By default , this value is false. When it is false, the aggregation strategy looks for the existing Exchange body, to aggregate the content returned from file.
When we make strategyMethodAllowNull="true", the existing body is considered as null. So every time , the content of the file is set into the current exchange body
I have a in/out producer in Camel that only hangs around for a limited time before getting back to the caller. Some times this naturally results in a dead letter item and an exception being caught by the caller when the response is late.
What I would like to do is have the caller receive a timeout message instead of an exception and the item to never end up in the DLQ. Naturally I could put a listener on the DLQ but as the item has a home to go to it shouldn't really ever get to the DLQ.
Does anyone have a pattern for this? How would it be done? There are redundant consumer patterns (see Camel in Action link) but this is kind of a combined producer/consumer problem generated by the in/out pattern.
Sounds like you are using the Dead Letter Channel error handler, try using the noErrorHandler - http://camel.apache.org/error-handler