Usecase:
We generate a csv file when user hits an endpoint, which we want to upload to different sources (FTP & S3). We need to encrypt & transform the files before uploading. Here's how the routes look like (oversimplified):
FTP Route:
from("file:temp?include=sample.csv&noop=true")
.routeId("ftpRoute" + LocalDateTime.now())
.marshal().pgp("file:temp/encryptionKey.asc", "someuserid", null, true, true)
.process(new SomeComplexProcessor())
.to("sftp:localhost:22/destinationDir?username=username&password=RAW(password)")
.setHeader(FILE_NAME, "metadata.txt")
.process(new MetadataFileGenerator())
.marshal()
.bindy(BindyType.Csv, MetadataFile.class)
.to("sftp:localhost:22/destinationDir?username=username&password=RAW(password)")
.process(new KillRouteProcessor());
S3 Route:
from("file:temp?include=sample.csv&noop=true")
.routeId("s3Route" + LocalDateTime.now())
.marshal().pgp("file:temp/encryptionKey.asc", "someuserid", null, true, true)
.process(new SomeComplexProcessor())
.to("aws-s3:bucketName?accessKey=ACCESS_KEY&secretKey=RAW(SECRET_KEY)®ion=REGION")
.setHeader(FILE_NAME, "metadata.txt")
.process(new MetadataFileGenerator())
.marshal()
.bindy(BindyType.Csv, MetadataFile.class)
.to("aws-s3:bucketName?accessKey=ACCESS_KEY&secretKey=RAW(SECRET_KEY)®ion=REGION")
.process(new KillRouteProcessor());
What's working:
S3 & SFTP upload routes are working correctly.
Requirements:
It'd be great if the code can be shared. Some different parameters / processors could be present in both routes.
File needs to be deleted after the execution of both routes.
We need to kill the routes after upload, as each request will have a different filename.
Above two routes cannot be merged into one, as there are too many if-else conditions inside the actual routes, which'd further complicate this.
Routes should be optional (eg. Choice of upload to s3/sftp or both should be available)
What I've tried:
Camel Direct: It helps with code reuse, but it doesn't allow multiple consumers (In my case these two routes)
Camel Seda: It allows multiple consumers, but it doesn't seem to allow synchronous routes.
Deleting the file outside of camel context. This is a problem, as we won't know how much time will the routes take to upload files.
Environment:
Camel 3.4.3, Spring Boot 2.2.3, Java8,
I managed to get it working by using a static route id (Removed LocalDateTime.now()). Here's what I learnt while fixing Camel issues.
Always provide route ids to every route. Especially while using 'direct' routes.
Never use dynamic route id. This is very important. I was seeing this issue when I had used LocalDateTime.now() in my route id. I was seeing this error before I changed:
Multiple consumers for the same endpoint is not allowed: direct://routeName .....
When using loops or calling any direct routes, always use enrich.
eg. .enrich("direct:subroute", AggregationStrategies.useOriginal()).
This will share a copy of parent route's headers to the subroute. It'll help you to avoid some weird issues.
Feel free to comment in case if you'd like to know more.
Related
Trying to google "route" in relation to Camel is like trying to google "the". Browsing the docs and can't find it either, only an interface called Route.
Inherited some code that looks like
rest("/someRoute")
.description("Some description")
.consumes("text/plain")
.produces("text/plain")
.post()
.route()
.to("direct:toSomewhere");
What does route() do? I have tried with and without route() and it doesn't seem to do anything.
Using .route allows you to define new route(s) within your rest-definition. It can be handy if your route is short or if you just want to process/transform/validate the message in someway before sending it to your actual consumer endpoint.
For example
rest("/someRoute")
.id("someRoute")
.description("Some description")
.post()
.consumes("text/plain")
.produces("text/plain")
.route()
.routeId("someRoutePost")
.process(new SomeMessageProcessor())
.to("direct:toSomewhere")
.end()
.endRest()
.get()
.route()
.routeId("someRouteGet")
.setHeader(Exchange.HTTP_RESPONSE_CODE, constant(405))
.setBody(constant("GET not allowed on this route"))
.end()
.endRest()
But if you just want to call direct consumer endpoint and do this stuff there instead you can do that.
it is up to ones preference really.
thanks, I see if I wanted to say call .log() I would have to put .route() first
Yes. Camel uses method-chaining with its Java-DSL where something like this is often required. When defining Rest most methods return RestDefinition but if you look closely .route method returns RouteDefition instead.
To get back to RestDefition from route one can use .endRest() as the .end() in the example doesn't really do anything other than make it easier to see where to RouteDefition block ends.
Update: Note that this example is for Camel 3.14.0. In Newer versions of Camel route() and endRest() methods have been Removed from RestDefition class. Example for Camel 3.18.1 can be found here.
I have this simple route in my RouteBuilder.
from("amq:MyQueue").routeId(routeId).log(LoggingLevel.DEBUG, "Log: ${in.headers} - ${in.body}")
As stated in the doc for HTTP-component:
Camel will store the HTTP response from the external server on the OUT body. All headers from the IN message will be copied to the OUT message, ...
I would like to know if this concept also applies to amq-component, routeId, and log? Is it the default behaviour, that IN always gets copied to OUT?
Thank you,
Hadi
First of all: The concept of IN and OUT messages is deprecated in Camel 3.x.
This is mentioned in the Camel 3 migration guide and also annotated on the getOut method of the Camel Exchange.
However, it is not (yet) removed, but what you can take from it: don't care about the OUT message. Use the getMessage method and don't use getIn and getOut anymore.
To answer your question:
Yes, most components behave like this
Every step in the route takes the (IN) message and processes it
The body is typically overwritten with the new processing result
The headers typically stay, new headers can be added
So while the Camel Exchange traverses the route, typically the body is continuously updated and the header list grows.
However, some components like aggregator create new messages based on an AggregationStrategy. In such cases nothing is copied automatically and you have to implement the strategy to your needs.
I would like to read files from a directory with camel file consumer but I need my route to be transacted. So I can not use threads inside the rout.
Is it ok to write multiply routes to read from the same endpoint (same directory) with a little change between the uris (for example the sort type) , and like this to avoid the Multiple consumers for the same endpoint is not allowed exception ?
Yeah sure you can do that, mind that you will have competing consumes for the same files now, so mind about read-locks. By default Camel use the marker file.
You can also use different delay so they dont poll at the same interval/time. And you can sort by random to make less chance of processing the same files.
It might be a silly question, but say I have a hughe message that I want to process with Camel. How will the number of steps in my route affect the memory usage? Does camel deep copy my message payload for every step in the route, even if the DSL-step only reads from the message or does it do something smart here?
Is it better to keep the route down and do things in a "hughe" bean for large messages or not?
This is an example route that does various things, but not changing the payload.
from("foo:bar")
.log(..)
.setProperty(..)
.setHeader(..)
.log(..)
.choice()
.when(simple(... ) )
.log(..)
.to(..)
.when(simple(..))
.log(..)
.to(..)
.end()
from my understanding, for a simple pipelined route like this, an Exchange is created containing the body once and passed along each step in the route. Other EIPs do cause the Exchange to be copied though (like multicast, wiretap, etc)...
as well, if you have steps along the route which interface with external resources which could result in any type of copy/clone/conversion/serialization of the body unnecessarily, then you might use something like the claim check pattern to reduce this.
The camel exchange is the same through the route the message objects are copied or recereated in the steps. The body is just referenced though. So normally you should not have a problem.
This is handled by each camel processor individually though. So some of the processors may copy the body. Typically this is the case when the processor really works on the body. So in this case it can not be avoided.
The DirectComponent documentation gives the following example:
from("activemq:queue:order.in")
.to("bean:orderServer?method=validate")
.to("direct:processOrder");
from("direct:processOrder")
.to("bean:orderService?method=process")
.to("activemq:queue:order.out");
Is there any difference between that and the following?
from("activemq:queue:order.in")
.to("bean:orderServer?method=validate")
.to("bean:orderService?method=process")
.to("activemq:queue:order.out");
I've tried to find documentation on what the behaviour of the to() method is on the Java DSL, but beyond the RouteDefinition javadoc (which gives the very curt "Sends the exchange to the given endpoint") I've come up blank :(
In the very case above, you will not notice much difference. The "direct" component is much like a method call.
Once you start build a bit more complex routes, you will want to segment them in several different parts for multiple reasons.
You can, for instance, create "sub routes" that could be reused among multiple routes in your Camel context. Much like you segment out methods in regular programming to allow reusability and make code more clear. The same goes for sub routes using, for instance the direct component.
The same approach can be extended. Say you want multiple protocols to be used as endpoints to your route. You can use the direct endpoint to create the main route, something like this:
// Three endpoints to one "main" route.
from("activemq:queue:order.in")
.to("direct:processOrder");
from("file:some/file/path")
.to("direct:processOrder");
from("jetty:http://0.0.0.0/order/in")
.to("direct:processOrder");
from("direct:processOrder")
.to("bean:orderService?method=process")
.to("activemq:queue:order.out");
Another thing is that one route is created for each "from()" clause in DSL. A route is an artifact in Camel, and you could do certain administrative tasks towards it with the Camel API, such as start, stop, add, remove routes dynamically. The "to" clause is just an endpoint call.
Once starting to do some real cases with somewhat complexity in Camel, you will note that you cannot get too many "direct" routes.
Direct Component is used to name the logical segment of the route. This is similar process to naming procedures in structural programming.
In your example there is no difference in message flow. In the terms of structural programming, we could say that you make a kind of inline expansion to your route.
Another difference is Direct component doesn't has any thread pool, the direct consumer process method is invoked by the calling thread of direct producer.
Mainly its used for break the complex route configuration like in java we used to have method for reusability. And also by configuring threads at direct route we can reduce the work for calling thread .
from(A).to(B).to(OUT)
is chaining
A --- B --- OUT
But
from(A ).to( X)
from(B ).to( X)
from( X).to( OUT )
where X is a direct:?
is basically like a join
A
\____ OUT
/
B
obviously these are different behaviours, and with the second you could implement anylogic you wanted, not just a serial chain