I created a RoutePolicy that suspends a consumer when a configurable number of errors occurred in the route.
Before I suspend the consumer I want to make sure it will be resumed after a configurable amount of time (for example 30 minutes after suspension).
What is the best way to achieve this?
I tried to use the onExchangeBegin method of the RoutePolicy. But in a test I found that it is no more executed when the route is suspended (as I assumed).
I tried to create a SimpleScheduledRoutePolicy before suspending the route but I didn't found a way to register this new Bean in the Camel context (backed by Spring).
Therefore I currently create a TimerTask that sends a message to Camel Control Bus to resume the route. That works, but feels a bit of alien since Camel does not know about such resume tasks.
Is there another, more "Camel native" way to reach my goal?
Before you suspend the consumer, you can create a dummy file.
Have another route polling for the dummy file with a filter to check if it was created 30 minutes prior. Something like:
from("file:dummyLocation?include=.dummy&delete=true&filter=#filterFileOlderThanThirtyMins)
.to("controlbus:route?routeId=suspendedRoute&action=start")
This is just off the top of my head, though!
Related
Is there a way using which we can stop the Camel route if we don't have any more message in ActiveMQ? My required scenario.
Fetch all the message from ActiveMQ queue and process them.
Poll for 2-3 more times to check if we have any other new message, is yes execute step #1.
If no message in present stop the route and start it back after say 5 min (which I guess can be achieved by Polling Strategy).
Have a look at this answer. It polls the queue with a scheduler and a polling strategy (POJO).
With the scheduler you can choose the interval of polling
With the timeout of the polling strategy consumer you can stop consuming (e.g. if no message arrives for 5 seconds the queue is probably empty)
If you want to stop/start the consumer completely, you can add Camel Control Bus to the mix. You could then start and stop the consumer route.
I have a file drop endpoint that I poll from. I need to poll the files in sequential order as they are received and I am using a cron expression to poll at only certain hours to the day. Here is my file input:
file:///tmp/input?idempotent=true&moveFailed=/tmp/error&readLock=changed&readLockCheckInterval=2500&sortBy=file:modified&move=processed/&scheduler=quartz2&scheduler.cron=0+0/5+0-3,5-23+*+*+?
The issue that I have is that Camel polls a batch of files but then subsequently newer files are written to the directory so in a subsequent poll a new file is processed before the previous batch is completed.
I added some properties to my route to show the batch size and whether or not it has been completed just for some info:
<camel:log message="Camel batch size: $simple{property.CamelBatchSize}, Camel Batch Index: $simple{property.CamelBatchIndex}, Camel Batch finished: $simple{property.CamelBatchComplete}"/>
How can I tell Camel not to poll until the previous batch is complete? I do this because order of file processing is important. Thanks!
Not sure if there is any existing method to accomplish your goal by cron job in file route directly. However, You could achieve your task by using 3 routes.
Cron job route
Emit suspend signal to Stopper route if Collector route is already started (check via controlBus component)
Startup Collector route at correct time (trigger by controlBus component)
Collector route
Control file consumer behavior
Emit complete signal to Stopper route when batch completed
Stopper route
Suspend Collector route when signal received (trigger by controlBus component)
I have a requirement where in at one point of time, I need to connect to multiple ftp/sftp endpoints (say 100 ftp endpoints) to download files and process them.
I have a route like below. The Seda queue further processes the messages by moving them into appropriate folders
from(ftp://username#host/foldername?password=XXXXX&include=.*).to("seda:"+routeId)
Now if I am starting all the FTP endpoints at the same time, which is resulting in JVM memory issues. How could I throttle the starting of the ftp endpoints? can I use a SEDA before the ftp to throttle (if so how can I use it)? Any other EIP's or ideas I could use to throttle the triggering of the polling ftp consumers?
You can look into the throttler dsl to if you want to throttle the fetching of the messages.
http://camel.apache.org/throttler.html
For controlling the startup you can look into the simplescheduleroutepolicy..
http://camel.apache.org/simplescheduledroutepolicy.html
It handles route activating and deactivating. Although I haven't used it myself but it looks like you can perhaps add a controlled delay on when routes should start and stop.
I have had this problem in the past solved it using cron in the following way:
from("ftp://username#host/foldername?password=XXXXX&include=.*&scheduler=quartz2&scheduler.cron=0/2+*+*+*+*+?")
You can set up every FTP consumer to pull at different times (say with one minute difference).
If you decided to go down this path, you can use the following website to construct your crons easily:
http://www.cronmaker.com/
Hope this helps.
R.
I have a Camel based application which receives a request and gives the reply from cache but in between this process it updates the database which i want it to run in a different thread , can anyone tell me how can i achieve this, i tried with WireTap and SEDA but it does not work that way...any help appreciated.
<camel:wireTap uri="seda:tap" processorRef="updateHitCountProcessor"/>
In updateHitCountProcessor I have written code to update table
it is updating the database in same thread (i.e main route thread)
You need to do
<camel:wireTap uri="ref:updateHitCountProcessor"/>
The processorRef attribute is creating and sending a new message, and not for tapping the existing message. So you should not use that.
The uri is used for sending the message which happens in a separate thread. So when you send it to the ref endpoint it will do that in another thread, and call your processor.
You can find details on the wire tap page at: http://camel.apache.org/wire-tap
From the documentation of the camel-seda component (here):
By default, the SEDA endpoint uses a single consumer thread, but you
can configure it to use concurrent consumer threads.
You can add a thread pool to a SEDA endpoint like this:
<from uri="seda:stageName?concurrentConsumers=5" />
I am using Apache camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc).
The way it is been implemented:
Boot a single camel context, the context is disabled for JMX and the
loadStatisticsEnabled is set to false on the ManagementStrategy. As
mentioned in a jira issue, addressed in 2.11.0 version, for disabling
the background management thread creation.
For each message a route is being constructed , the message is being
pushed to the route for delivery.
After the message is processed route is shutdown and removed from
context.
Did a small perf test by having 200 threads of dispatcher component, each sharing the same context.
Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds.
Issue CAMEL-5675 mentions that this has been fixed but still observing significant time being taken in starting up routes.
https://issues.apache.org/jira/browse/CAMEL-5675
The route that is being creating for http is
from("direct:"+dispatchItem.getID())
.toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis);
Each dispatchItem has a unique ID.
This is being active discussed elsewhere, where the user posted this question first: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html