I am trying to use camel throttle to delay message processing if my cxf endpoint is down, need suggestion as How to disable throttling once cxf is up and running back.
If the cxf endpoint is down, throttling will still lead to errors-- just slower. I suggest 'stopping' the route instead, or breaking your route into two routes.. one to store messages in a queue and the other to read and send to the cxf endpoint.
The message queue acts like a natural buffer for traffic spikes, throttle and holding pen if the remote endpoint is down.
If you really must stick with the existing design using a throttle, then extract the throttle values in a config. During "down" periods set the throttle count to really high values. During "up" periods set the throttle time to 0ms and the throttle count to a really high value-- the throttle will effectively be disabled.
Related
How can I log actual seda queue depth every second on a timer.
E.g.
from("seda:messageParser?concurrentConsumers=5&size=5000)
.process(messageProcessor)
from("timer://sedaQueueDepthLogger?fixedRate=true")
.to(LOG_SEDA_QUEUE_DEPTH)
What should be LOG_SEDA_QUEUE_DEPTH above to achieve this?
You cannot log it that easily. From a custom processor, you can get the seda endpoint, and from there you have APIs to get the queue, and get the size, which you can then log.
There is also JMX API to find that information.
Trying to understand how throttling work is two consumers lead to a direct consumer that does some work and then sends a transformed message onward.
I can specify throttle on each consumer, but if the intent is not to overwhelm the destination, can I apply throttle to the direct route?
More importantly, would it act just as if it is throttling the 2 consumers, or would it consume and potentially create a "build up" of messages between the initial routes and the direct route?
Maybe, instead of direct it has to be seda?
Follow-up question: Can the throttled messages be flushed out if a graceful-shutdown begins?
There is a nice example here corresponding to your problem - in this case, a JMS consumer + a file consumer, both sending to a same seda endpoint.
You will notice that a single throttling policy is defined, and that each consumer is referring to this policy, so that the final destination is not overwhelmed.
Hope this helps.
I have a requirement where in at one point of time, I need to connect to multiple ftp/sftp endpoints (say 100 ftp endpoints) to download files and process them.
I have a route like below. The Seda queue further processes the messages by moving them into appropriate folders
from(ftp://username#host/foldername?password=XXXXX&include=.*).to("seda:"+routeId)
Now if I am starting all the FTP endpoints at the same time, which is resulting in JVM memory issues. How could I throttle the starting of the ftp endpoints? can I use a SEDA before the ftp to throttle (if so how can I use it)? Any other EIP's or ideas I could use to throttle the triggering of the polling ftp consumers?
You can look into the throttler dsl to if you want to throttle the fetching of the messages.
http://camel.apache.org/throttler.html
For controlling the startup you can look into the simplescheduleroutepolicy..
http://camel.apache.org/simplescheduledroutepolicy.html
It handles route activating and deactivating. Although I haven't used it myself but it looks like you can perhaps add a controlled delay on when routes should start and stop.
I have had this problem in the past solved it using cron in the following way:
from("ftp://username#host/foldername?password=XXXXX&include=.*&scheduler=quartz2&scheduler.cron=0/2+*+*+*+*+?")
You can set up every FTP consumer to pull at different times (say with one minute difference).
If you decided to go down this path, you can use the following website to construct your crons easily:
http://www.cronmaker.com/
Hope this helps.
R.
I have a requirement where I need to throttle by shaping (queuing) inbound traffic when client app sends more than 1000 requests in a 5 sec time span.
The solution I followed is:
I have a camel:throttle setting max requests to 1000 and timespan to 5 sec. When threshold is exceeded I am catching throttle exception and within the onException block, I am sending the throttled messages to an ActiveMQ request queue for further processing later as Camel is overloaded based on 1000 req/ 5 sec config.
I am successful in implementing the above, however I would like to have Camel consumer to further process later not all messages from ActiveMQ request queue at one shot instead process each message with a delay of 10 sec for e.g.
I am not able to set a parameter in ActiveMQ to say delay the message to consumer nor delay Camel consumer pulling off the message from request queue.
How do I cater to my above requirement
Please help
Thanks
Ramesh.
In another SO thread the winning answers promotes the following solution:
from("activemq:queueA").throttle(10).to("activemq:queueB")
To me this solution only makes sense, if you define a prefetch limit, without which the consumer would not care about any downstream throttling. This route should work:
from("activemq:queueA?&destination.consumer.prefetchSize=10").throttle(10).to("activemq:queueB")
This is the threory behind it, right from http://activemq.apache.org/what-is-the-prefetch-limit-for.html
So ActiveMQ uses a prefetch limit on how many messages can be streamed to a consumer at any point in time. Once the prefetch limit is reached, no more messages are dispatched to the consumer until the consumer starts sending back acknowledgements of messages (to indicate that the message has been processed). The actual prefetch limit value can be specified on a per consumer basis.
You can enable scheduled delivery of ActiveMQ and then set in your Camel Route AMQ_SCHEDULED_DELAY header and then send your exchange to a queue. This will result to introducing a delay of AMQ_SCHEDULED_DELAY millis before the message appears in the queue (i.e. be available for consumption).
Check this: http://activemq.apache.org/delay-and-schedule-message-delivery.html
I am using Apache camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc).
The way it is been implemented:
Boot a single camel context, the context is disabled for JMX and the
loadStatisticsEnabled is set to false on the ManagementStrategy. As
mentioned in a jira issue, addressed in 2.11.0 version, for disabling
the background management thread creation.
For each message a route is being constructed , the message is being
pushed to the route for delivery.
After the message is processed route is shutdown and removed from
context.
Did a small perf test by having 200 threads of dispatcher component, each sharing the same context.
Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds.
Issue CAMEL-5675 mentions that this has been fixed but still observing significant time being taken in starting up routes.
https://issues.apache.org/jira/browse/CAMEL-5675
The route that is being creating for http is
from("direct:"+dispatchItem.getID())
.toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis);
Each dispatchItem has a unique ID.
This is being active discussed elsewhere, where the user posted this question first: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html