I am using Apache camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc).
The way it is been implemented:
Boot a single camel context, the context is disabled for JMX and the
loadStatisticsEnabled is set to false on the ManagementStrategy. As
mentioned in a jira issue, addressed in 2.11.0 version, for disabling
the background management thread creation.
For each message a route is being constructed , the message is being
pushed to the route for delivery.
After the message is processed route is shutdown and removed from
context.
Did a small perf test by having 200 threads of dispatcher component, each sharing the same context.
Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds.
Issue CAMEL-5675 mentions that this has been fixed but still observing significant time being taken in starting up routes.
https://issues.apache.org/jira/browse/CAMEL-5675
The route that is being creating for http is
from("direct:"+dispatchItem.getID())
.toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis);
Each dispatchItem has a unique ID.
This is being active discussed elsewhere, where the user posted this question first: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
Related
I have a simple Camel route consuming messages from ActiveMQ, processing and forwarding them to Rest webservices:
from("activemq:MyQueue").process("MyProcessor").to("http4:uri");
I configure concurrentConsumers=100 in the connectionfactory from activemq-component.
In the documentation:
if asyncConsumer is disabled(default) then the Exchange is fully processed before the JmsConsumer will pickup the next message from the JMS queue
Question:
In my route, when is the exchange of each message is fully processed? After the http-callee receives http response? If that is the case, I assume, my route configuration means:
At beginning, 1 message is consumed from each consumers and forwarded to the http
Each of these 100 consumers is waiting and will only consume again if the current http call gets http response from the current message.
Another question:
I found out that the default value of http4 component option connectionsPerRoute=20. As I have 100 consumers, should I set connectionsPerRoute=100?
Thank you,
Hadi
each jms thread runs simultaneously without knowing each other. in your example 100 threads are processed at the same time without getting blocked.You do not need to play in the number of threads of the http component, as this is done through jms threads from start to finish.
Is it legal in Apache Camel to start route (and consumer) after it was stopped programmatically?
I have route which is not started automatically (noAutoStartup()). App is also using Spring Boot.
Now, starting this route, stopping and starting again causing to consumers to be duplicated; observed on Hazelcast consumer.
I've tried to add ServiceHelper.startService(consumer) and ServiceHelper.stopService(consumer) with no effect.
I tried to stop route using camelContext.stopRoute(route.getId()) and control bus - same effect.
Camel 2.19.4; 2.20.1
Solution:
It turned out that HazelcastComponent cannot be stopped because it listens to events from Hazelcast and do not unregister this listener during component shutdown.
However, this component is deprecated since 2.20.x, and part of its functionality is available in HazelcastQueueComponent. Finally, HazelcastQueueConsumer in POLL mode respects the fact that service is stopped.
I'm looking for a best practise how to monitor the functionality of camel routes.
I know there are monitoring tools like hawtio and camelwatch, but that's not exactly what I'm looking for.
I want to know if a route is "working" as aspected, for example you have a route which listens on a queue(from("jms...")). Maybe there are messages in the queue, but the listener is not able to dequeue them because of some db issues or something else(depends on the jms provider). With the monitoring tools mentioned above you just see inflight/failed/completed messages but you don't see if the listener is able to get the messages -> so the route is not "working".
I know there is also apache BAM, maybe I have to do some more research, but somehow it looks like BAM creates new routes and you can't monitor existing routes. I also don't want to implement/define such business cases for each route, I look for a more generic way. It's also mentioned on the camel 3.0 idea board that BAM wasn't touched for 5 years, so I think people don't use it that often(which means for me it doesn't fit their needs exactly).
I had similar requirement some time ago and at the end I developed a small Camel application for monitoring.
It run on timer, query different Camel applications installed in remote servers through JMX/Jolokia and if LastExchangeCompletedTimestamp of the route I am interested in is older than some time interval, send a mail to administrators.
Maybe this approach is too simple for your scenario, but could be an option.
(Edit: more details added)
Principal points:
Main routes queries DB for entities to control and spawns controlling routes
Controlling routes fires on quartz and http post the following url
.to("http://server:port/app/jolokia/?"+
"maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false")
sending the following jsonRequest body
LinkedHashMap<String,Object> request=new LinkedHashMap<String,Object>();
request.put("type","read");
request.put("mbean","org.apache.camel:"+entity.getRouteId());
jsonRequest=mapper.writeValueAsString(request);
As response you get another JSON, parse it and get LastExchangeCompletedTimestamp value
We're having some came routes defined in a single CamelContext which contains Web services,activemq.. in the Route.
Initially we've deployed the Routes as WAR in single Jboss node.
To scale out(usually we're doing for web services) , I've deployed the same CamelContext in multiple Jboss nodes.
But the performance is actually decreased.
FYI: All the CamelContexts points to the Same activemq brokers.
Here are my questions:
How to load balance/ Fail over camel context in different machines?
If CamelContexts are deployed in multiple nodes, Will aggregation work correctly?
Kindly give your thoughts!
Without seeing your system in detail, there is no way of knowing why it has slowed down so I'll pass over that. For your other two questions:
Failover
You don't say what sort of failover/load balancing behaviour you want. The not-very-helpful Camel documentation is here: http://camel.apache.org/clustering-and-loadbalancing.html.
One mechanism that works easily with Camel and ActiveMQ is to deploy to multiple servers and run active-active, sharing the same ActiveMQ queues. Each route attempts to read from the same queue to get a message to process. Only one route will get the message and therefore only one route processes it. Other routes are free to read subsequent messages, giving you simple load balancing. If one route crashes, the other routes will continue to process the messages, there will just be reduced capacity on your system.
If you need to provide fault tolerance for your web services then you need to look outside Camel and use something like Elastic Load Balancing. http://aws.amazon.com/elasticloadbalancing/
Aggregation
Each Camel context will run independently of the other contexts so one context will aggregate messages independently of what other contexts are up to. For example, suppose you have an aggregator that stores messages from ActiveMQ queue until receives a special end-of-batch message. If you have the aggregator running in two different routes, the messages will be split between the two routes and only one route will receive the end-of-batch message. So one aggregator will sit there with half the messages and do nothing. The other aggregator will have the other messages and will process the end-of-batch message but won't know about the messages the other route picked up.
I have a Camel based application which receives a request and gives the reply from cache but in between this process it updates the database which i want it to run in a different thread , can anyone tell me how can i achieve this, i tried with WireTap and SEDA but it does not work that way...any help appreciated.
<camel:wireTap uri="seda:tap" processorRef="updateHitCountProcessor"/>
In updateHitCountProcessor I have written code to update table
it is updating the database in same thread (i.e main route thread)
You need to do
<camel:wireTap uri="ref:updateHitCountProcessor"/>
The processorRef attribute is creating and sending a new message, and not for tapping the existing message. So you should not use that.
The uri is used for sending the message which happens in a separate thread. So when you send it to the ref endpoint it will do that in another thread, and call your processor.
You can find details on the wire tap page at: http://camel.apache.org/wire-tap
From the documentation of the camel-seda component (here):
By default, the SEDA endpoint uses a single consumer thread, but you
can configure it to use concurrent consumer threads.
You can add a thread pool to a SEDA endpoint like this:
<from uri="seda:stageName?concurrentConsumers=5" />