We have a stand alone application where we should be able to stand up JMS listeners using database configuration dynamically. If there is any change in the concurrentConsumers count in the database, then we need to update the listeners accordingly. We are using camel 2.16.2 and Tibco 6.3.0. Precisely, the below are our requirements.
Ability to add and remove JMS listeners dynamically.
To achieve this, we are creating our OwnRouteBuilder (which extends camel RouteBuilder) and adding it using camelContext.addRoutes(RouteBuilder) dynamically which is working as expected.
Ability to increase or decrease JMS listeners dynamically.
To achieve this, initially we tried to stop the route, update concurrentConsumers count, and then start route. It is working fine only when we increase the concurrentConsumers count. But its not working if we decrease( its not changing the concurrentConsumers count).We can still see the same receivers count from tibco console.
As an alternate solution , we tried with camelContext.stopRoute(routeId) and then camelContext.removeRoute(routeId) so that we can add a fresh route with updated concurrentConsumers count later. In this case also, though it is stopping the routes from consuming messages, it is not changing the receivers count from tibco console.
I tried with camelContext.removeEndpoints(endpointUri), camelContext.shutdownRoute(routeId), camelContext.removeRouteDefinition(routedefinition) methods.
But all these methods only stopping the routes from consuming messages, but not clearing those receivers count from tibco console.
The receivers count from tibco console is becoming zero only when we terminate the application.
Is there any way that we can make it from camel so that those receivers count from tibco console becomes zero.
You can change the concurrentConsumer counts dynamically at runtime using JMX to set the values on the JMX attribute on the endpoint. Then the route dont have to be stopped/removed/started etc.
Related
We are building a system where an external system sends us "Refresh" events. These events tell us to refresh the state of an item identified by the id provided in the RefreshEvent.
We handle these events through a Camel route.
The refresh may go wrong, though. If this is case the event is moved to a "Retry" SEDA queue with appropiate retry/backup setup. However, we may experience multiple "Refresh" event for the same item, while we are unable to fetch the new state.
Ideally we'd like to only have unique itemIds in the Retry queue. The inflight itemIds should be unique.
Is there a way to do this in Camel.
I have looked at the IdempotentConsumer, but that has a scope of all received exchanges, I'm only interested in the current inflight exchanges.
If you are not able to use the IndempotentConsumer pattern, you will have to implement the logic yourself and possible change the Seda endpoint to something more sophisticated.
You could use a message queue supporting JMS selectors or property filtering.
ActiveMQ, to name one, has some capabilities that could be used in implementing the solution you need.
I am using Apache camel to trigger various actions based on timer events. For timer events to trigger, I am using camel-quartz. And a sample trigger looks like this:
from("quartz://job_timers/customer_activation?cron=0+0+0+*+*+?+*&trigger.timeZone=America/Chicago")
.routeId("TrigCustActivation")
.log(LoggingLevel.INFO, "Triggered Customer Activation Job")
.to("direct://set/headers/activation")
.to("direct://customer/activation");
Over the period of time, the number of such triggers have increased a lot. And they will be increasing even more. So, instead of keeping a hard-copy of when which job will trigger, I was planning to create a web-route, so that when I perform a GET request, it would fetch all the active routes, and print their RouteIds (I will update all the RouteIds to have time as well. Say, the routeId for above flow will become "TrigCustActivation_00_00_CST_Daily").
I was able to print all the route at application startup, but failed to fetch those dynamically through a GET request.
Can we access other RouteIds from a running route? Is my approach feasible?
Yes there is API on CamelContext where you can fetch all the current routes, and you can then get their IDs etc. See the javadoc of this class.
http://static.javadoc.io/org.apache.camel/camel-core/2.21.0/org/apache/camel/CamelContext.html
I'm looking for a best practise how to monitor the functionality of camel routes.
I know there are monitoring tools like hawtio and camelwatch, but that's not exactly what I'm looking for.
I want to know if a route is "working" as aspected, for example you have a route which listens on a queue(from("jms...")). Maybe there are messages in the queue, but the listener is not able to dequeue them because of some db issues or something else(depends on the jms provider). With the monitoring tools mentioned above you just see inflight/failed/completed messages but you don't see if the listener is able to get the messages -> so the route is not "working".
I know there is also apache BAM, maybe I have to do some more research, but somehow it looks like BAM creates new routes and you can't monitor existing routes. I also don't want to implement/define such business cases for each route, I look for a more generic way. It's also mentioned on the camel 3.0 idea board that BAM wasn't touched for 5 years, so I think people don't use it that often(which means for me it doesn't fit their needs exactly).
I had similar requirement some time ago and at the end I developed a small Camel application for monitoring.
It run on timer, query different Camel applications installed in remote servers through JMX/Jolokia and if LastExchangeCompletedTimestamp of the route I am interested in is older than some time interval, send a mail to administrators.
Maybe this approach is too simple for your scenario, but could be an option.
(Edit: more details added)
Principal points:
Main routes queries DB for entities to control and spawns controlling routes
Controlling routes fires on quartz and http post the following url
.to("http://server:port/app/jolokia/?"+
"maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false")
sending the following jsonRequest body
LinkedHashMap<String,Object> request=new LinkedHashMap<String,Object>();
request.put("type","read");
request.put("mbean","org.apache.camel:"+entity.getRouteId());
jsonRequest=mapper.writeValueAsString(request);
As response you get another JSON, parse it and get LastExchangeCompletedTimestamp value
We're having some came routes defined in a single CamelContext which contains Web services,activemq.. in the Route.
Initially we've deployed the Routes as WAR in single Jboss node.
To scale out(usually we're doing for web services) , I've deployed the same CamelContext in multiple Jboss nodes.
But the performance is actually decreased.
FYI: All the CamelContexts points to the Same activemq brokers.
Here are my questions:
How to load balance/ Fail over camel context in different machines?
If CamelContexts are deployed in multiple nodes, Will aggregation work correctly?
Kindly give your thoughts!
Without seeing your system in detail, there is no way of knowing why it has slowed down so I'll pass over that. For your other two questions:
Failover
You don't say what sort of failover/load balancing behaviour you want. The not-very-helpful Camel documentation is here: http://camel.apache.org/clustering-and-loadbalancing.html.
One mechanism that works easily with Camel and ActiveMQ is to deploy to multiple servers and run active-active, sharing the same ActiveMQ queues. Each route attempts to read from the same queue to get a message to process. Only one route will get the message and therefore only one route processes it. Other routes are free to read subsequent messages, giving you simple load balancing. If one route crashes, the other routes will continue to process the messages, there will just be reduced capacity on your system.
If you need to provide fault tolerance for your web services then you need to look outside Camel and use something like Elastic Load Balancing. http://aws.amazon.com/elasticloadbalancing/
Aggregation
Each Camel context will run independently of the other contexts so one context will aggregate messages independently of what other contexts are up to. For example, suppose you have an aggregator that stores messages from ActiveMQ queue until receives a special end-of-batch message. If you have the aggregator running in two different routes, the messages will be split between the two routes and only one route will receive the end-of-batch message. So one aggregator will sit there with half the messages and do nothing. The other aggregator will have the other messages and will process the end-of-batch message but won't know about the messages the other route picked up.
I am using Apache camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc).
The way it is been implemented:
Boot a single camel context, the context is disabled for JMX and the
loadStatisticsEnabled is set to false on the ManagementStrategy. As
mentioned in a jira issue, addressed in 2.11.0 version, for disabling
the background management thread creation.
For each message a route is being constructed , the message is being
pushed to the route for delivery.
After the message is processed route is shutdown and removed from
context.
Did a small perf test by having 200 threads of dispatcher component, each sharing the same context.
Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds.
Issue CAMEL-5675 mentions that this has been fixed but still observing significant time being taken in starting up routes.
https://issues.apache.org/jira/browse/CAMEL-5675
The route that is being creating for http is
from("direct:"+dispatchItem.getID())
.toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis);
Each dispatchItem has a unique ID.
This is being active discussed elsewhere, where the user posted this question first: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html