I have application which allows dynamic creation and starting of routes. For example user can create route from IMAP to file and start it.
Problem: Multiple routes reading from same mail account through IMAP.
Caused by: org.apache.camel.FailedToStartRouteException: Failed to start route c1152_route because of Multiple consumers for the same endpoint is not allowed: imap://localhost:3143?delay=1000&password=xxxxxx&searchTerm.fromSentDate=now-24h&searchTerm.unseen=false&username=user#user
I've tried two approaches:
1.Creating additional single route from IMAP to multiple recipient routes. This one works but I'm not sure about this solution. It requires additional checks on starting/stopping of route because few routes might be dependent on this one.
from(imap())
.recipientList(imapMsgToDirectChannelRouter())
2.Creating IMAP endpoints with unique uri. For example unique searchTerm.fromSentDate for each routes. This one also works.
Is there any better solution to this problem?
I would definitely go for option#1 (recipientList).
I do not understand why would have to stop & start the 'master' route (the "imap" one).
Depending on the recipients you add or remove, you would start and stop the corresponding child routes, but not touch the master one.
I'm thinking on something like this:
from("imap:...")
.id("master-route")
.recipientList( simple("bean:computeRecipients") )
.parallelProcessing();
Where the on-the-fly calculation of the exact list of recipients is delegated to some bean.
This way, you do not have to touch the "master-route", instead just make the bean aware of the recipients
Related
I want to listen on ActiveMQ topic based on the hostname of system and some other logic. I planned to use pollEnrich for it so I evaluate my logic and provide topic name in pollEnrich but as per document:
pollEnrich or enrich does not access any data from the current Exchange which means when polling it cannot use any of the existing headers you may have set on the Exchange. For example you cannot set a filename in the Exchange.FILE_NAME header and use pollEnrich to consume only that file. For that you must set the filename in the endpoint URI.
How i can figure out this
from("timer://ipc?repeatCount=1")
.. some logic..
.setHeader("topic_no",simple("{{env:HOSTNAME}}"))
.pollEnrich("mqtt:foo?host=tcp://0.0.0.0:1883&subscribeTopicNames=${header.topic_no}/status&clientId=ipc")
.to("log:my?showAll=true&multiline=true");
Please don't suggest to use hostname directly in URI. As I highlighted I have to compute other logic too.
What other option or way I can use?
Will pollEnrich kept listening on topic or it will listen once and end the route?
Update1:
I figured out we can use simple expression with for dynamic URI, But one issue with pollEnrich it only pick one message how i can make sure it kept on listening as consumer? I want that before pollEnrich part get execute once and TopicListener kept listening till application is up.
Will pollEnrich kept listening on topic or it will listen once and end the route?
Same as the fact you have figure out, Camel pollEnrich component will listen on topic and consume at most one message per call.
What other option or way I can use?
Repeat pollEnrich by loop
Create new route at run-time by routeBuilder
Option 1 is naive, but simple in concept. pollEnrich will do once and loop will repeat it. However, this method need to handle more scenario than you might expected.
Option 2 is a better approach. You create a route at run-time and the consumer endpoint URI is pass by variable. That said, you can create the consumer route dynamically after your computation logic.
Example for routeBuilder
I'm looking for a best practise how to monitor the functionality of camel routes.
I know there are monitoring tools like hawtio and camelwatch, but that's not exactly what I'm looking for.
I want to know if a route is "working" as aspected, for example you have a route which listens on a queue(from("jms...")). Maybe there are messages in the queue, but the listener is not able to dequeue them because of some db issues or something else(depends on the jms provider). With the monitoring tools mentioned above you just see inflight/failed/completed messages but you don't see if the listener is able to get the messages -> so the route is not "working".
I know there is also apache BAM, maybe I have to do some more research, but somehow it looks like BAM creates new routes and you can't monitor existing routes. I also don't want to implement/define such business cases for each route, I look for a more generic way. It's also mentioned on the camel 3.0 idea board that BAM wasn't touched for 5 years, so I think people don't use it that often(which means for me it doesn't fit their needs exactly).
I had similar requirement some time ago and at the end I developed a small Camel application for monitoring.
It run on timer, query different Camel applications installed in remote servers through JMX/Jolokia and if LastExchangeCompletedTimestamp of the route I am interested in is older than some time interval, send a mail to administrators.
Maybe this approach is too simple for your scenario, but could be an option.
(Edit: more details added)
Principal points:
Main routes queries DB for entities to control and spawns controlling routes
Controlling routes fires on quartz and http post the following url
.to("http://server:port/app/jolokia/?"+
"maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false")
sending the following jsonRequest body
LinkedHashMap<String,Object> request=new LinkedHashMap<String,Object>();
request.put("type","read");
request.put("mbean","org.apache.camel:"+entity.getRouteId());
jsonRequest=mapper.writeValueAsString(request);
As response you get another JSON, parse it and get LastExchangeCompletedTimestamp value
We're having some came routes defined in a single CamelContext which contains Web services,activemq.. in the Route.
Initially we've deployed the Routes as WAR in single Jboss node.
To scale out(usually we're doing for web services) , I've deployed the same CamelContext in multiple Jboss nodes.
But the performance is actually decreased.
FYI: All the CamelContexts points to the Same activemq brokers.
Here are my questions:
How to load balance/ Fail over camel context in different machines?
If CamelContexts are deployed in multiple nodes, Will aggregation work correctly?
Kindly give your thoughts!
Without seeing your system in detail, there is no way of knowing why it has slowed down so I'll pass over that. For your other two questions:
Failover
You don't say what sort of failover/load balancing behaviour you want. The not-very-helpful Camel documentation is here: http://camel.apache.org/clustering-and-loadbalancing.html.
One mechanism that works easily with Camel and ActiveMQ is to deploy to multiple servers and run active-active, sharing the same ActiveMQ queues. Each route attempts to read from the same queue to get a message to process. Only one route will get the message and therefore only one route processes it. Other routes are free to read subsequent messages, giving you simple load balancing. If one route crashes, the other routes will continue to process the messages, there will just be reduced capacity on your system.
If you need to provide fault tolerance for your web services then you need to look outside Camel and use something like Elastic Load Balancing. http://aws.amazon.com/elasticloadbalancing/
Aggregation
Each Camel context will run independently of the other contexts so one context will aggregate messages independently of what other contexts are up to. For example, suppose you have an aggregator that stores messages from ActiveMQ queue until receives a special end-of-batch message. If you have the aggregator running in two different routes, the messages will be split between the two routes and only one route will receive the end-of-batch message. So one aggregator will sit there with half the messages and do nothing. The other aggregator will have the other messages and will process the end-of-batch message but won't know about the messages the other route picked up.
Suppose I have 3 webservers behind a reverse proxy/loadbalancer. Currently I use Apache 2 but don't mind switching.
Those webservers are not equal. They have different sets of local data meaning that different kinds of requests will be handled more efficiently on specific webserver.
I want to plug some custom routing logic into my loadbalancer, that would on start of a new session select a webserver for it, then tie that session to that node until it breaks down (sticky session).
For example: a new user comes to the webapp, load balancer runs my custom code somehow, this code finds out information about that user and makes a conclusion that this user should be handled on node1. The balancer proxies that user to node1 and sticks it to it. When another user comes, custom routing logic suggests node3, to which loadbalancer then proxies this user.
Is it possible to do with Apache 2? If not, what proxy/loadbalancer would give me that ability?
Thank you.
Apache mode_proxy_balancer has stickyness if you want to stay on apache.
Otherwise, have a look at Haproxy. The first line of the abstract states: "route HTTP requests depending on statically assigned cookies ;"
You just have to set a cookie to know which backend you want the user to be stuck to.
First of all thanks to folks who are currently involved in the development of Camel, I am grateful for all the hard work they have put in.
I am looking for some design advice.
The architecture is something like this:
I have a bunch of Java classes which when instantiated are required to connect to each other and send messages using Apache Camel. The design constraints require me to create a framework such that all routing information, producers, consumers, endpoints etc should be a part of the camel-context.xml.
An individual should have the capability to modify such a file and completely change the existing route without having the Java code available to him.(The Java code would not be provided, only the compiled Jar would be)
For example in One setup,
Bean A ->Bean B->Bean C->file->email.
in another
Bean B->Bean A->Bean C->ftp->file->email
We have tried various approached, but if the originating bean is not implemented as a Java DSL, the messages rate is very high because camel constantly invokes Bean A in the first example and Bean B in the second(they being the source).
Bean A and Bean B originate messages and are event driven. In case the required event occurs, the beans send out a notification message.
My transformations are very simple and I do not require the power of Java DSL at all.
To summarize, I have the following questions:
1) Considering the above constraints, I do I ensure all routing information, including destination addresses, everything is a part of the camel context file?
2) Are there example I can look at for keeping the routing information completely independent of the java code?
3) How do I ensure Camel does not constantly invoke the originating bean?
4) Does Camel constantly invoke just the originating bean or any bean it sends & messages to irrespective of the position of the bean in the entire messaging queue?
I have run out of options trying various ways to set this up. Any help would be appreciated.
Read about hiding the middleware on the Camel wiki pages. This allows you to let clients use an interface to send/receive messages but totally unaware of Camel (no Camel API used at all).
Even better consider buying the Camel in Action book and read chapter 14 which talks about this.
http://www.manning.com/ibsen/
Save 41% on Manning books: Camel in Action or ActiveMQ in Action. Use code s2941. Expires 6th oct. http://www.manning.com/ibsen/
If you consider using ServiceMix of FuseESB, you might want to separate your routes in two parts.
First part would be the Event-driver bean that trigger the route. It could push messages to the ServiceNMR (see http://camel.apache.org/nmr.html).
The other part would be left to the framework users, using Spring DSL. It would just listen to message on the NMR (push by the other route) and do whatever they want with it.
Of course endpoint definition could be propertized using servicemix configuration service (see http://camel.apache.org/properties.html#Properties-UsingBlueprintpropertyplaceholderwithCamelroutes)