I'm looking for a best practise how to monitor the functionality of camel routes.
I know there are monitoring tools like hawtio and camelwatch, but that's not exactly what I'm looking for.
I want to know if a route is "working" as aspected, for example you have a route which listens on a queue(from("jms...")). Maybe there are messages in the queue, but the listener is not able to dequeue them because of some db issues or something else(depends on the jms provider). With the monitoring tools mentioned above you just see inflight/failed/completed messages but you don't see if the listener is able to get the messages -> so the route is not "working".
I know there is also apache BAM, maybe I have to do some more research, but somehow it looks like BAM creates new routes and you can't monitor existing routes. I also don't want to implement/define such business cases for each route, I look for a more generic way. It's also mentioned on the camel 3.0 idea board that BAM wasn't touched for 5 years, so I think people don't use it that often(which means for me it doesn't fit their needs exactly).
I had similar requirement some time ago and at the end I developed a small Camel application for monitoring.
It run on timer, query different Camel applications installed in remote servers through JMX/Jolokia and if LastExchangeCompletedTimestamp of the route I am interested in is older than some time interval, send a mail to administrators.
Maybe this approach is too simple for your scenario, but could be an option.
(Edit: more details added)
Principal points:
Main routes queries DB for entities to control and spawns controlling routes
Controlling routes fires on quartz and http post the following url
.to("http://server:port/app/jolokia/?"+
"maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false")
sending the following jsonRequest body
LinkedHashMap<String,Object> request=new LinkedHashMap<String,Object>();
request.put("type","read");
request.put("mbean","org.apache.camel:"+entity.getRouteId());
jsonRequest=mapper.writeValueAsString(request);
As response you get another JSON, parse it and get LastExchangeCompletedTimestamp value
Related
Couple of questions:
I have list of components on the client app which has some near real time info e.g. status, which I want to display.
I have server app, which can pull the status info from a third party REST endpoint.
My question is, should I cycle through all the components in the client app and request the server app for the status?
Or should I have a server worker thread, which pulls the status info and publishes on the websocket, which the client can then update the state of the component.
Or is there background thread which I can run on the client app, which will update the status and the state. How will this conflict with the dispatch/queuing of events from user interactions.
I think I might be asking some of the architectural questions and the answers might be "it depends" ambiguous, but anyone who has done this before and any guidance is appreciated.
Thanks, Rajesh
It depends :) But it's safe to go with server-side approach since with client-side you would have to deal with CORS and cross-domain ajax calls in general. Most of the 3rd party API do not allow to make arbitrary AJAX calls from other domains. Those that do allow that usually have API quota which is again easier to manage since you can keep your keys secret on the server and throttle and cache requests.
Server side approach requires more effort though. So it's a prototype and 3rd party API allows cross-domain requests – go for it, it's easier, for production app, consider doing this on the server.
For client-side approach if 3rd party API doesn't offer subscriptions, yes you would have to poll, but you don't have to cycle through components. You can abstract this polling in one, root component and then just pass props down.
You are right, the answer is "it depends".
Basically, you have two options:
Poll the server for the current status.
I believe you do not have to cycle through all the components and query their status. You could just have an API that provides the server the timestamp when you last queried the status, and the server will respond with just the information that has changed since the last query.
This is simple and will work fine if the updates are not huge, and you can afford to be a little late.
You could have a dedicated websocket connection with the server
In this case, the server will push any new updates to your front-end whenever any new update is available. This is a little cumbersome to implement, but is the right approach if the updates are near real-time.
To answer your question about having a background-thread on the front-end: no, you cannot have background-threads on the front-end. Javascript doesn't work that way. What you do have are callbacks. Whenever the server pushes you any new information, a callback, that you define, will be called and you can do whatever UI changes you need from here.
My question is, should I cycle through all the components in the client app and request the server app for the status?
No, this sounds very inefficient to me.
However, if you decide to poll from the client, it should be done only from a single component that is parent to all children that need the information. The parent then passes its state to the children and they update on each poll.
Or should I have a server worker thread, which pulls the status info and publishes on the websocket, which the client can then update the state of the component.
Absolutely. Let this socket be in your top-level application component which holds the real-time info in its state and passes down to its children. Whenever information gets published to the socket, update the top-level application state with the new real-time info and all children will rerender displaying the most current information.
We're having some came routes defined in a single CamelContext which contains Web services,activemq.. in the Route.
Initially we've deployed the Routes as WAR in single Jboss node.
To scale out(usually we're doing for web services) , I've deployed the same CamelContext in multiple Jboss nodes.
But the performance is actually decreased.
FYI: All the CamelContexts points to the Same activemq brokers.
Here are my questions:
How to load balance/ Fail over camel context in different machines?
If CamelContexts are deployed in multiple nodes, Will aggregation work correctly?
Kindly give your thoughts!
Without seeing your system in detail, there is no way of knowing why it has slowed down so I'll pass over that. For your other two questions:
Failover
You don't say what sort of failover/load balancing behaviour you want. The not-very-helpful Camel documentation is here: http://camel.apache.org/clustering-and-loadbalancing.html.
One mechanism that works easily with Camel and ActiveMQ is to deploy to multiple servers and run active-active, sharing the same ActiveMQ queues. Each route attempts to read from the same queue to get a message to process. Only one route will get the message and therefore only one route processes it. Other routes are free to read subsequent messages, giving you simple load balancing. If one route crashes, the other routes will continue to process the messages, there will just be reduced capacity on your system.
If you need to provide fault tolerance for your web services then you need to look outside Camel and use something like Elastic Load Balancing. http://aws.amazon.com/elasticloadbalancing/
Aggregation
Each Camel context will run independently of the other contexts so one context will aggregate messages independently of what other contexts are up to. For example, suppose you have an aggregator that stores messages from ActiveMQ queue until receives a special end-of-batch message. If you have the aggregator running in two different routes, the messages will be split between the two routes and only one route will receive the end-of-batch message. So one aggregator will sit there with half the messages and do nothing. The other aggregator will have the other messages and will process the end-of-batch message but won't know about the messages the other route picked up.
I am new to the concepts of Camel and I would like someone to confirm my findings so far.
Please correct if wrong
The way I see it, Camel has Components, which can have several communication interfaces called Endpoints. A Component can communicate with another component by sending a message to on of its Endpoints, which then will use a Processor to deliver it to the destination. I also read about Routes but I am not sure what they are used for because they seem to overlap in function with Processors (aka, they seem to be the same thing?)
Camel Routes and Endpoints
http://java.dzone.com/articles/open-source-integration-apache
Please confirm if correct
So far I believe that a Component (C1) in machine A can use an Endpoint (E1) to communicate with another component (C2) using its endpoint (E2) in machine B, by using websockets:
http://camel.apache.org/how-do-i-configure-endpoints.html
I would be grateful if someone could enlighten me in the first question or confirm my second one. Also, is there a better way to communicate between machines that doesn't involve using webscokets?
Thanks in advance, Pedro.
A Route is what connects endpoints and processors together. When you define a route in Camel it starts with a Component that exposes a communication Endpoint, invokes processors that implement the business logic of the route, and then can output the Exchange (the message being processed) to another Endpoint that is exposed by another Component.
Certain components allow Camel routes to communicate with endpoints in different machines. For example, the JMS component can be used to communicate with other machines via JMS messages. However, some components are limited to communicating with endpoints within the same VM like the SEDA component.
Camel provides many different ways to send data between machines (Here is a list of a few of them off the top of my head):
AMQP
JMS
Amazon SQS
HTTP
Mina
Netty
Take a look at the Camel Components page for a complete list: http://camel.apache.org/components.html
First of all thanks to folks who are currently involved in the development of Camel, I am grateful for all the hard work they have put in.
I am looking for some design advice.
The architecture is something like this:
I have a bunch of Java classes which when instantiated are required to connect to each other and send messages using Apache Camel. The design constraints require me to create a framework such that all routing information, producers, consumers, endpoints etc should be a part of the camel-context.xml.
An individual should have the capability to modify such a file and completely change the existing route without having the Java code available to him.(The Java code would not be provided, only the compiled Jar would be)
For example in One setup,
Bean A ->Bean B->Bean C->file->email.
in another
Bean B->Bean A->Bean C->ftp->file->email
We have tried various approached, but if the originating bean is not implemented as a Java DSL, the messages rate is very high because camel constantly invokes Bean A in the first example and Bean B in the second(they being the source).
Bean A and Bean B originate messages and are event driven. In case the required event occurs, the beans send out a notification message.
My transformations are very simple and I do not require the power of Java DSL at all.
To summarize, I have the following questions:
1) Considering the above constraints, I do I ensure all routing information, including destination addresses, everything is a part of the camel context file?
2) Are there example I can look at for keeping the routing information completely independent of the java code?
3) How do I ensure Camel does not constantly invoke the originating bean?
4) Does Camel constantly invoke just the originating bean or any bean it sends & messages to irrespective of the position of the bean in the entire messaging queue?
I have run out of options trying various ways to set this up. Any help would be appreciated.
Read about hiding the middleware on the Camel wiki pages. This allows you to let clients use an interface to send/receive messages but totally unaware of Camel (no Camel API used at all).
Even better consider buying the Camel in Action book and read chapter 14 which talks about this.
http://www.manning.com/ibsen/
Save 41% on Manning books: Camel in Action or ActiveMQ in Action. Use code s2941. Expires 6th oct. http://www.manning.com/ibsen/
If you consider using ServiceMix of FuseESB, you might want to separate your routes in two parts.
First part would be the Event-driver bean that trigger the route. It could push messages to the ServiceNMR (see http://camel.apache.org/nmr.html).
The other part would be left to the framework users, using Spring DSL. It would just listen to message on the NMR (push by the other route) and do whatever they want with it.
Of course endpoint definition could be propertized using servicemix configuration service (see http://camel.apache.org/properties.html#Properties-UsingBlueprintpropertyplaceholderwithCamelroutes)
I am in the process of learning ActiveMQ and Camel, with the goal to create a little prototype system that works something like this:
(source: paulstovell.com)
(big)
When an order is placed in the Orders system, a message is sent out to any subscribers (a pub/sub system), and they can play their part in processing the order. The Orders, Shipping and Invoicing applications have their own ActiveMQ installations, so that if any of the three systems are offline, the others can continue to function. Something takes care of moving messages between the ActiveMQ installs.
Getting Apache Camel to move messages from one queue to another via routes is quite easy, if they are on the same ActiveMQ instance. So this works for managing the subscription queues.
The next challenge is pushing messages from one ActiveMQ instance to another, and it's the bit where I am not sure what to look at next.
Can Camel route between different ActiveMQ installations? (I can't figure out what the JMI endpoint URI would be if they are on different machines).
I understand ActiveMQ has store and forward capabilities. Is this what I would use to move messages between Orders and Shipping/Invoicing?
Or is this what Apache ServiceMix is meant to solve?
This is a pretty straightforward asynchronous, event-driven application that is well-suited for ActiveMQ and Camel.
Actually you do not move messages explicitly from one ActiveMQ instance to another. The way it works is using what's known as a network of brokers. In your case, you'd have three brokers: ActiveMQ-purple, ActiveMQ-green and ActiveMQ-blue. ActiveMQ-purple creates a uni-directional broker network with ActiveMQ-green and ActiveMQ-blue. This allows ActiveMQ-purple to store-and-forward messages to ActiveMQ-green and ActiveMQ-blue based on consumer demand.
The Orders app accepts orders on the orders queue on ActiveMQ-purple. The Orders app uses Camel to consume and process a message to determine if it is an invoicing message or a shipping message. Camel routes the messages to either the invoicing queue or the shipping queue on ActiveMQ-purple.
Consumer demand comes from the Invoicing app and the Shipping app. The Invoicing uses Camel to consume messages from the invoicing queue on ActiveMQ-green. The Shipping app uses Camel to consume messages from the shipping queue on ActiveMQ-blue. Because of the broker network and because of the consumer demand on the ActiveMQ-green.invoicing queue and the ActiveMQ-blue.shipping queue, messages will be forwarded from ActiveMQ-purple to the appropriate broker and queue. There is no need to explicitly route messages to specific broker.
I hope this answers your questions. Let me know if you have anymore.
Bruce
Hmmmm, I've only dabbled at best, and not for a fair while, but I'll try and offer something.
ActiveMQ can route between different installations and just uses standard URIs to my knowledge so I'm not sure what the problem is here. I would think that using TCP you'd be fine. Using ServiceMix (you mention it later) you'd just specify a connectionFactory & then provide the URI in that. This link shows some examples http://servicemix.apache.org/servicemix-jms-new-endpoints.html.
Camel has support for Durable Subscriber if that's what you were after (http://camel.apache.org/durable-subscriber.html)? This pattern will ensure that if the subscriber is offline when the message is ready, it will be held until the subscriber is back online. This is also supported by ServiceMix (see link given above and look for 'subscriptionDurable'.