My idea is to use camel to decouple modules. In order to support scalability and failover, I am wondering if the following architecture is adviced?
I have two applications with Camel embedded AppCamel1 and AppCamel2. Then I have standalone camel nodes Camel1 and Camel2.
AppCamel1 would have a route with fail-over/load balancing to Camel1 and Camel2. This way, if Camel1 crashes for example, Camel2 is used for failover.
Camel1 and 2 would do a REST call with the http component for example. Also there would be a request-reply from AppCamel1 up to camel1 or 2.
Is it a valid scenario?
What should I use to interconnect the different Camel instances (AppCamel1 to Camel1 or 2)? (I would like to know if it's possible to avoid another component like a jms server in the middle)
Thank you!
Edited following Boday's answer
the REST calls are from Camel1/2. I'd like to interconnect AppCamel1/2 to Camel1/2 and see if I can avoid anything in between. I guess mina is a possibility or even http but in that case a AppCamel1 and AppCamel2 need to know Camel1/2 which is not so good.
This is also being discussed at the Camel mailing list, where there is also some pointers and suggestions
http://camel.465427.n5.nabble.com/scalable-bus-with-multiple-Camel-instances-tp5606593p5606593.html
If you are trying to load balance HTTP requests to your AppCamel1/2, then you'd need a proxy server in between (apache mod_proxy, perlbal, etc). To load balance from AppCamel1/2 to Camel1/2, you can use Camel's load balancer or even JMS request/reply...
From AppCamel1/2 to Camel1/2, it sounds like you are using REST as the interface. If you need more complex communication between the instances, then I'd use JMS (via camel-activemq) for messaging and Hazelcast (via camel-hazelcast) for distributed caching/locking, etc.
If you use jms to communicate then you do not need a special load balancer. Just use one queue and let both Camel1/2 listen to the queue. Then they will automatically failover and load balance.
I would definetly go for a jms middleware. Activemq is the natural choice (camel is even considered a sub project of activemq). It is trivial to embedd amq along with your canel instances and cluster them. Activemq will then be able to handle both load balancing and failover for you.
Related
I am new to the concepts of Camel and I would like someone to confirm my findings so far.
Please correct if wrong
The way I see it, Camel has Components, which can have several communication interfaces called Endpoints. A Component can communicate with another component by sending a message to on of its Endpoints, which then will use a Processor to deliver it to the destination. I also read about Routes but I am not sure what they are used for because they seem to overlap in function with Processors (aka, they seem to be the same thing?)
Camel Routes and Endpoints
http://java.dzone.com/articles/open-source-integration-apache
Please confirm if correct
So far I believe that a Component (C1) in machine A can use an Endpoint (E1) to communicate with another component (C2) using its endpoint (E2) in machine B, by using websockets:
http://camel.apache.org/how-do-i-configure-endpoints.html
I would be grateful if someone could enlighten me in the first question or confirm my second one. Also, is there a better way to communicate between machines that doesn't involve using webscokets?
Thanks in advance, Pedro.
A Route is what connects endpoints and processors together. When you define a route in Camel it starts with a Component that exposes a communication Endpoint, invokes processors that implement the business logic of the route, and then can output the Exchange (the message being processed) to another Endpoint that is exposed by another Component.
Certain components allow Camel routes to communicate with endpoints in different machines. For example, the JMS component can be used to communicate with other machines via JMS messages. However, some components are limited to communicating with endpoints within the same VM like the SEDA component.
Camel provides many different ways to send data between machines (Here is a list of a few of them off the top of my head):
AMQP
JMS
Amazon SQS
HTTP
Mina
Netty
Take a look at the Camel Components page for a complete list: http://camel.apache.org/components.html
Does Camel provide anything out of the box which tells if it is able to connect all endpoints?
These endpoints could be MQ, webservice etc.
If not then I have to write a servlet which will send test request to all the endpoints. I will be using multicast or splitter for this implementation.
From my experience Camel will only provide warning logs if a from() endpoint is not available since it is constantly trying to read from them. Every other endpoint won't be accessed until the exchange tries to use that endpoint. If your goal is to test if various resources are alive I believe you would need to create your own testing program. I don't think this will be implemented as a feature because typically applications build in error handling if a resource is down and definte appropriate behaviors.
If we're talking about producers, then no. If your route is sending messages to an amq or http4 endpoint for instance, camel with not automatically send TCP-packets on these connections for monitoring purposes. A common way to handle failure of external endpoints is by using "circuit breakers". Take a look at https://camel.apache.org/load-balancer.html. A more robust alternative, imho, is Netflix's Hystrix.
If you have a polling consumer, say a from:ftp://.. then the polling consumer will poll messages every n-th millisecond, and you'll get an error if the connection is broken.
I wanted to know if Apache Camel can be used as a load balancer for any HTTP web server.
I am thinking of Apache as I can add some customization to it.
Yes you can use camel for that.
Something like this might do it for you (in a route builder):
from("jetty://http://0.0.0.0:8080/my/path")
.loadBalance()
.roundRobin()
.to("http://server1:8080/my/path","http://server2:8080/my/path");
You can check out more load balancing options here: http://camel.apache.org/load-balancer.html
Since you want to load balance HTTP, then see this page as well, as you would need to configure the http endpoints to be bridged:
http://camel.apache.org/how-to-use-camel-as-a-http-proxy-between-a-client-and-server.html
And as well the matchOnUriPrefix=true, to match any requests coming in.
And if you use jetty on all the endpoints it can scale up, using non-blocking continuations.
Yeah of course you can use camel as a Load balancer. I have so far used it very successfully. Have a look at this discussion Load balancing using camel. This will be useful to get started. Have fun riding on Camel!
First of all thanks to folks who are currently involved in the development of Camel, I am grateful for all the hard work they have put in.
I am looking for some design advice.
The architecture is something like this:
I have a bunch of Java classes which when instantiated are required to connect to each other and send messages using Apache Camel. The design constraints require me to create a framework such that all routing information, producers, consumers, endpoints etc should be a part of the camel-context.xml.
An individual should have the capability to modify such a file and completely change the existing route without having the Java code available to him.(The Java code would not be provided, only the compiled Jar would be)
For example in One setup,
Bean A ->Bean B->Bean C->file->email.
in another
Bean B->Bean A->Bean C->ftp->file->email
We have tried various approached, but if the originating bean is not implemented as a Java DSL, the messages rate is very high because camel constantly invokes Bean A in the first example and Bean B in the second(they being the source).
Bean A and Bean B originate messages and are event driven. In case the required event occurs, the beans send out a notification message.
My transformations are very simple and I do not require the power of Java DSL at all.
To summarize, I have the following questions:
1) Considering the above constraints, I do I ensure all routing information, including destination addresses, everything is a part of the camel context file?
2) Are there example I can look at for keeping the routing information completely independent of the java code?
3) How do I ensure Camel does not constantly invoke the originating bean?
4) Does Camel constantly invoke just the originating bean or any bean it sends & messages to irrespective of the position of the bean in the entire messaging queue?
I have run out of options trying various ways to set this up. Any help would be appreciated.
Read about hiding the middleware on the Camel wiki pages. This allows you to let clients use an interface to send/receive messages but totally unaware of Camel (no Camel API used at all).
Even better consider buying the Camel in Action book and read chapter 14 which talks about this.
http://www.manning.com/ibsen/
Save 41% on Manning books: Camel in Action or ActiveMQ in Action. Use code s2941. Expires 6th oct. http://www.manning.com/ibsen/
If you consider using ServiceMix of FuseESB, you might want to separate your routes in two parts.
First part would be the Event-driver bean that trigger the route. It could push messages to the ServiceNMR (see http://camel.apache.org/nmr.html).
The other part would be left to the framework users, using Spring DSL. It would just listen to message on the NMR (push by the other route) and do whatever they want with it.
Of course endpoint definition could be propertized using servicemix configuration service (see http://camel.apache.org/properties.html#Properties-UsingBlueprintpropertyplaceholderwithCamelroutes)
I am in the process of learning ActiveMQ and Camel, with the goal to create a little prototype system that works something like this:
(source: paulstovell.com)
(big)
When an order is placed in the Orders system, a message is sent out to any subscribers (a pub/sub system), and they can play their part in processing the order. The Orders, Shipping and Invoicing applications have their own ActiveMQ installations, so that if any of the three systems are offline, the others can continue to function. Something takes care of moving messages between the ActiveMQ installs.
Getting Apache Camel to move messages from one queue to another via routes is quite easy, if they are on the same ActiveMQ instance. So this works for managing the subscription queues.
The next challenge is pushing messages from one ActiveMQ instance to another, and it's the bit where I am not sure what to look at next.
Can Camel route between different ActiveMQ installations? (I can't figure out what the JMI endpoint URI would be if they are on different machines).
I understand ActiveMQ has store and forward capabilities. Is this what I would use to move messages between Orders and Shipping/Invoicing?
Or is this what Apache ServiceMix is meant to solve?
This is a pretty straightforward asynchronous, event-driven application that is well-suited for ActiveMQ and Camel.
Actually you do not move messages explicitly from one ActiveMQ instance to another. The way it works is using what's known as a network of brokers. In your case, you'd have three brokers: ActiveMQ-purple, ActiveMQ-green and ActiveMQ-blue. ActiveMQ-purple creates a uni-directional broker network with ActiveMQ-green and ActiveMQ-blue. This allows ActiveMQ-purple to store-and-forward messages to ActiveMQ-green and ActiveMQ-blue based on consumer demand.
The Orders app accepts orders on the orders queue on ActiveMQ-purple. The Orders app uses Camel to consume and process a message to determine if it is an invoicing message or a shipping message. Camel routes the messages to either the invoicing queue or the shipping queue on ActiveMQ-purple.
Consumer demand comes from the Invoicing app and the Shipping app. The Invoicing uses Camel to consume messages from the invoicing queue on ActiveMQ-green. The Shipping app uses Camel to consume messages from the shipping queue on ActiveMQ-blue. Because of the broker network and because of the consumer demand on the ActiveMQ-green.invoicing queue and the ActiveMQ-blue.shipping queue, messages will be forwarded from ActiveMQ-purple to the appropriate broker and queue. There is no need to explicitly route messages to specific broker.
I hope this answers your questions. Let me know if you have anymore.
Bruce
Hmmmm, I've only dabbled at best, and not for a fair while, but I'll try and offer something.
ActiveMQ can route between different installations and just uses standard URIs to my knowledge so I'm not sure what the problem is here. I would think that using TCP you'd be fine. Using ServiceMix (you mention it later) you'd just specify a connectionFactory & then provide the URI in that. This link shows some examples http://servicemix.apache.org/servicemix-jms-new-endpoints.html.
Camel has support for Durable Subscriber if that's what you were after (http://camel.apache.org/durable-subscriber.html)? This pattern will ensure that if the subscriber is offline when the message is ready, it will be held until the subscriber is back online. This is also supported by ServiceMix (see link given above and look for 'subscriptionDurable'.