I am in the process of learning ActiveMQ and Camel, with the goal to create a little prototype system that works something like this:
(source: paulstovell.com)
(big)
When an order is placed in the Orders system, a message is sent out to any subscribers (a pub/sub system), and they can play their part in processing the order. The Orders, Shipping and Invoicing applications have their own ActiveMQ installations, so that if any of the three systems are offline, the others can continue to function. Something takes care of moving messages between the ActiveMQ installs.
Getting Apache Camel to move messages from one queue to another via routes is quite easy, if they are on the same ActiveMQ instance. So this works for managing the subscription queues.
The next challenge is pushing messages from one ActiveMQ instance to another, and it's the bit where I am not sure what to look at next.
Can Camel route between different ActiveMQ installations? (I can't figure out what the JMI endpoint URI would be if they are on different machines).
I understand ActiveMQ has store and forward capabilities. Is this what I would use to move messages between Orders and Shipping/Invoicing?
Or is this what Apache ServiceMix is meant to solve?
This is a pretty straightforward asynchronous, event-driven application that is well-suited for ActiveMQ and Camel.
Actually you do not move messages explicitly from one ActiveMQ instance to another. The way it works is using what's known as a network of brokers. In your case, you'd have three brokers: ActiveMQ-purple, ActiveMQ-green and ActiveMQ-blue. ActiveMQ-purple creates a uni-directional broker network with ActiveMQ-green and ActiveMQ-blue. This allows ActiveMQ-purple to store-and-forward messages to ActiveMQ-green and ActiveMQ-blue based on consumer demand.
The Orders app accepts orders on the orders queue on ActiveMQ-purple. The Orders app uses Camel to consume and process a message to determine if it is an invoicing message or a shipping message. Camel routes the messages to either the invoicing queue or the shipping queue on ActiveMQ-purple.
Consumer demand comes from the Invoicing app and the Shipping app. The Invoicing uses Camel to consume messages from the invoicing queue on ActiveMQ-green. The Shipping app uses Camel to consume messages from the shipping queue on ActiveMQ-blue. Because of the broker network and because of the consumer demand on the ActiveMQ-green.invoicing queue and the ActiveMQ-blue.shipping queue, messages will be forwarded from ActiveMQ-purple to the appropriate broker and queue. There is no need to explicitly route messages to specific broker.
I hope this answers your questions. Let me know if you have anymore.
Bruce
Hmmmm, I've only dabbled at best, and not for a fair while, but I'll try and offer something.
ActiveMQ can route between different installations and just uses standard URIs to my knowledge so I'm not sure what the problem is here. I would think that using TCP you'd be fine. Using ServiceMix (you mention it later) you'd just specify a connectionFactory & then provide the URI in that. This link shows some examples http://servicemix.apache.org/servicemix-jms-new-endpoints.html.
Camel has support for Durable Subscriber if that's what you were after (http://camel.apache.org/durable-subscriber.html)? This pattern will ensure that if the subscriber is offline when the message is ready, it will be held until the subscriber is back online. This is also supported by ServiceMix (see link given above and look for 'subscriptionDurable'.
Related
I have a micro-service which subscribes to a topic in WebSphere MQ. The subscription is managed and durable. I explicitly set the subscription name, so that it can be used to connect back to the queue, after recovering from any micro service failure. The subscription works as expected.
But I might have to scale up the micro service and run multiple instances. In this case I will end up with having multiple consumers to the same topic. But here it fails with error 2429 : MQRC_SUBSCRIPTION_IN_USE. I am not able to run more than one consumer to the topic subscription. Note : A message should be sent only to one of the consumers.
Any thought ?
IBM Websphere Version : 7.5
I use the C-client API to connect to the MQ.
When using a subscriber what you describe is only supported via the IBM MQ Classes for JMS API. In v7.0 and later you can use Cloned subscriptions (this is a IBM extension to the JMS spec), in addition in MQ v8.0 and later you can alternately use Shared subscriptions which is part of the JMS 2.0 spec. With these two options, multiple subscribers can be connected to the same subscription and only one of them will receive each published message.
UPDATE 20170710
According to this APAR IV96489: XMS.NET DOESN'T ALLOW SHARED SUBSCRIPTIONS EVEN WHEN CLONESUP PROPERTY IS ENABLED, XMS.NET is also supposed to support Cloned subscriptions but due to a defect this is will not be supported until 8.0.0.8 or 9.0.0.2 or if you request the IFIX for the APAR above.
You can accomplish something similar with other APIs like C by converting your micro-service to get from a queue instead of subscribing to a topic.
To get the published messages to the queue you have two options:
Setup a administrative subscription on the queue manager. You can do this a few different ways. The example below would be using a MQSC command.
DEFINE SUB('XYZ') TOPICSTR('SOME/TOPIC') DEST(SOME.QUEUE)
Create a utility app that can open a queue and create a durable subscription with that provided queue, the only purpose of this app would be to subscribe and unsubscribe a provided queue, it would not be used to consume any of the published messages.
Using the above method, each published message can only be read (GET) from the queue by one process or thread.
The MuleSoft HTTP ListenerConnector transforms an HTTP request into a Mule Message. It is my understanding that, as a consequence, the message is stored in a queue - therefore it can be guaranteed that the message will be delivered (once the request reached the integration platform).
In contrast, apache camel does not come with a messaging system. Therefore, I assume, the message is not stored anywhere and is lost when the destination of the message is temporarily not available.
Is that right? When using camel together with some messaging system, can the mule behavior be implemented easily?
Yes, that is true Camel does not come with a messaging or runtime system. You need to do decide on those yourself. I have used Camel with Karaf and messaging system such as ActiveMQ and RabbitMQ. You can easily ensure delivery use both of those together with the correct configuration. Both support persistent messaging, persistent queues and redelivery. Off course the main to do decide is, what happens if the host running your eventbus dies. In that case you need to consider clustering. The same applies for Mule.
We're having some came routes defined in a single CamelContext which contains Web services,activemq.. in the Route.
Initially we've deployed the Routes as WAR in single Jboss node.
To scale out(usually we're doing for web services) , I've deployed the same CamelContext in multiple Jboss nodes.
But the performance is actually decreased.
FYI: All the CamelContexts points to the Same activemq brokers.
Here are my questions:
How to load balance/ Fail over camel context in different machines?
If CamelContexts are deployed in multiple nodes, Will aggregation work correctly?
Kindly give your thoughts!
Without seeing your system in detail, there is no way of knowing why it has slowed down so I'll pass over that. For your other two questions:
Failover
You don't say what sort of failover/load balancing behaviour you want. The not-very-helpful Camel documentation is here: http://camel.apache.org/clustering-and-loadbalancing.html.
One mechanism that works easily with Camel and ActiveMQ is to deploy to multiple servers and run active-active, sharing the same ActiveMQ queues. Each route attempts to read from the same queue to get a message to process. Only one route will get the message and therefore only one route processes it. Other routes are free to read subsequent messages, giving you simple load balancing. If one route crashes, the other routes will continue to process the messages, there will just be reduced capacity on your system.
If you need to provide fault tolerance for your web services then you need to look outside Camel and use something like Elastic Load Balancing. http://aws.amazon.com/elasticloadbalancing/
Aggregation
Each Camel context will run independently of the other contexts so one context will aggregate messages independently of what other contexts are up to. For example, suppose you have an aggregator that stores messages from ActiveMQ queue until receives a special end-of-batch message. If you have the aggregator running in two different routes, the messages will be split between the two routes and only one route will receive the end-of-batch message. So one aggregator will sit there with half the messages and do nothing. The other aggregator will have the other messages and will process the end-of-batch message but won't know about the messages the other route picked up.
Does Camel provide anything out of the box which tells if it is able to connect all endpoints?
These endpoints could be MQ, webservice etc.
If not then I have to write a servlet which will send test request to all the endpoints. I will be using multicast or splitter for this implementation.
From my experience Camel will only provide warning logs if a from() endpoint is not available since it is constantly trying to read from them. Every other endpoint won't be accessed until the exchange tries to use that endpoint. If your goal is to test if various resources are alive I believe you would need to create your own testing program. I don't think this will be implemented as a feature because typically applications build in error handling if a resource is down and definte appropriate behaviors.
If we're talking about producers, then no. If your route is sending messages to an amq or http4 endpoint for instance, camel with not automatically send TCP-packets on these connections for monitoring purposes. A common way to handle failure of external endpoints is by using "circuit breakers". Take a look at https://camel.apache.org/load-balancer.html. A more robust alternative, imho, is Netflix's Hystrix.
If you have a polling consumer, say a from:ftp://.. then the polling consumer will poll messages every n-th millisecond, and you'll get an error if the connection is broken.
My idea is to use camel to decouple modules. In order to support scalability and failover, I am wondering if the following architecture is adviced?
I have two applications with Camel embedded AppCamel1 and AppCamel2. Then I have standalone camel nodes Camel1 and Camel2.
AppCamel1 would have a route with fail-over/load balancing to Camel1 and Camel2. This way, if Camel1 crashes for example, Camel2 is used for failover.
Camel1 and 2 would do a REST call with the http component for example. Also there would be a request-reply from AppCamel1 up to camel1 or 2.
Is it a valid scenario?
What should I use to interconnect the different Camel instances (AppCamel1 to Camel1 or 2)? (I would like to know if it's possible to avoid another component like a jms server in the middle)
Thank you!
Edited following Boday's answer
the REST calls are from Camel1/2. I'd like to interconnect AppCamel1/2 to Camel1/2 and see if I can avoid anything in between. I guess mina is a possibility or even http but in that case a AppCamel1 and AppCamel2 need to know Camel1/2 which is not so good.
This is also being discussed at the Camel mailing list, where there is also some pointers and suggestions
http://camel.465427.n5.nabble.com/scalable-bus-with-multiple-Camel-instances-tp5606593p5606593.html
If you are trying to load balance HTTP requests to your AppCamel1/2, then you'd need a proxy server in between (apache mod_proxy, perlbal, etc). To load balance from AppCamel1/2 to Camel1/2, you can use Camel's load balancer or even JMS request/reply...
From AppCamel1/2 to Camel1/2, it sounds like you are using REST as the interface. If you need more complex communication between the instances, then I'd use JMS (via camel-activemq) for messaging and Hazelcast (via camel-hazelcast) for distributed caching/locking, etc.
If you use jms to communicate then you do not need a special load balancer. Just use one queue and let both Camel1/2 listen to the queue. Then they will automatically failover and load balance.
I would definetly go for a jms middleware. Activemq is the natural choice (camel is even considered a sub project of activemq). It is trivial to embedd amq along with your canel instances and cluster them. Activemq will then be able to handle both load balancing and failover for you.