I am new to Messaging and Integration. As I am trying to understand how Apache Camel makes EIP easy, Message Router caught my attention to a specific scenario where it consumes a message from one queue and transfers to some other queue based on a predicate.
Here are my questions:
Why use Message Router to perform this task?
Why not the message Producer directly send the message to the right Queue/Destination?
Why not the Consumer can rely on Message Selectors to do the same?
Can someone share your thoughts on real life use cases?
Well, your questions are more about SoC than messaging.
The Message Router is an EIP that can perform this task well and with Camel it is also very easy to implement, but as you write, there are other possibilities too.
If the message producer (client) sends messages directly to final destinations, it must know about all possible destinations. It must also be changed whenever a destination is added/removed or changed. This is tight coupling and in most situations not desirable.
If the message receiver(s) consume(s) with message selectors, the client can send all messages to the same queue. This is totally OK, you can implement individual consumers with different selectors. The selector feature of the Broker basically enables you to "divide a queue into multiple queues".
So if you want to implement and run an intermediate integration component between the client and the receivers, you can use the Message Router in this component.
If you want to omit this integration component, you can use message selectors to implement the receivers individual and de-coupled from each other.
Related
We are building a system where an external system sends us "Refresh" events. These events tell us to refresh the state of an item identified by the id provided in the RefreshEvent.
We handle these events through a Camel route.
The refresh may go wrong, though. If this is case the event is moved to a "Retry" SEDA queue with appropiate retry/backup setup. However, we may experience multiple "Refresh" event for the same item, while we are unable to fetch the new state.
Ideally we'd like to only have unique itemIds in the Retry queue. The inflight itemIds should be unique.
Is there a way to do this in Camel.
I have looked at the IdempotentConsumer, but that has a scope of all received exchanges, I'm only interested in the current inflight exchanges.
If you are not able to use the IndempotentConsumer pattern, you will have to implement the logic yourself and possible change the Seda endpoint to something more sophisticated.
You could use a message queue supporting JMS selectors or property filtering.
ActiveMQ, to name one, has some capabilities that could be used in implementing the solution you need.
We're having some came routes defined in a single CamelContext which contains Web services,activemq.. in the Route.
Initially we've deployed the Routes as WAR in single Jboss node.
To scale out(usually we're doing for web services) , I've deployed the same CamelContext in multiple Jboss nodes.
But the performance is actually decreased.
FYI: All the CamelContexts points to the Same activemq brokers.
Here are my questions:
How to load balance/ Fail over camel context in different machines?
If CamelContexts are deployed in multiple nodes, Will aggregation work correctly?
Kindly give your thoughts!
Without seeing your system in detail, there is no way of knowing why it has slowed down so I'll pass over that. For your other two questions:
Failover
You don't say what sort of failover/load balancing behaviour you want. The not-very-helpful Camel documentation is here: http://camel.apache.org/clustering-and-loadbalancing.html.
One mechanism that works easily with Camel and ActiveMQ is to deploy to multiple servers and run active-active, sharing the same ActiveMQ queues. Each route attempts to read from the same queue to get a message to process. Only one route will get the message and therefore only one route processes it. Other routes are free to read subsequent messages, giving you simple load balancing. If one route crashes, the other routes will continue to process the messages, there will just be reduced capacity on your system.
If you need to provide fault tolerance for your web services then you need to look outside Camel and use something like Elastic Load Balancing. http://aws.amazon.com/elasticloadbalancing/
Aggregation
Each Camel context will run independently of the other contexts so one context will aggregate messages independently of what other contexts are up to. For example, suppose you have an aggregator that stores messages from ActiveMQ queue until receives a special end-of-batch message. If you have the aggregator running in two different routes, the messages will be split between the two routes and only one route will receive the end-of-batch message. So one aggregator will sit there with half the messages and do nothing. The other aggregator will have the other messages and will process the end-of-batch message but won't know about the messages the other route picked up.
I am new to the concepts of Camel and I would like someone to confirm my findings so far.
Please correct if wrong
The way I see it, Camel has Components, which can have several communication interfaces called Endpoints. A Component can communicate with another component by sending a message to on of its Endpoints, which then will use a Processor to deliver it to the destination. I also read about Routes but I am not sure what they are used for because they seem to overlap in function with Processors (aka, they seem to be the same thing?)
Camel Routes and Endpoints
http://java.dzone.com/articles/open-source-integration-apache
Please confirm if correct
So far I believe that a Component (C1) in machine A can use an Endpoint (E1) to communicate with another component (C2) using its endpoint (E2) in machine B, by using websockets:
http://camel.apache.org/how-do-i-configure-endpoints.html
I would be grateful if someone could enlighten me in the first question or confirm my second one. Also, is there a better way to communicate between machines that doesn't involve using webscokets?
Thanks in advance, Pedro.
A Route is what connects endpoints and processors together. When you define a route in Camel it starts with a Component that exposes a communication Endpoint, invokes processors that implement the business logic of the route, and then can output the Exchange (the message being processed) to another Endpoint that is exposed by another Component.
Certain components allow Camel routes to communicate with endpoints in different machines. For example, the JMS component can be used to communicate with other machines via JMS messages. However, some components are limited to communicating with endpoints within the same VM like the SEDA component.
Camel provides many different ways to send data between machines (Here is a list of a few of them off the top of my head):
AMQP
JMS
Amazon SQS
HTTP
Mina
Netty
Take a look at the Camel Components page for a complete list: http://camel.apache.org/components.html
Does Camel provide anything out of the box which tells if it is able to connect all endpoints?
These endpoints could be MQ, webservice etc.
If not then I have to write a servlet which will send test request to all the endpoints. I will be using multicast or splitter for this implementation.
From my experience Camel will only provide warning logs if a from() endpoint is not available since it is constantly trying to read from them. Every other endpoint won't be accessed until the exchange tries to use that endpoint. If your goal is to test if various resources are alive I believe you would need to create your own testing program. I don't think this will be implemented as a feature because typically applications build in error handling if a resource is down and definte appropriate behaviors.
If we're talking about producers, then no. If your route is sending messages to an amq or http4 endpoint for instance, camel with not automatically send TCP-packets on these connections for monitoring purposes. A common way to handle failure of external endpoints is by using "circuit breakers". Take a look at https://camel.apache.org/load-balancer.html. A more robust alternative, imho, is Netflix's Hystrix.
If you have a polling consumer, say a from:ftp://.. then the polling consumer will poll messages every n-th millisecond, and you'll get an error if the connection is broken.
I am in the process of learning ActiveMQ and Camel, with the goal to create a little prototype system that works something like this:
(source: paulstovell.com)
(big)
When an order is placed in the Orders system, a message is sent out to any subscribers (a pub/sub system), and they can play their part in processing the order. The Orders, Shipping and Invoicing applications have their own ActiveMQ installations, so that if any of the three systems are offline, the others can continue to function. Something takes care of moving messages between the ActiveMQ installs.
Getting Apache Camel to move messages from one queue to another via routes is quite easy, if they are on the same ActiveMQ instance. So this works for managing the subscription queues.
The next challenge is pushing messages from one ActiveMQ instance to another, and it's the bit where I am not sure what to look at next.
Can Camel route between different ActiveMQ installations? (I can't figure out what the JMI endpoint URI would be if they are on different machines).
I understand ActiveMQ has store and forward capabilities. Is this what I would use to move messages between Orders and Shipping/Invoicing?
Or is this what Apache ServiceMix is meant to solve?
This is a pretty straightforward asynchronous, event-driven application that is well-suited for ActiveMQ and Camel.
Actually you do not move messages explicitly from one ActiveMQ instance to another. The way it works is using what's known as a network of brokers. In your case, you'd have three brokers: ActiveMQ-purple, ActiveMQ-green and ActiveMQ-blue. ActiveMQ-purple creates a uni-directional broker network with ActiveMQ-green and ActiveMQ-blue. This allows ActiveMQ-purple to store-and-forward messages to ActiveMQ-green and ActiveMQ-blue based on consumer demand.
The Orders app accepts orders on the orders queue on ActiveMQ-purple. The Orders app uses Camel to consume and process a message to determine if it is an invoicing message or a shipping message. Camel routes the messages to either the invoicing queue or the shipping queue on ActiveMQ-purple.
Consumer demand comes from the Invoicing app and the Shipping app. The Invoicing uses Camel to consume messages from the invoicing queue on ActiveMQ-green. The Shipping app uses Camel to consume messages from the shipping queue on ActiveMQ-blue. Because of the broker network and because of the consumer demand on the ActiveMQ-green.invoicing queue and the ActiveMQ-blue.shipping queue, messages will be forwarded from ActiveMQ-purple to the appropriate broker and queue. There is no need to explicitly route messages to specific broker.
I hope this answers your questions. Let me know if you have anymore.
Bruce
Hmmmm, I've only dabbled at best, and not for a fair while, but I'll try and offer something.
ActiveMQ can route between different installations and just uses standard URIs to my knowledge so I'm not sure what the problem is here. I would think that using TCP you'd be fine. Using ServiceMix (you mention it later) you'd just specify a connectionFactory & then provide the URI in that. This link shows some examples http://servicemix.apache.org/servicemix-jms-new-endpoints.html.
Camel has support for Durable Subscriber if that's what you were after (http://camel.apache.org/durable-subscriber.html)? This pattern will ensure that if the subscriber is offline when the message is ready, it will be held until the subscriber is back online. This is also supported by ServiceMix (see link given above and look for 'subscriptionDurable'.