We have multiple consumers connecting to an ActiveMQ broker. When ActiveMQ is shut down or dies the consumers also die. Is there a way to keep them running and to automatically reconnect if ActiveMQ is restarted?
It would also need to work with a failover.
Cheers.
ActiveMQ has failover support with the failover transport.
Refer: http://activemq.apache.org/failover-transport-reference.html
Here's a good article that explains the same.
http://bsnyderblog.blogspot.co.uk/2010/01/how-to-use-automatic-failover-in.html
You should use a durable subscription for this:
from(activemq:queue:Q.FOO?durableSubscriptionName=TopicReader1&subscriptionDurable=true")
.to("....");
Related
If I have a Camel JMS consumer, with
maxConcurrentConsumers=10
reading from MQ with max pool size =10 on
the connection), and
disableReplyTo=true
Q1. Can increasing the maxConcurrentConsumers help scale the route? Once the message is read from the queue, is the connection relinquished?
Q2. Can placing a SEDA producer-consumer pattern right after consuming the message help with scaling? Or, is it that one might as well simply increase the maxConcurrentConsumers on the JMS consumer?
Its generally better to let the scaling be from the JMS consumer, and then scale horizontally by adding more nodes.
The SEDA is an in-memory queue in the JVM and even if you can "scale" by consuming quickly from the JMS queue to the SEDA queue, then you just move the messages from a "safe" storage in the broker to a more "unsafe" store in a JVM in-memory storage.
The JMS broker is built for scaling, and has several architecture styles and topologies for setting up a broker system according to your needs. So its better to leverage it.
The JMS component has options to set concurrency you can tweak as well. And on the JMS client/broker side as well. For example ActiveMQ has prefetch size and others that can be tweaked.
With my commercial hat on: If you are a Fuse subscriber then we have guides for scaling Fuse/AMQ you can also read, and get assistance from our team.
If several instance of a same microservice contain their own database,for scalability, how update all the databases when a create, update or delete operation is made ? which tool compatible with Eureka and Zuul spring propose for that ?
I would suggest you to use RabbitMQ
The basic architecture of a message queue is simple, there are client applications called producers that create messages and deliver them to the broker (the message queue). Other applications, called consumers, connects to the queue and subscribes to the messages to be processed. A software can be a producer, or consumer, or both a consumer and a producer of messages. Messages placed onto the queue are stored until the consumer retrieves them.
Why to use this RabbitMQ??
https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html
Official document for rabbitMQ....
https://www.rabbitmq.com/
How to install rabbitMQ:
https://www.journaldev.com/11655/spring-rabbitmq
Configuration in spring boot application:
https://spring.io/guides/gs/messaging-rabbitmq/
I would suggest you use an Event-based architecture where any service has done his work it produces the event and other services subscribe that event will also start his work.
you can use Kafka queue for same. also, read Distributed Sagas for Microservices
One more thing is that inter-communication use UDP instead of TCP.
Most of the databases offer replication these days with near 0 latency. Unless you use the other databases you can let the database do the synchronization for you.
I'm trying to create a router to integrate a number of JMS topics & Queues. I am constrained by the fact the client I am working for can't change the JMS implementation (TibCo EMS with some custom client libraries) and the fact that they have written their own XA transaction manager which doesn't quite conform with the JTA spec. It is very important that message delivery is guaranteed.
I've done a lot of reading and experimenting with Camel and I've realised that I probably need to write my own JMS component, as the standard JMS component is not going to integrate with the JMS client libraries or TM I have.
I need to be able to put hooks into the route lifecycle at the following points:
During the route startup, I need to identify all JMS connections and enlist them as XA resources with the TM implementation
When a message is received at the consumer, I need to start a transaction including all the JMS connections in the route
When a routing decision is made, I need to send the message to the producer and commit the transaction
Given the above, I think I can implement a very simplified version of the camel-jms component which strips out all the Spring parts and only contains the bare minimum required to interact with my JMS libraries.
Where would be the best place to initialise the transaction manager? I've been looking at DefaultCamelContext, RoutePolicy and RouteContext but I can't find a place where all the endpoints are resolved and initialised.
I solved this problem by implementing the UserTransaction and TransactionManager interfaces and creating a PlatformTransactionManager which the Camel JMS component uses to create the DefaultMessageListenerContainer.
One important point to note is that the transacted property on the Camel JMSComponent refers to local transactions, not XA transactions. If you set this property to true after passing a PlatformTransactionManager to the component, the DMLC will effectively try to commit your transaction twice, which won't work.
This leaves me with a nice working example consuming from one JMS broker and producing to another, but it is very slow - ~5 messages per second. Unfortunately Spring JMS does not support batching so it seems the best solution here is to adjust the JMS topic configurations such that routing only takes place between topics on the same broker.
I'm using akka + camel to consume message from activemq, and I'm trying to figure out how to deploy this consumer in multiple machines without duplicate the message. In this case I'm consuming message from a topic and the activemq should know I have one akka system in various machines, instead of various single independent systems.
I tried to accomplish that using akka cluster, but that example using a frontend that subscribe to a cluster of backend does not help since my "backend" actor is the activemq consumer itself and I can't tell activemq to subscribe to my cluster.
Any ideas?
JMS versions < 2.0 does not allow multiple nodes to share a topic subscription (not duplicating the message to each consumer). To cope with that ActiveMQ provides Virtual Topic (you can consume messages published to a topic from a Queue which allows for multiple consumers - load balancing).
It's all naming conventions. So you simply publish to the topic VirtualTopic.Orders and then consume from the queue Consumer.ClusterX.VirtualTopic.Orders. Naming conventions could be changed - see docs.
http://activemq.apache.org/virtual-destinations.html
I am looking to use apache-camel to poll an imap inbox, but I am wondering how this setup would behave in a cluster. I would deploy apache camel on each node of the cluster, and each node would poll the inbox.
How can I avoid having many consumers pick up the same message?
I decided to take the simple road and not install additional components. I used a clustered quartz job to trigger the polling of the inbox. The poller then places a retrieval command on a Hazelcast distributed queue, which is received by an array of Message retrieval components in the cluster.
Installing, Jms, James, in addition to Camel smelled to me, just to solve this task.
Not very easy, since imap is not really a protocol for these kind of tasks.
The trick is still to have one consumer do the polling, not many. If you have many nodes for high availablility, you could do some tricks with JMS to trigger IMAP polls.
For instance, you could use a jms trigger message to init a poll and have all members of the cluster listen to that poll. Keep the concurrentConsumer to 1 and async. JMS disabled in Camel. You can rely on Message Groups or ActiveMQ exclusive consumer to be sure that only one node gets the trigger messages (when alive, otherwise another node will take over). Generating the polling messages might be tricky, but could be done as simply as a timer route from each camel node. Just tune the frequency.
This setup will avoid race conditions in IMAP, while not beeing load balanced, at least fail over secured. It might be good enough to just go ahead and do concurrent polling, with few issues. However, I don't think you will be 100% safe without only allowing one consumer.
In a clustered environment you may consider having a way of electing a single Camel route that is active, which does the imap polling. And then have logic for failover if the node goes down.
In Camel you can take a look at route policy which can be applied to routes.
http://camel.apache.org/routepolicy
The zookeeper component has a policy for electing a leader in a cluster, and only allow one route to be active. This requires though that you use zookeeper.
http://camel.apache.org/zookeeper