If we have to send a message to multiple end points, in Camel there is a multicast EIP. What if the end points are ONLY JMS endpoints? In that case, I feel that topic is better that multicast EIP. Or is there any specific behaviour difference or pattern which differentiates these two approaches if the end points are JMS endpoints? Please clarify, thanks in advance!
If we have to send a message to multiple JMS end points, then we need to use Topic instead of multicast as it will be a better and less time consuming approach. Send a message in a topic through which consumers receive message and do different process.
Related
I have written code (mosquitto_publish()) using Mosquitto to publish data to AWS.
My problem is the sequence with which data is arriving on the MQTT broker. In the Paho client, I see waitForCompletion(), but nothing similar in Mosquitto. Would anyone please help me in dealing with this problem ?
Based on the mosquitto_publich documentation, the function returns when sending has been "successful". MQTT does not guarantee the order in which messages arrive, so you should arguably watch for the arrival rather than the sending, and avoid having two messages race each other to the broker. With QoS 0, the client never knows if a message arrived; that requires QoS 1 or 2, for which additional communications are exchanged. Raise the quality of service, and you can use mosquitto_max_inflight_messages_set(mosq, 1) so that the client queues any additional messages until it receives confirmation from the server. This may be even more efficient than "waiting" for completion, since non-MQTT operations can continue. The queue might pile up if you send bursts of many messages.
The more complex alternative is to send messages unrestricted, but include an index with each, so that the subscriber can sort them upon receipt (for which it would need its own queue and delay). Not recommended if this burden is going to fall on multiple subscribers.
If I have a Camel JMS consumer, with
maxConcurrentConsumers=10
reading from MQ with max pool size =10 on
the connection), and
disableReplyTo=true
Q1. Can increasing the maxConcurrentConsumers help scale the route? Once the message is read from the queue, is the connection relinquished?
Q2. Can placing a SEDA producer-consumer pattern right after consuming the message help with scaling? Or, is it that one might as well simply increase the maxConcurrentConsumers on the JMS consumer?
Its generally better to let the scaling be from the JMS consumer, and then scale horizontally by adding more nodes.
The SEDA is an in-memory queue in the JVM and even if you can "scale" by consuming quickly from the JMS queue to the SEDA queue, then you just move the messages from a "safe" storage in the broker to a more "unsafe" store in a JVM in-memory storage.
The JMS broker is built for scaling, and has several architecture styles and topologies for setting up a broker system according to your needs. So its better to leverage it.
The JMS component has options to set concurrency you can tweak as well. And on the JMS client/broker side as well. For example ActiveMQ has prefetch size and others that can be tweaked.
With my commercial hat on: If you are a Fuse subscriber then we have guides for scaling Fuse/AMQ you can also read, and get assistance from our team.
We have created a thing using AWS-IoT service. We have created a topic in that particular thing. Subscriber has subscribed to that topic and publisher is sending messages to that topic.
Below is the publisher messaging order:
message 0
message 1
message 2
message 3
message 4
At the subscriber end the sequence of messages is not maintained. It's showing like this:
message 0
message 1
message 4
message 2
message 3
True, in AWS IoT, the message broker does not guarantee order while they deliver messages to the devices.
The reason being that in a typical distributed systems architecture, a single message from the publisher to the subscriber shall take multiple paths to ensure that the system is highly available and scalable. In the case of AWS IoT, the Device Gateway supports the publisher subscriber messaging pattern and enables scalable, low-latency, and low-overhead communication.
However, based on the type of use case, there are many possible solutions that can be worked out. There should be a logic such that the publishers themselves shall do the co-ordination. One generic or simple approach could be that a sequence number addition at the device side should be sufficient to handle the ordering of the messages between publisher and subscriber. On the receiver, a logic to process or discard based on checking of the ordering based on sequence number should be helpful.
As written in the documentation of AWS
The message broker does not guarantee the order in which messages and
ACK are received.
I guess its too late to answer to this question but I'll still go ahead so others facing this issue can have a work around. I faced a similar scenario and I did the following to make sure that the order is maintained.
I added sequence ID or timestamp to the payload sent to the broker from my iot device (can be any kind of client)
I then configured the IoT rules engine (add actions) to send the messages directly to DynamoDB where the data was automatically stored in a sorted manner (needs to be configured to sort by seqID).
Then I used Lambda to pull out the data from DynamoDB for my further workflow but you can use whatever service according to yours.
I'm trying to create a router to integrate a number of JMS topics & Queues. I am constrained by the fact the client I am working for can't change the JMS implementation (TibCo EMS with some custom client libraries) and the fact that they have written their own XA transaction manager which doesn't quite conform with the JTA spec. It is very important that message delivery is guaranteed.
I've done a lot of reading and experimenting with Camel and I've realised that I probably need to write my own JMS component, as the standard JMS component is not going to integrate with the JMS client libraries or TM I have.
I need to be able to put hooks into the route lifecycle at the following points:
During the route startup, I need to identify all JMS connections and enlist them as XA resources with the TM implementation
When a message is received at the consumer, I need to start a transaction including all the JMS connections in the route
When a routing decision is made, I need to send the message to the producer and commit the transaction
Given the above, I think I can implement a very simplified version of the camel-jms component which strips out all the Spring parts and only contains the bare minimum required to interact with my JMS libraries.
Where would be the best place to initialise the transaction manager? I've been looking at DefaultCamelContext, RoutePolicy and RouteContext but I can't find a place where all the endpoints are resolved and initialised.
I solved this problem by implementing the UserTransaction and TransactionManager interfaces and creating a PlatformTransactionManager which the Camel JMS component uses to create the DefaultMessageListenerContainer.
One important point to note is that the transacted property on the Camel JMSComponent refers to local transactions, not XA transactions. If you set this property to true after passing a PlatformTransactionManager to the component, the DMLC will effectively try to commit your transaction twice, which won't work.
This leaves me with a nice working example consuming from one JMS broker and producing to another, but it is very slow - ~5 messages per second. Unfortunately Spring JMS does not support batching so it seems the best solution here is to adjust the JMS topic configurations such that routing only takes place between topics on the same broker.
I'm using akka + camel to consume message from activemq, and I'm trying to figure out how to deploy this consumer in multiple machines without duplicate the message. In this case I'm consuming message from a topic and the activemq should know I have one akka system in various machines, instead of various single independent systems.
I tried to accomplish that using akka cluster, but that example using a frontend that subscribe to a cluster of backend does not help since my "backend" actor is the activemq consumer itself and I can't tell activemq to subscribe to my cluster.
Any ideas?
JMS versions < 2.0 does not allow multiple nodes to share a topic subscription (not duplicating the message to each consumer). To cope with that ActiveMQ provides Virtual Topic (you can consume messages published to a topic from a Queue which allows for multiple consumers - load balancing).
It's all naming conventions. So you simply publish to the topic VirtualTopic.Orders and then consume from the queue Consumer.ClusterX.VirtualTopic.Orders. Naming conventions could be changed - see docs.
http://activemq.apache.org/virtual-destinations.html