How to get retry count from ActiveMQ Artemis message - apache-camel

I'm taking the message from an ActiveMQ Artemis queue and trying to send it to an invalid HTTP endpoint (through a Camel route) it fails and the message gets retried. Whenever message delivery fails it gets retried at ActiveMQ Artemis level. Retry configuration is available in broker. Is there any way to get current retry count for a JMS message in client applications?
I can maintain retry count in DB per message - looking for better solution / existing methods etc.
If I try to maintain the count and set it in the message header then during retry those get lost. Retry happens as if it's a new message. So can't use it.
We are using org.apache.camel.component.jms.JmsComponent with the Qpid JMS client.

You should inspect the JMSXDeliveryCount property on the message. Section 3.5.9 of the JMS 1.1 specification says this about JMSXDeliveryCount:
The number of message delivery attempts; the first is 1, the second 2,...
I just tested this with Qpid JMS 1.7.0 and ActiveMQ Artemis 2.27.1 and it worked fine in both transacted and non-transacted use-cases. In both cases the JMSRedelivered header and the JMSXDeliveryCount property were set appropriately.
Here's the code for the transacted use-case:
try (Connection connection = new JmsConnectionFactory("amqp://127.0.0.1:5672").createConnection()) {
Session session = connection.createSession(true, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue("myQueue");
MessageConsumer consumer = session.createConsumer(queue);
MessageProducer producer = session.createProducer(queue);
TextMessage message = session.createTextMessage("test-message");
producer.send(message);
session.commit();
producer.close();
connection.start();
message = (TextMessage) consumer.receive(1000);
System.out.println("Redelivered? " + message.getJMSRedelivered());
System.out.println("Delivery count: " + message.getIntProperty("JMSXDeliveryCount"));
message.acknowledge();
session.rollback();
message = (TextMessage) consumer.receive(1000);
System.out.println("Redelivered? " + message.getJMSRedelivered());
System.out.println("Delivery count: " + message.getIntProperty("JMSXDeliveryCount"));
}
Here's the code for the non-transacted use-case:
try (Connection connection = new JmsConnectionFactory("amqp://127.0.0.1:5672").createConnection()) {
Session session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
Queue queue = session.createQueue("myQueue");
MessageConsumer consumer = session.createConsumer(queue);
MessageProducer producer = session.createProducer(queue);
TextMessage message = session.createTextMessage("test-message");
producer.send(message);
producer.close();
connection.start();
message = (TextMessage) consumer.receive(1000);
System.out.println("Redelivered? " + message.getJMSRedelivered());
System.out.println("Delivery count: " + message.getIntProperty("JMSXDeliveryCount"));
session.close();
session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
consumer = session.createConsumer(queue);
message = (TextMessage) consumer.receive(1000);
System.out.println("Redelivered? " + message.getJMSRedelivered());
System.out.println("Delivery count: " + message.getIntProperty("JMSXDeliveryCount"));
}
In both cases this is what was printed:
Redelivered? false
Delivery count: 1
Redelivered? true
Delivery count: 2
The behavior you're seeing with the message being redelivered as "a new message" is expected. As noted in section 3.10 of the JMS 1.1 specification:
A consumer can modify a received message after calling either the clearBody or clearProperties method to make the body or properties writable. If the consumer modifies a received message, and the message is subsequently redelivered, the redelivered message must be the original, unmodified message (except for headers and properties modified by the JMS provider as a result of the redelivery, such as the JMSRedelivered header and the JMSXDeliveryCount property).
I'm not a Camel expert so I can't say exactly is Camel doing with the JMS message when delivery to the HTTP endpoint fails. All I know is that this works fine with straight JMS. I would expect this to work with Camel also, but Camel may be doing their own form of retry rather than using the redelivery which JMS provides.
This would be relatively trivial to validate by turning on AMQP frame tracing from the Qpid JMS client and inspecting the frames around message delivery and the resulting disposition.

Related

PAHO MQTT 5 throwing exception when using same clientId in routes

When using paho-mqtt5:test more than once with same clientId then it throw exception Client not connected but if i will use different clientId for each to and from then it will work fine
2021-10-05 19:25:28,650 ERROR [org.apa.cam.pro.err.DefaultErrorHandler] (Camel (camel-1) thread #0 - timer://test) Failed delivery for (MessageId: 871E4623819E4FB-000000000000001B on ExchangeId: 871E4623819E4FB-000000000000001B). Exhausted after delivery attempt: 1 caught: Client is not connected (32104)
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route1 ] [route1 ] [from[timer://test?period=1000] ] [ 0]
...
[route1 ] [to1 ] [paho:test ] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
: Client is not connected (32104)
at org.eclipse.paho.mqttv5.client.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:32)
at org.eclipse.paho.mqttv5.client.internal.ClientComms.sendNoWait(ClientComms.java:231)
at org.eclipse.paho.mqttv5.client.MqttAsyncClient.publish(MqttAsyncClient.java:1530)
at org.eclipse.paho.mqttv5.client.MqttClient.publish(MqttClient.java:564)
at org.apache.camel.component.paho.mqtt5.PahoMqtt5Producer.process(PahoMqtt5Producer.java:55)
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:172)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:463)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:179)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:184)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398)
at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:210)
at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:76)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)
Here is my code which is throwing exception
#ApplicationScoped
class TestRouter : RouteBuilder() {
override fun configure() {
val mqtt5Component = PahoMqtt5Component()
mqtt5Component.configuration = PahoMqtt5Configuration().apply {
brokerUrl = "tcp://192.168.99.101:1883"
clientId = "paho123"
isCleanStart = true
}
context.addComponent("paho-mqtt5", mqtt5Component)
from("timer:test?period=1000").setBody(constant("Testing timer2")).to("paho-mqtt5:test")
from("paho-mqtt5:test").process { e ->
val body = (e.`in`?.body as? ByteArray)?.let { String(it) }
println("test body 1 => $body")
}
}
}
#William, this is expected behavior
The message broker uses the client id to differentiate between clients so it can perform housekeeping for a client connection that is no longer used
In addition, a client may have a "Last Will and Testament" that the broker keeps track of
It is acceptable to append a random number to the end of your current 'clientId' since it is likely no one but you will care about this
If you have access to the individuals login, you could use that as well but you would still want to make each session unique in case they run multiple sessions
Maybe I don't understand what your problem is
Each client must have a unique Id
What are you observing that makes you think that it is creating multiple connections for a single client?
Is there a chance you are opening multiple windows and each is generating a different clientId?
This is a good way to diagnose issues by monitoring what the server is seeing
My paho-mqtt client (Javascript) is connecting as "webclient" and I append a randome number (webclient173) to identify this client
To troubleshoot, I would suggest you close all connections on the client and monitor the log of the MQTT process
When the monitor is in place, open a connection from a client that currently has no connections
This is an example connection to my Mosquitto log file
$ tail -f /var/log/mosquitto/mosquitto.log
1635169943: No will message specified.
1635169943: Sending CONNACK to webclient173 (0, 0)
1635169943: Received SUBSCRIBE from webclient173
1635169943: testtopic (QoS 0)
1635169943: Sending SUBACK to webclient173
1635170003: Received PINGREQ from webclient173
1635170003: Sending PINGRESP to webclient173
1635170003: Received PINGREQ from webclient173
1635170003: Sending PINGRESP to webclient173
What does your log show?

get JMSMessageID when use producerTemplate

In Camel,
ProducerTemplate producerTemplate = exchange.getContext().createProducerTemplate();
producerTemplate.sendBody("endpointqueue?includeSentJMSMessageID=true", ExchangePattern.InOnly, body);
I would need to get JMSMessageID that is returned from IBM MQ/ActiveMQ. I am looking at exchange values on debug mode but cannot find it. I can only find sessionID. Where is it stored and how to get it?
The Camel documentation says:
includeSentJMSMessageID - only applicable when sending to jms destination using InOnly. enabling this option will enrich the Camel Exchange with the actual JMSMessageID that was used by the JMS client when the message was sent to the JMS destination.
Is includeSentJMSMessageID different than my needs or am I missing something?
The doc says the JMS MessageID is available as header.
So you should be able to return it like this:
from("direct:queueMessage")
.to("jms://myqueue?includeSentJMSMessageID=true")
.setBody().simple("${header.JMSMessageID}");
And then send your message using:
String msgId = producerTemplate.requestBody("direct:queueMessage", body, String.class);

Google Cloud Run pubsub pull listener app fails to start

I'm testing pubsub "pull" subscriber on Cloud Run using just listener part of this sample java code (SubscribeAsyncExample...reworked slightly to fit in my SpringBoot app):
https://cloud.google.com/pubsub/docs/quickstart-client-libraries#java_1
It fails to startup during deploy...but while it's trying to start, it does pull items from the pubsub queue. Originally, I had an HTTP "push" receiver (a #RestController) on a different pubsub topic and that worked fine. Any suggestions? I'm new to Cloud Run. Thanks.
Deploying...
Creating Revision... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined
by the PORT environment variable. Logs for this revision might contain more information....failed
Deployment failed
In logs:
2020-08-11 18:43:22.688 INFO 1 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 4606 ms
2020-08-11T18:43:25.287759Z Listening for messages on projects/ce-cxmo-dev/subscriptions/AndySubscriptionPull:
2020-08-11T18:43:25.351650801Z Container Sandbox: Unsupported syscall setsockopt(0x18,0x29,0x31,0x3eca02dfd974,0x4,0x28). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/setsockopt for more information.
2020-08-11T18:43:25.351770555Z Container Sandbox: Unsupported syscall setsockopt(0x18,0x29,0x12,0x3eca02dfd97c,0x4,0x28). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/setsockopt for more information.
2020-08-11 18:43:25.680 WARN 1 --- [ault-executor-0] i.g.n.s.i.n.u.internal.MacAddressUtil : Failed to find a usable hardware address from the network interfaces; using random bytes: ae:2c:fb:e7:92:9c:2b:24
2020-08-11T18:45:36.282714Z Id: 1421389098497572
2020-08-11T18:45:36.282763Z Data: We be pub-sub'n in pull mode2!!
Nothing else after this and the app stops running.
#Component
public class AndyTopicPullRecv {
public AndyTopicPullRecv()
{
subscribeAsyncExample("ce-cxmo-dev", "AndySubscriptionPull");
}
public static void subscribeAsyncExample(String projectId, String subscriptionId) {
ProjectSubscriptionName subscriptionName =
ProjectSubscriptionName.of(projectId, subscriptionId);
// Instantiate an asynchronous message receiver.
MessageReceiver receiver =
(PubsubMessage message, AckReplyConsumer consumer) -> {
// Handle incoming message, then ack the received message.
System.out.println("Id: " + message.getMessageId());
System.out.println("Data: " + message.getData().toStringUtf8());
consumer.ack();
};
Subscriber subscriber = null;
try {
subscriber = Subscriber.newBuilder(subscriptionName, receiver).build();
// Start the subscriber.
subscriber.startAsync().awaitRunning();
System.out.printf("Listening for messages on %s:\n", subscriptionName.toString());
// Allow the subscriber to run for 30s unless an unrecoverable error occurs.
// subscriber.awaitTerminated(30, TimeUnit.SECONDS);
subscriber.awaitTerminated();
System.out.printf("Async subscribe terminated on %s:\n", subscriptionName.toString());
// } catch (TimeoutException timeoutException) {
} catch (Exception e) {
// Shut down the subscriber after 30s. Stop receiving messages.
subscriber.stopAsync();
System.out.printf("Async subscriber exception: " + e);
}
}
}
Kolban question is very important!! With the shared code, I would like to say "No". The Cloud Run contract is clear:
Your service must answer to HTTP request. Out of request, you pay nothing and no CPU is dedicated to your instance (the instance is like a daemon when no request is processing)
Your service must be stateless (not your case here, I won't take time on this)
If you want to pull your PubSub subscription, create an endpoint in your code with a Rest controller. While you are processing this request, run your pull mechanism and process messages.
This endpoint can be called by Cloud Scheduler regularly to keep the process up.
Be careful, you have a max request processing timeout at 15 minutes (today, subject to change in a near future). So, you can't run your process more than 15 minutes. Make it resilient to fail and set your scheduler to call your service every 15 minutes

Rollback the message to Dead Letter Queue - Apache Camel

I have set up the Apache camel in which i consumes the message from one queue and do some kind of operation on it and then transfers it to other queue .
Now if the exception comes then i want that it should rollback and then after 6 attempts it to send to dead letter queue , Currently rollback happens 5-6 times but my message is not transferred to dead letter queue .
Here What happens -->
Queue1-->(Consumes)-->Operation(Exception thrown)--> rollback --> again Queue1-->(Consumes) --> Operation(Exception thrown)-->rollback -->... this happens 5-6 times and then my message is lost
I dont know where my message is going and why it is getting lost , and from my Active MQ GUI i can see it is dequeued.
#Bean
public RedeliveryPolicy redeliveryPolicy() {
RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy();
redeliveryPolicy.setMaximumRedeliveries(2);
redeliveryPolicy.setMaximumRedeliveryDelay(10000);
redeliveryPolicy.setRedeliveryDelay(10000);
return redeliveryPolicy;
}
---------------------Route extends SpringRouteBuilder-------------------
onException(MyException.class)
.markRollbackOnly()
.redeliveryPolicy(redeliveryPolicy)
.useExponentialBackOff()
.handled(true)
from("jms:queue:Queue1")
.process(new Processor(){
public void process(Exchange ex){
throw new RuntimeException();
}
}).to("jms:queue:myQueue)
I assume there are multiple problems.
markRollbackOnly stops the message. After this statement no further routing is done.
That is the reason why your RedeliveryPolicy and the rest of your onException route is completely ignored. You configure 2 redelivery attempts but you write it does 5 (the default redelivery of ActiveMQ).
To fix this, move markRollbackOnly to the end of your onException route
If you consume transacted from your JMS broker the message must not get lost.
Since you lose it in case of an error, there is a problem with your transaction config. Configure the ActiveMQ component of Camel to use local JMS transactions when consuming.
#Bean(name = "activemq")
#ConditionalOnClass(ActiveMQComponent.class)
public ActiveMQComponent activeMQComponent(ConnectionFactory connectionFactory) {
ActiveMQComponent activeMQComponent = new ActiveMQComponent();
activeMQComponent.setConnectionFactory(connectionFactory);
activeMQComponent.setTransacted(true);
activeMQComponent.setLazyCreateTransactionManager(false);
return activeMQComponent;
}
When this is in place, you can in fact remove the onException route because the redelivery is done by the JMS broker, so you have to configure the redelivery settings on your JMS connection. If the configured redelivery is exhausted and the message still produces a rollback, it is moved to the DLQ.
Be aware when using an additional onException route because this is pure Camel. The Camel error handler does NOT redeliver on route level, but on processor level. So if you configure both broker and Camel redelivery, it can multiply them.

Camel with RabbitMQ exception only occurs on second message - mis-spelt exchange name

I'm using Camel within a Spring boot application and integrate with RabbitMQ but am encountering strange behaviour.
My app has Restful endpointswhich convert the http request to a RabbitMQ message and publish this to a predefined exchange. There is a separate consumer app which listens to a queue and processes the messages.
I have deliberately entered an incorrect rabbitmq exchange name (invalidxchangename)to check that the application will fail if the exchange does not exist however the camel context starts without error and when I send in a first request is does not report any error. This message gets lost as there is no matching RabbitMQ exchange. When I submit a second request I receive the following exception which I would have expected on route startup.
com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'invalidxchangename' in vhost
EDIT:
I've tried a more simple example to show the issue in Camel.
I've created a simple route as follows:
from("file:in?fileName=in.txt").log(LoggingLevel.DEBUG, "in here!").to("rabbitmq://localhost:5762/invalidexchange?declare=false");
where there is an existing RabbitMQ exchange called validexchange (so I have deliberately made a typo in the RabbitMQ uri). I would expect the camel route to fail at startup since the exchange doesn't exist, or even the first time it tries to process a new in.txt file.
What I am actually seeing in the logs is that on start up it reports no error and only on the 2nd invocation of the route does it report an error.
2015-03-11 16:17:04.356 INFO 9756 : ID-SBMELW7W-06220-59960-1426051020468-0-2 >>> (route2) from(file://in?fileName=in.txt) --> log[in here!] <<< Pattern:InOnly, Headers:...
2015-03-11 16:17:04.360 INFO 9756 : ID-SBMELW7W-06220-59960-1426051020468-0-2 >>> (route2) log[in here!] --> rabbitmq://localhost:5762/customerchannel.exchang?declare=false <<< Pattern:InOnly, Headers:...
2015-03-11 16:17:45.073 INFO 9756 : ID-SBMELW7W-06220-59960-1426051020468-0-4 >>> (route2) from(file://in?fileName=in.txt) --> log[in here!] <<< Pattern:InOnly, Headers: ...
2015-03-11 16:17:45.079 INFO 9756 : ID-SBMELW7W-06220-59960-1426051020468-0-4 >>> (route2) log[in here!] --> rabbitmq://localhost:5762/customerchannel.exchang?declare=false <<< Pattern:InOnly, Headers:...
2015-03-11 16:17:45.092 ERROR 9756 : Failed delivery for (MessageId: ID-SBMELW7W-06220-59960-1426051020468-0-3 on ExchangeId: ID-SBMELW7W-06220-59960-1426051020468-0-4). Exhausted after delivery attempt: 1 caught: com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'customerchannel.exchang' in vhost '/', class-id=60, method-id=40)
It looks like the first request is causing an error which closes the connection and logs the reason, and when you try to use the channel the second time it's returning an AlreadyClosedException with the message that caused the channel to close in the first call.
You can test this by trying to publish the second message to a different exchange name in the same channel and checking which exchange is in the error. E.g. publish the second message to invalidxchangename2 and you should still see invalidxchangename as the exchange in the error.
To fix, you should handle the publish result when you publish and re-establish the connection if there's an error.
If you want to be sure that a message got delivered to a RabbitMQ queue, then you have to use publisher confirms: https://www.rabbitmq.com/confirms.html
That you are able to publish a message it doesn't mean that the message will reach a queue. You could go to a mailbox and leave a letter inside, but between the time you left the letter there and a postman picked up, many things could have happened, for example, the mailbox catching fire and so on.

Resources