I have Apache Camel client(consumer) which listen messages from remote ActiveMQ Topic, I observed it only read messages when Consumer is running.
if consumer is not listening and producer send messages they get enqueued forever and picked by consumer.
I want that my Client should read all en-queued messages when its up
public static void main(String[] args) throws Exception
{
ActiveMQComponent amq = new ActiveMQComponent();
amq.setConnectionFactory( new ActiveMQConnectionFactory() );
amq.setUsername("admin");
amq.setPassword("admin");
amq.setBrokerURL("tcp://localhost:8161");
Main main = new Main();
main.bind("activemq", amq);
main.addRouteBuilder(new MyRouter());
main.run(args);
}
Following is router code
public void configure() throws Exception
{
from("activemq:topic:saadtopic")
.transform(simple(" ${body}"))
.to("stream:out");
}
ActiveMQ=5.15.9
ApacheCamel=2.24.1
If you are not using durable subscriptions and you send a message to a topic that has no subscribers the message is essentially dropped never to be seen again. The broker only stores Topic messages when there is a durable subscription with an offline subscriber and the message is marked as persistent.
If you want the have access to messages sent when consumers are down then you need to ensure that those consumers have run at least once and created a durable topic subscriptions.
Related
I've some flink jobs which uses kafka as source and sink and I want to add tracing to it, so that any message consumed/produced from/to Kafka is well traced, for that I'm using kafka interceptors to intercepts messages and log trace, span and parent traceId, for that I'm using
opentracing-kafka-client(v0.1.11) in conjunction with brave-opentracing(v0.35.1), the reason why I'm using custom interceptors because I need to log messages in a specified format.
After configuring interceptors they are getting invoked and it uses tracing information (from headers) coming from upstream system and logs it but when it comes to producing message again to kafka then tracing context is lost for instance consider below scenario
1) Message put on Kafka by some rest service
2) Message consumed by flink job and interceptors kicks in and uses tracing information from header and logs it
3) After processing message is produced by flink job to Kafka
It works well until step #2 but when it comes to producing message then tracing information from previous step is not used because it does not have any headers information and hence it produces entirely new trace.
I'm registering tracer as below :-
public class MyTracer {
private static final Tracer INSTANCE = BraveTracer.create(Tracing.newBuilder().build());
public static void registerTracer() {
GlobalTracer.registerIfAbsent(INSTANCE);
}
public static Tracer getTracer() {
return INSTANCE;
}
}
And I'm using TracingConsumerInterceptor and TracingProducerInterceptor from opentracing kafka.
Please help me in finalizing architecture for Asynchronous delivery support using Apache Camel and ActiveMQ and below I have explained point by point basis about my requirement.
I have Jetty Server receiving incoming messages and ActiveMQ to store it in disk using Kaha DB.
Active MQ sends ack once it stores in kaha DB back to client.
I have Spring AbstractPollingMessageListenerContainer JMS Message listener which picks up the message from activemq queue every 1 second and dispatch to Camel HTTP endpoints and then finally sent to actual remote receivers.
Once Dispatcher thread gets response from remote receivers it deletes message from ActiveMQ.
Assume that I have many slow remote receivers in that case my Dispatcher thread created by AbstractPollingMessageListenerContainer remains blocked until I get response from remote receivers. This results to creation of new Dispatcher threads since already created Dispatcher threads are not able to dispatch new messages from ActiveMQ queue.
Now creation of many Dispatcher threads result into more CPU usage which impacts overall performance.
Now my requirement is I want Dispatcher thread only to dispatch messages from ActiveMQ queue to HTTP endpoint and forget and also not do acknowledgement so that message is still in queue.
Also I will not let Dispatcher thread to wait till I get response so I have thought to handle response using separate thread and this same thread will only delete message from ActiveMQ queue.
So my current architecture is like below:
Camel Jetty Server ----> ActiveMQ queue ----> Dispatcher Thread ---> Camel Direct endpoint ----> Camel HTTP endpoint ---> remote receivers sending response back ---> response ---> Dispatcher Thread (sends ack to delete messages from ActiveMQ queue) ----> ActiveMQ Queue.
Here I feel since we are using Direct endpoint which is synchronous so Dispatcher thread remains active till it gets response and so same dispatcher thread is not able to process further new message from ActiveMQ queue.
Please suggest if some thing else I can use here to avoid Direct endpoint.
I used SEDA endpoint but drawback is it processes 1 message using 1 thread and also gets blocked till it gets response from receivers.
In this approach previously Dispatcher thread gets blocked but now Seda consumer threads gets blocked and could not dispatch new messages from in memory queue of SEDA towards remote receivers.
I feel some kind of design which helps me in keep on sending message to remote receivers and only when response comes back some daemeon thread gets notified and it will handle acknowledgement towards activeMQ. Also I thought to use NIO framework implementation like Camel netty/netty4-http component but could not find exact usage and how to fit it in current architecture.
Modified architecture should be like below:
Camel Jetty Server ----> ActiveMQ queue ----> Dispatcher Thread--->Unknown Stuff ----> Camel HTTP Endpoint ---> remote receivers sending response back--->Unknown Stuff (sends ack to delete messages from ActiveMQ queue) ----> ActiveMQ Queue
Please help me in finalising Unknown Stuff and I am posting my query after doing enough R & D.
Also new ideas are welcome and please give me idea with a restriction that I must persist the message and delete it only after getting success response from remote receiver. Also I have to design architecture only using Apache Camel routes.
Route Definitions:
1. Dispatcher Route:
from(fromUri)to(toUris);
fromUri:
[ActiveMQueue.http1270018081testEndpoint1:queue:ActiveMQueue?maxConcurrentConsumers=15&concurrentConsumers=3&maxMessagesPerTask=10&messageListenerContainerFactoryRef=AbstractPollingMessageListenerContainer ]
ToUris:
[ActiveMQ.DLQ:queue:ActiveMQ.DLQ, direct:http1270018081testEndpoint1]
2.Remote Receiver Proxy Route:
fromUri:direct:http1270018081testEndpoint1
from(fromUri).to(toUri).process(responseProcessor)
toUri:http://127.0.0.1:8081/testEndpoint1?bridgeEndpoint=true
responseProcessor: To process response received by remote receiver.
Overall Route looks like below:
Dispatcher Route---> Remote Receiver Route---> Remote Server
JMS message acknowledgement is done under the covers, so your only way to really "send the acknowledgement back to the queue" is to use a JMS transaction (doesn't need to be XA)
It sounds like a LLR-style transaction would be useful and drastically simplify things for you. If you consume the message from the queue using a JMS-local transaction, and only have one other endpoint, the message will only be acknowledged and removed from the queue when the http send is completed-- even though HTTP doesn't support transactions. You can then have a number of concurrent consumers to run in parallel and combine with throttling to help with rate limiting.
from: amq:queue:INPUT.REQUESTS?.. concurrentConsumers.. and transacted enabled
throttle
to: http://url
I have a small Camel route which just forward messages to another queue with an expiration time like this:
#Override
public void configure() throws Exception {
defaultOnException();
// Route all messages generated by system A (in OUTBOUND_A) to system B (INBOUND_B)
// #formatter:off
from("activemq:queue:OUTBOUND_A")
// ASpecificProcessor transform the coming message to another one.
.processor(new ASpecificProcessor())
.to("activemq:INBOUND_B?explicitQosEnabled=true&timeToLive={{b.inbound.message.ttl}}");
// #formatter:on
}
I need the messages posted in INBOUND_B to be persistent and by default the expired message goes to ActiveMQ.DLQ queue after expired.
I know I can modify the ActiveMQ configuration in the conf/activemq.xml with
<policyEntry queue="INBOUND_B">
<!--
Tell the dead letter strategy not to process expired messages
so that they will just be discarded instead of being sent to
the DLQ
-->
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
</policyEntry>
But I would prefer not to change the ActiveMQ configuration (because it needs a restart) and I am wondering if it is possible to send such policy through the Camel endpoint configuration?
No, ActiveMQ broker side configuration cannot be updated via the client, that would lead to all sorts of security problems. You would need to update the broker configuration and possibly not need a restart if you use the runtime configuration plugin on the broker.
I'm using akka-camel to subscribe to a rabbitmq exchange. There will be several of these actors created... one per requested routingKey. The exchange and queue doesn't change. Each time a new routingKey is requested I create a new actor and, instead of a new channel being created, a brand new connection is being created, which is undesirable. I don't quite understand why a new connection is being created each time the Consumer actor is being created.
Here's the actor code:
class CommandConsumer(routingKey: String)
extends Consumer with ActorLogging {
override def endpointUri = s"rabbitmq://localhost/hub_commands?exchangeType=topic&queue=test&autoDelete=false&routingKey=$routingKey"
override def receive: Receive = {
case msg: CamelMessage => {
log.debug(s"received {}", msg.bodyAs[String])
sender ! msg.bodyAs[String]
}
}
}
I'm creating the actor like this:
context.actorOf(CommandConsumer.props("my.routing.key", sender))
UPDATE
Here's exactly what I need to accomplish:
I'm writing a TCP/IP server that, when a client connection is accepted, needs to receive messages from other components in the back-end architecture. To do this, I'd like to use RabbitMQ. After a successful connection to my server, the client will send an id, which will be used as part of a routing key (e.g. command.<id>). The RabbitMQ connection and queue is created when the first client connects and the routing key would be something like command.first_id. When the next client connects I would like to add command.second_d routing key to the list of routing keys that are already accepted, without creating a new connection to RabbitMQ.
I believe that is expected. Each Akka Camel actor will have their own Camel Context which will be independent from others. This means that for each new actor you create, you will be creating a new camel context with a new RabbitMQ endpoint that will hold a new RabbitMQ connection and a new Channel.
If in your scenario the queue and exchange does not change but just the routing key, why don't you just have one Akka Camel actor consuming from the queue and another actor managing the bindings. Every time there is a new routing key that needs to be consumed, this actor will create a rabbitmq connection, channel and call queueBind() with the new routing key. Also, you can have the same actor unbinding the undesired routing keys.
I am trying to figure out throttling concepts in Camel. I have already seen Camel's route policy, but this works for number of inflight exchanges.
My route is following:
routeBuilders.add(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("rabbitmq://127.0.0.1:5672/abc?queue=a&username=XXX&password=XXX&autoAck=false&durable=true&exchangeType=direct&autoDelete=false")
.to("rabbitmq://127.0.0.1:5672/abc?queue=b&username=XXX&password=XXX&autoAck=false&durable=true&exchangeType=direct&autoDelete=false");
}
});
Now my use case is that I want to transfer say 2000 messages between these routes, and I know that it can be done via .throttle(2000). But I am stuck at the point where I have to decide that how would I control that when the next 2000 messages should be routed. I want to route next 2000 messages only when the receiver queue becomes empty.
For example, messages are getting routed from queue a to b. Say 2k messages have been routed successfully, now I want to suspend my route so that it won't transfer more messages until the queue b becomes empty (assume that there is a consumer which is pulling messages from queue b)
Any help/direction on this is appreciated.
You can use a route policy for that. The you implement logic in that route policy to suspend/resume the route to throttle the route.
We have a out of the box policy that is using the number of inflight messages in Camel as its metrics. But you should add logic to check that queue if its empty or not.
The documentation for route policy is here
http://camel.apache.org/routepolicy
And the code for the inflight throttler
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/impl/ThrottlingInflightRoutePolicy.java