I am trying to get time taken for a message just to be delivered to the IBM MQ queues in Apache Camel. Below is the route configuration,
from("direct:deliver-route-2").process("mySampleProcessor").split().method("messageSplitterBean", "splitMessage").shareUnitOfWork().stopOnException().toD("mqDeliverJms:${headers.Deliver}?preserveMessageQos=true");
I have a splitter in place to split the list of messages and dynamically deliver (toD) it using the messages header "Deliver".
I used the MessageHistory suggestion given in one of the questions, but then I don't see the elapsed time for the (toD) delivery of the messages to IBM MQ.
String MessageHist = org.apache.camel.util.MessageHelper.dumpMessageHistoryStacktrace(exchange,
new DefaultExchangeFormatter(), false);
Here is the message history,
You can use the 'EventNotifier' to capture the ExchangeSent events where you have the time, eg there is little example in the Camel docs at: http://camel.apache.org/eventnotifier-to-log-details-about-all-sent-exchanges.html
Related
I have about 10K messages in a CSV file. Each message has an associated timestamp for it. When the time is reached, I want the message delivered to an MQ. The timestamps are not uniformly spread. Is this possible with Apache Camel?
As far as I know Apache Camel by default has no consumer endpoint components that you could configure to trigger with specific messages at specified times.
There is however a timer component that you can setup to trigger for example once a second. Then in the route you could use a processor to check if a list contains any messages that should be send at the given time and send them to the MQ.
You could also trigger route from java code using a ProducerTemplate that you can create using the CamelContext.
The list could be populated using your csv file and ordered by timestamp so you could use it like a stack and only check the first few entries instead of going through all 10K every second.
The real problem here would be persistence i.e to figure out which of the messages listed on the csv have already been sent if the application closes before all 10K messages have been sent.
In my Apache Camel application, I have a very simple route:
from("aws-sqs://...")
.aggregate(constant(true), new AggregationStrategy())
.completionSize(100)
.to("SEND_AGGREGATE_VIA_HTTP");
That is, it takes messages from AWS SQS, groups them in batches of 100, and sends them via HTTP somewhere.
Exchanges with messages from SQS are completed successfully on getting into the aggregate stage, and SqsConsumer deletes them from the queue at this point.
The problem is that something might happen with an aggregated exchange (it might be delivered with an error), and messages will be lost. I would really like these original exchanges to be completed successfully (messages to be deleted from a queue) only when an aggregated exchange they're in is also completed successfully (a batch of messages is delivered). Is there a way to do this?
Thank you.
You could set deleteAfterRead to false and manually delete the messages after you've sent them to you HTTP endpoint; You could use a bean or a processor and send the proper SQS delete requests through the AWS SDK library. It's a workaround, granted, but I don't see a better way of doing it.
I am using Apache camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc).
The way it is been implemented:
Boot a single camel context, the context is disabled for JMX and the
loadStatisticsEnabled is set to false on the ManagementStrategy. As
mentioned in a jira issue, addressed in 2.11.0 version, for disabling
the background management thread creation.
For each message a route is being constructed , the message is being
pushed to the route for delivery.
After the message is processed route is shutdown and removed from
context.
Did a small perf test by having 200 threads of dispatcher component, each sharing the same context.
Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds.
Issue CAMEL-5675 mentions that this has been fixed but still observing significant time being taken in starting up routes.
https://issues.apache.org/jira/browse/CAMEL-5675
The route that is being creating for http is
from("direct:"+dispatchItem.getID())
.toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis);
Each dispatchItem has a unique ID.
This is being active discussed elsewhere, where the user posted this question first: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
I’m stuck at one particular requirement in camel , I’m processing a set of files and when any exception occurs i send a message to an exception queue , and from the exception queue im sending an email about the failure , the functionality is working fine but I end up sending many mails, like if 10 files fail im sending 10 emails, is there a way to send only one mail , like I would want to wait for the entire route to finish , then go look the exception queue and send a single mail stating what has failed (by processing the exception queue )
I'm open for suggestions.
I had to do this scenario once (inversed though - mail on success). I had a handy MySQL database configured and ready, so I just added each event from the queue to the database. Then once every now and then, extracted all info (and deleted it)- simply select * from events; delete from events; from the database and created a mail.
You could process the error queue with the aggregator pattern, it is very nice for these tasks. http://camel.apache.org/aggregate-example.html . You still need to know WHEN the aggregator should fire off a message. If you can, trigger a "finish, send mail" event such as in the example in the link above.
The most simple way would be to time schedule these mail notifications. Take a look at: http://camel.apache.org/simplescheduledroutepolicy.html . You can set it to run your route for some good choice of time, then when it fires off, you set the aggreator to complete upon timeout, and make the timeout good enough to empty any reasonable queue size of errors, but not too large.
At least that's my suggestions to your issue
I am in the process of learning ActiveMQ and Camel, with the goal to create a little prototype system that works something like this:
(source: paulstovell.com)
(big)
When an order is placed in the Orders system, a message is sent out to any subscribers (a pub/sub system), and they can play their part in processing the order. The Orders, Shipping and Invoicing applications have their own ActiveMQ installations, so that if any of the three systems are offline, the others can continue to function. Something takes care of moving messages between the ActiveMQ installs.
Getting Apache Camel to move messages from one queue to another via routes is quite easy, if they are on the same ActiveMQ instance. So this works for managing the subscription queues.
The next challenge is pushing messages from one ActiveMQ instance to another, and it's the bit where I am not sure what to look at next.
Can Camel route between different ActiveMQ installations? (I can't figure out what the JMI endpoint URI would be if they are on different machines).
I understand ActiveMQ has store and forward capabilities. Is this what I would use to move messages between Orders and Shipping/Invoicing?
Or is this what Apache ServiceMix is meant to solve?
This is a pretty straightforward asynchronous, event-driven application that is well-suited for ActiveMQ and Camel.
Actually you do not move messages explicitly from one ActiveMQ instance to another. The way it works is using what's known as a network of brokers. In your case, you'd have three brokers: ActiveMQ-purple, ActiveMQ-green and ActiveMQ-blue. ActiveMQ-purple creates a uni-directional broker network with ActiveMQ-green and ActiveMQ-blue. This allows ActiveMQ-purple to store-and-forward messages to ActiveMQ-green and ActiveMQ-blue based on consumer demand.
The Orders app accepts orders on the orders queue on ActiveMQ-purple. The Orders app uses Camel to consume and process a message to determine if it is an invoicing message or a shipping message. Camel routes the messages to either the invoicing queue or the shipping queue on ActiveMQ-purple.
Consumer demand comes from the Invoicing app and the Shipping app. The Invoicing uses Camel to consume messages from the invoicing queue on ActiveMQ-green. The Shipping app uses Camel to consume messages from the shipping queue on ActiveMQ-blue. Because of the broker network and because of the consumer demand on the ActiveMQ-green.invoicing queue and the ActiveMQ-blue.shipping queue, messages will be forwarded from ActiveMQ-purple to the appropriate broker and queue. There is no need to explicitly route messages to specific broker.
I hope this answers your questions. Let me know if you have anymore.
Bruce
Hmmmm, I've only dabbled at best, and not for a fair while, but I'll try and offer something.
ActiveMQ can route between different installations and just uses standard URIs to my knowledge so I'm not sure what the problem is here. I would think that using TCP you'd be fine. Using ServiceMix (you mention it later) you'd just specify a connectionFactory & then provide the URI in that. This link shows some examples http://servicemix.apache.org/servicemix-jms-new-endpoints.html.
Camel has support for Durable Subscriber if that's what you were after (http://camel.apache.org/durable-subscriber.html)? This pattern will ensure that if the subscriber is offline when the message is ready, it will be held until the subscriber is back online. This is also supported by ServiceMix (see link given above and look for 'subscriptionDurable'.