I have defined a Route which consumes messages from a queue without any problems, and right now I am trying to consume the messages that have been in the queue for 12 hours, I got the idea to use a selector, but no idea how to use it with JMSTimestamp to meet the 12 hours criteria.
This is the Route:
<route id="INBOUND.RECEIVE.IN">
<from
uri="activemq:queue:QXL.INBOUND.RECEIVE.IN?selector=JMSTimestamp%3D${date.time}&concurrentConsumers=10&destination.consumer.prefetchSize=0&deliveryPersistent=true&username=admin01&password=001!admin01001!" />
<pipeline>
<bean method="inboundReceive" ref="logipalServices"/>
</pipeline>
</route>
And this is the date bean
<bean id="date" class="java.util.Date" scope="prototype"/>
Based on your explanation, I think something is wrong with your selector.
JMSTimestamp%3D${date.time} means JMSTimestamp = ${date.time}
It means you're looking for all messages with the exact JMSTimestamp. Since Date.getTime() returns time in milliseconds, it's very unlikely, you will retrieve any message from the queue.
I think you should try something like :
JMSTimestamp > current date - 12 hours (pseudo code for selector)
I hope this will help you.
Related
I am trying to use selectors on an amqp with Azure Service Bus consumer. However, for some reason the route is also consuming messages that do not match the selector.
Here's an example:
This route generates messages and append a header:
<route id="MessageGenerator">
<from uri="timer:generator?delay=5000&period=5000"/>
<setHeader headerName="INSTANCE_ID">
<simple>{{env:INSTANCE_ID}}</simple>
</setHeader>
<to uri="amqp:queue:external_queue" />
</route>
While this route should consume only those that contain INSTANCE_ID matching 2 possible values: env:INSTANCE_ID or Any.
<route id="ExternalConsumer">
<from uri="amqp:queue:external_queue?selector=INSTANCE_ID IN ('{{env:INSTANCE_ID}}', 'Any')"/>
<log message="{{env:INSTANCE_ID}} consumed message with Instance ID: ${header.INSTANCE_ID}" logName="AMQP_TEST" loggingLevel="INFO"/>
</route>
But the the log shows that it is consuming any message, regardless of the selector specifying which ones.
Am I missing something?
Thanks!
Issue here was that Azure Service Bus does NOT support selectors on queues. I switched to topics, which already have filters per subscription.
I am using Apache Camel in OSGI scenario using Karaf in version 2.15.1. I am using the exchange.getExchangeId() to print the exchange id in a request/reply. The exchange pattern is set to InOnly. The route looks like this:
<route id="ip_client_rpc">
<from uri="restlet:http://localhost:7070/lsp/patron/id?restletMethod=POST&synchronous=true"/>
<to uri="log:${headers}"/>
<setExchangePattern pattern="InOnly"/>
<process ref="rabbit_client"/>
<to uri="log:${headers}"/>
</route>
However when I print the exchange id sent to the rabbitmq queue it always ends with an even number.
Request from client:ID-VirtualDev-49301-1443430754519-5-6
Request from client:ID-VirtualDev-49301-1443430754519-5-8
Request from client:ID-VirtualDev-49301-1443430754519-5-10
Request from client:ID-VirtualDev-49301-1443430754519-5-12
Request from client:ID-VirtualDev-49301-1443430754519-5-14
Is there a reason why the final digit is always even? Is there another exchange being created that I am missing?
Thanks
Camel uses the same id generator for generating unique ids for different things, its just by chance that its even in this case. Could be that a breadcrumb or message id was also generated that takes the odd number.
We have a camel route that looks at a file and processes potentially hundreds of records on this file, almost like a batch routine (yet there will only be one message in camel). Thus the message will take potentially minutes or maybe hours to complete. We want to shut down the queue once this message (and any others waiting) are complete.
We have the following to consider:
The shutdown strategy defines the time to wait for a route to stop before a forced shutdown
<bean id="shutdown" class="org.apache.camel.impl.DefaultShutdownStrategy">
<property name="timeout" value="#[bpf.defaultShutdownStrategy.timeout]"/>
</bean>
The route has a parameter shutdownRunningTask="CompleteAllTasks" which should wait untill all messages processed.
Not sure which is going to take presidence as the timeout once exceeded is not graceful, it will force shutdown and for our scneario it is likely we will exceed a timeout, as we cannot predict how long processing will take.
Any ideas/considerations?
Thanks in advance.
You should look at the onCompletion functionality. It adds a new route in a separated thread when the Exchange is complete.
Here is some examples from the Camel documentation:
Java DSL
// define a global on completion that is invoked when the exchange is complete
onCompletion().to("log:global").to("mock:sync");
from("direct:start")
.process(new MyProcessor())
.to("mock:result");
XML DSL
<!-- this is a global onCompletion route that is invoke when any exchange is complete
as a kind of after callback -->
<onCompletion>
<to uri="log:global"/>
<to uri="mock:sync"/>
</onCompletion>
<route>
<from uri="direct:start"/>
<process ref="myProcessor"/>
<to uri="mock:result"/>
</route>
Then, here is documentation on how to stop a route in Camel.
I want to use Camel to take a message from ActiveMQ and then, based on the message contents (a protobuf), send one or more messages to Twitter. I've written a bean that is called from within a route and which uses injection to send multiple messages to a "direct:xyz" endpoint.
However, Camel is complaining at runtime that:
2012-11-16 09:56:33,376 | WARN | ication.twitter] | DirectProducer | 160 - org.apache.camel.camel-core - 2.10.2 | No consumers available on endpoint: Endpoint[direct://twitter] to process: Exchange[Message: hello world]
If I instead inject directly to the Twitter endpoint from within the bean, it works fine. However, in order to ease testing, simplify configuration etc, I'd like to keep the actual Twitter config separate, hence wanting to send to a separate route.
The camel context config looks like:-
<camelContext id="NotificationTwitter"
trace="false" xmlns="http://camel.apache.org/schema/blueprint">
<dataFormats>
<protobuf id="notificationProto" instanceClass="org.abc.schemas.protobuf.NotificationDef$NotificationMsg" />
</dataFormats>
<route id="TwitterPreparation">
<from uri="activemq:notification.twitter" />
<unmarshal ref="notificationProto" />
<log logName="abc" loggingLevel="INFO"
message="Twitter request received: ${body}" />
<bean ref="NotificationTweeter" method="createTweets" />
</route>
<route id="Twitter">
<from uri="direct:twitter" />
<log logName="abc" loggingLevel="INFO"
message="Tweeting: ${body}" />
<to uri="twitter://timeline/user?consumerKey=itsasecret&consumerSecret=itsasecret&accessToken=itsasecret&accessTokenSecret=itsasecret" />
</route>
</camelContext>
The bean looks like:-
public class NotificationTweeter {
#EndpointInject(uri = "direct:twitter")
private ProducerTemplate producerTemplate;
public void createTweets(NotificationMsg notification) {
String tweet = notification.getMessageDetail().getTitle();
try {
// only send tweets where the notification message contains the Twitter mechanism
for (MechanismMsg mechanism : notification.getMechanismList()) {
if (mechanism.getType() == MechanismTypeEnum.TWITTER) {
// Cycle round the recipients
for (RecipientMsg recipient : mechanism.getRecipientList()) {
tweet = "#" + recipient.getIdentifier() + " " + tweet;
producerTemplate.sendBody(tweet);
}
// TODO exceptions if no recipients found, etc
}
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
I've had this problem in other routes (it's certainly not related to the Twitter feature) but have just worked around it. This time, however, I'd like to actually understand what the issue is! Any help gratefully received, thanks.
According to your setup, it might also depend on the CamelContext you have picked up. I got the same error message because I was sending messages on a route that existed in another CamelContext than the one I actually was using.
(Although the previous answer was already accepted, this might be the working solution for other people searching for that error message.)
It sounds like a problem with the startup ordering of your routes. See more detail here http://camel.apache.org/configuring-route-startup-ordering-and-autostartup.html
You can configure the "direct" route to start before the other route, then that issue should be resolved.
For others coming here, this error can also be caused by an OSGI error for a dependency that has not been deployed.
A bit late to the party but this error happened to me when I had two separate blueprint files, one for normal running and one for test. In my test I was referring to the test blueprint but noticed that the normal one was also automatically started which caused errors.
In the documentation http://camel.apache.org/blueprint-testing.html it says you can disable certain bundles from starting up. That helped me in my case.
This can also be caused by having a . in the route name. Replace my.Route.Name with myRouteName fixed the issue for me.
I would like to know if it's possible with Camel to do throttling based on the content of the exchange.
The situation is the following: I have to call a webservice via soap. Among, the parameters sent to that webservice there is a customerId. The problem is that the webservice send back an error if there are more than 1 request per minute for a given customerId.
I'm wondering if it would be possible to implement throttling per customerId with Camel. So the throttling should not be implemented for all messages but only for messages with the same customerId.
Let me know how I could implement this or if I need to clarify my question.
ActiveMQ Message Groups is designed to handle this case. So, if you can introduce a JMS queue hop in your route, then just set the JMSXGroupId header to the customerId. Then in another route, you can consume from this queue and send to your web service to get the behavior you described.
also see http://camel.apache.org/parallel-processing-and-ordering.html for more information...
While ActiveMQ Message Groups would definitely address the parallel processing of unique customer ID's, in my assessment Claus is correct that introducing a throttle for each unique group represents an unimplemented feature for Camel/ActiveMQ.
Message Groups alone will not meet the SLA described. While each group of messages (correlated by the customer ID) will be processed in order with one thread per group, as long as requests take less than a minute to receive a response, the requirement of one request per minute per customer would not be enforced.
That said, I would be very interested to know if it would be possible to combine Message Groups and a throttle strategy in a way that would simulate the feature request in JIRA. My attempts so far have failed. I was thinking something along these lines:
<route>
<from uri="activemq:pending?maxConcurrentConsumers=10"/>
<throttle timePeriodMillis="60000">
<constant>1</constant>
<to uri="mock:endpoint"/>
</throttle>
</route>
However, the throttle seems to be applied to the entire set of requests moving to the endpoint, and not to each individual consumer. I have to admit, I was a bit surprised to find that behavior. My expectation was that the throttle would apply to each consumer individually, which would satisfy the SLA in the original question, provided that the messages include the customer ID in the JMSXGroupId header.
I came across a similar problem and finally came up with the solution described here.
My assumptions are:
Order of messages is not important (though it can be solved by re-sequencer)
Total volume of messages per customer ID is not great so the runtime is not saturated.
The solution approach:
Run aggregator for 1 minute while using customerID to assemble messages with the same customer ID into a list
Use Splitter to split the list into individual messages
Send the first message from the splitter to the actual service
Re-route the rest of the list back into the aggregator.
Java DSL version is a bit easier to understand:
final AggregationStrategy aggregationStrategy = AggregationStrategies.flexible(Object.class)
.accumulateInCollection(ArrayList.class);
from("direct:start")
.log("Receiving ${body}")
.aggregate(header("customerID"), aggregationStrategy).completionTimeout(60000)
.log("Aggregate: releasing ${body}")
.split(body())
.choice()
.when(header(Exchange.SPLIT_INDEX).isEqualTo(0))
.log("*** Processing: ${body}")
.to("mock:result")
.otherwise()
.to("seda:delay")
.endChoice();
from("seda:delay")
.delay(0)
.to("direct:start");
Spring XML version looks like the following:
<!-- this is our aggregation strategy defined as a spring bean -->
<!-- see http://stackoverflow.com/questions/27404726/how-does-one-set-the-pick-expression-for-apache-camels-flexibleaggregationstr -->
<bean id="_flexible0" class="org.apache.camel.util.toolbox.FlexibleAggregationStrategy"/>
<bean id="_flexible2" factory-bean="_flexible0" factory-method="accumulateInCollection">
<constructor-arg value="java.util.ArrayList" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:start"/>
<log message="Receiving ${body}"/>
<aggregate strategyRef="_flexible2" completionTimeout="60000" >
<correlationExpression>
<xpath>/order/#customerID</xpath>
</correlationExpression>
<log message="Aggregate: releasing ${body}"/>
<split>
<simple>${body}</simple>
<choice>
<when>
<simple>${header.CamelSplitIndex} == 0</simple>
<log message="*** Processing: ${body}"/>
<to uri="mock:result"/>
</when>
<otherwise>
<log message="--- Delaying: ${body}"/>
<to uri="seda:delay" />
</otherwise>
</choice>
</split>
</aggregate>
</route>
<route>
<from uri="seda:delay"/>
<to uri="direct:start"/>
</route>
</camelContext>