I have an issue with Camel.
I am creating dynamic queues like this
for (Client client : clients) {
// Get the ids
String clientId = client.getId();
String queueId = "jms:sendRoute" + clientId;
// Generate the queues
from(queueId)
.routeId(queueId)
.errorHandler(defaultErrorHandler().maximumRedeliveries(-1).redeliveryDelay(1000 * 60 * 5)) // Retries indefinitively
.transacted()
.process(new MyProcessor(clientId, clientService))
.to("mock:end");
}
Which are called by this initial route
from("direct:availabilityRoute")
.routeId("availabilityRoute")
.unmarshal(bindy)
.split(body())
.process(new AProcessor(controler))
.split(body())
.process(new DBProcessor(controler))
.process(new RoutesProcessor(controler))
.recipientList(header("clientIds"), ",")
.to("mock:end");
The route processor generates the list of queue ids to which a essage is multicasted
ids.append("jms:sendRoute" + client.getId());
and Ids (a string) is put in the message header
I am using activeMQ
I don't understand the behaviour
When it starts, it creates 3 routes dynamically. They are named:
sendRoute1
sendRoute2
sendRoute3
(this is what I see in the admin console)
The list of ids generated by the processor is: jms:sendRoute1,jms:sendRoute2, jms:sendRoute3
The issue I have is that when the message is sent to the multiple route, when I put a breakpoint I see that the first message is consumed by sendRoute1, and that 2 new routes named jms:sendRoute2 and jms:sendRoute3 have been added.
The message has been sent to:
- sendRoute1
- jms:sendRoute2
- jms:sendRoute3
But I have no consumer on jsm:sendRoute2 and jms:sendRoute3 so the messages are not read.
Could somebody explain me why this happens and how I can solve this?
Regards
Gilles
When a message is multicasted to all the queues in scope
Related
I am setting up a Camel Route with ackMode=NONE meaning acknowlegements are not done automatically. How do I explicitly acknowledge the message in the route?
In my Camel Route definition I've set ackMode to NONE. According to the documentation, I should be able to manually acknowledge the message downstream:
https://github.com/apache/camel/blob/master/components/camel-google-pubsub/src/main/docs/google-pubsub-component.adoc
"AUTO = exchange gets ack’ed/nack’ed on completion. NONE = downstream process has to ack/nack explicitly"
However I cannot figure out how to send the ack.
from("google-pubsub:<project>:<subscription>?concurrentConsumers=1&maxMessagesPerPoll=1&ackMode=NONE")
.bean("processingBean");
My PubSub subscription has an acknowledgement deadline of 10 seconds and so my message keeps getting re-sent every 10 seconds due to ackMode=NONE. This is as expected. However I cannot find a way to manually acknowledge the message once processing is complete and stop the re-deliveries.
I was able to dig through the Camel components and figure out how it is done. First I created a GooglePubSubConnectionFactory bean:
#Bean
public GooglePubsubConnectionFactory googlePubsubConnectionFactory() {
GooglePubsubConnectionFactory connectionFactory = new GooglePubsubConnectionFactory();
connectionFactory.setCredentialsFileLocation(pubsubKey);
return connectionFactory;
}
Then I was able to reference the ack id of the message from the header:
#Header(GooglePubsubConstants.ACK_ID) String ackId
Then I used the following code to acknowledge the message:
List<String > ackIdList = new ArrayList<>();
ackIdList.add(ackId);
AcknowledgeRequest ackRequest = new AcknowledgeRequest().setAckIds(ackIdList);
Pubsub pubsub = googlePubsubConnectionFactory.getDefaultClient();
pubsub.projects().subscriptions().acknowledge("projects/<my project>/subscriptions/<my subscription>", ackRequest).execute();
I think it is best if you look how the Camel component does it with ackMode=AUTO. Have a look at this class (method acknowledge)
But why do you want to do this extra work? Camel is your fried to simplify integration by abstracting away low level code.
So when you use ackMode=AUTO Camel automatically commits your successfully processed messages (when the message has successfully passed the whole route) and rolls back your not processable messages.
I am having two camel routes sending fixed messages(heartbeats) using timer:monitor. This will in turn be sent to two endpoints from the Producer processor. When the message is not consumed by the consumer then the same message is retired 3 times from the producer. After successfully consuming, the timer should send the next message. But for me, it sends all the messages that were pending in the timer monitor while the message was retried in the producer.
I have used the timer monitor route as follows in my route builder
for (final EndpointInfo endpointInfo : endpointInfos) {
final String uri = SIA_ENDPOINT_PREFIX + endpointInfo.getUri();
from("timer:monitor" + uri + "?fixedRate=true&period=" + (heartbeatInterval * 1000))
.routeId(endpointInfo.getHeartbeatRouteId())
.autoStartup(false)
.process(new SetTimeStamp(nullMessageBuilder))
.setBody(constant(nullMessageBuilder))
.doTry()
.process(new EndpointInfoProcessor(endpointInfo))
.to(uri)
.process(new HealthProcessor(endpointInfo, true))
.doCatch(Throwable.class)
.process(new HealthProcessor(endpointInfo, false))
.end();
uris[index++] = uri;
routeIds[index] = endpointInfo.getHeartbeatRouteId();
}
How can I discard the new bulk of messages/stop the timer from sending all the messages at once?
Set fixedRate=false. You can read more about what fixed rate means in the JDK documentation about Timer: https://docs.oracle.com/javase/7/docs/api/java/util/Timer.html#scheduleAtFixedRate(java.util.TimerTask,%20java.util.Date,%20long)
From a programming point of view, I have a very simple business case. However, I can't figure out how to implement it using Apache Camel... Well, I have 2 JMS queues: one to receive commands, another - to store large number of message which should be delivered to external system in a batches of 1000 or less.
Here is the concept message exchange algorithm:
upon receiving a command message in 1st JMS queue I prepare XML
message
Send the XML message to external SOAP Web Service to obtain a usertoken
Using the usertoken, prepare another XML message and send it to a REST service to obtain jobToken
loop:
4.1. aggregate messages from 2nd JMS queue in batches of 1000, stop aggregation at timeout
4.2. for every batch, convert it to CSV file
4.3. send csv via HTTP Post to a REST service
4.4. retain batchtoken assigned to each batch
using the jobtoken prepare XML message and send to REST service to commit the batches
using batchtoken check execution status of each batch via XML message to REST service
While looking at Camel I could create a sample project where I can model out the exchange 1-3, 5:
from("file:src/data?noop=true")
.setHeader("sfUsername", constant("a#fd.com"))
.setHeader("sfPwd", constant("12345"))
.to("velocity:com/eip/vm/bulkPreLogin.vm?contentCache=false")
.setHeader(Exchange.CONTENT_TYPE, constant("text/xml; charset=UTF-8"))
.setHeader("SOAPAction", constant("login"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://bulklogin") // send login
.to("xslt:com/eip/xslt/bulkLogin.xsl") //xslt transformation to retrieve userToken
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String body = (String) exchange.getIn().getBody();
String[] bodyParts = body.split(",");
exchange.getProperties().put("userToken", bodyParts[0]);
.....
}
})
.to("velocity:com/eip/vm/jobInsertTeamOppStart.vm")
.setHeader(Exchange.CONTENT_TYPE, constant("application/xml; charset=UTF-8"))
.setHeader("X-Session", property("userToken"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://scheduleJob") //schedule job
.to("xslt:com//eip/xslt/jobInfoTransform.xsl")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String body = (String) exchange.getIn().getBody();
exchange.getProperties().put("jobToken",body.trim());
}
})
//add batches in a loop ???
.to("velocity:com/eip/vm/jobInsertTeamOppEnd.vm")
.setHeader(Exchange.HTTP_URI, simple("https://na15.com/services/async/job/${property.jobToken}"))
.setHeader(Exchange.CONTENT_TYPE, constant("application/xml; charset=UTF-8"))
.setHeader("X-ID-Session", property("userToken"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://closeJob") //schedule job
//check batch?
.bean(new SomeBean());
So, my question is:
How can I read messages from my 2nd JMS queue?
This doesn't strike me as a very good use-case for a single camel route. I think you should implement the main functionality in a POJO and use Camels Bean Integration for consuming and producing messages. This will result in much more easy to maintain code, and also for easier Exception handling.
See https://camel.apache.org/pojo-consuming.html
I am trying to validate message with Apache Camel (ver 2.10.4) that is sent to a Virtual Topic in FuseESB (based on Apache ServiceMix, ver 7.1.0), using validator:xsd (message is XML in TextMessage), and when it fails the validation, I want to redirect the message to another topic, and stop processing, so do not send it to usual consumers. Because consumers will fail with invalid message.
I wanted to do validation with routing, so validating once, instead of doing it on multiple consumers.
Is this possible with Camel? and what would be the syntax?
my current approach is like this:
static final String ACTIVEMQ_TOPIC_PREFIX = "activemq:topic:";
static final String ACTIVEMQ_CONSUMER_PREFIX = "activemq:queue:Consumer.*.";
static final String TOPIC_ORDER_CREATED = "VirtualTopic.order.created";
static final String TOPIC_ORDER_CREATED_ERROR =
"VirtualTopic.order.created.error";
static final String DIRECT_ORDER_CREATED_ERROR = "direct:orderCreatedError";
from(DIRECT_ORDER_CREATED_ERROR)
.to(ACTIVEMQ_TOPIC_PREFIX + TOPIC_ORDER_CREATED_ERROR)
.log("Message sent to " + TOPIC_ORDER_CREATED_ERROR);
// validate order.created topic message
// before sending to consumer queues.
from(ACTIVEMQ_TOPIC_PREFIX + TOPIC_ORDER_CREATED)
.errorHandler(deadLetterChannel(DIRECT_ORDER_CREATED_ERROR))
.choice() // validation is enabled with property
.when(simple("${properties:" + PROP_VALIDATION_ENABLED + "} == true"))
.log("Validating order created body")
.to("validator:xsd/myxsd.xsd") // validate against xsd
.onException(ValidationException.class)
.handled(true)
.maximumRedeliveries(0)
.useOriginalMessage()
// if invalid send to error topic
.to(DIRECT_ORDER_CREATED_ERROR)
.stop()
.end()
.end()
.to(ACTIVEMQ_CONSUMER_PREFIX + TOPIC_ORDER_CREATED)
.log("Message sent to " + TOPIC_ORDER_CREATED);
I see "Validating order created body" and "Message sent to VirtualTopic.order.created.error" in the logs. On webconsole, I see a message enqueued in error topic for one message of main topic.
The problem is the consumer of VirtualTopic.order.created still gets the invalid message
Could you please help me to find the right syntax to intercept message before it goes to consumers of VirtualTopic?
Thanks
You can just use the deadLetterChannel, and have it use the original message, and then any errors gets handled and moved to the DLQ.
from(ACTIVEMQ_TOPIC_PREFIX + TOPIC_ORDER_CREATED)
.errorHandler(deadLetterChannel(DIRECT_ORDER_CREATED_ERROR).useOriginalMessage())
.choice() // validation is enabled with property
.when(simple("${properties:" + PROP_VALIDATION_ENABLED + "} == true"))
.log("Validating order created body")
.to("validator:xsd/myxsd.xsd") // validate against xsd
.end()
.to(ACTIVEMQ_CONSUMER_PREFIX + TOPIC_ORDER_CREATED)
.log("Message sent to " + TOPIC_ORDER_CREATED);
Also if you are using onException in the route, then you should put that in the top of the route, not in the middle.
I'm working on a camel prototype which uses two start points in the same camel context.
The first route consumes messages which are used to "configure" the application. Messages are loaded in a configuration repository through a configService bean:
// read configuration files
from("file:data/config?noop=true&include=.*.xml")
.startupOrder(1)
.to("bean:configService?method=loadConfiguration")
.log("Configuration loaded");
The second route implements a recipient list eip pattern, delivering a different kind of input messages to a number of recipients, which are read dinamically from the same configuration repository:
// process some source files (using configuration)
from("file:data/source?noop=true")
.startupOrder(2)
.unmarshal()
.to("setupProcessor") // set "recipients" header
.recipientList(header("recipients"))
// ...
The question that arises now is how to synchronize them, so the second route "waits" if the first is processing new data.
I'm new to Apache Camel and pretty lost on how to approach such a problem, any suggestion would be appreciated.
Use aggregate in combination with the possibility to start and stop routes dynamically:
from("file:data/config?noop=true&include=.*.xml")
.id("route-config")
.aggregate(constant(true), new MyAggregationStrategy()).completionSize(2).completionTimeout(2000)
.process(new Processor() {
#Override
public void process(final Exchange exchange) throws Exception {
exchange.getContext().startRoute("route-source");
}
});
from("file:data/source?noop=true&idempotent=false")
.id("route-source") // the id is needed so that the route is found by the start and stop processors
.autoStartup(false) // this route is only started at runtime
.aggregate(constant(true), new MyAggregationStrategy()).completionSize(2).completionTimeout(2000)
.setHeader("recipients", constant("direct:end")) // this would be done in a separate processor
.recipientList(header("recipients"))
.to("seda:shutdown"); // shutdown asynchronously or the route would be waiting for pending exchanges
from("seda:shutdown")
.process(new Processor() {
#Override
public void process(final Exchange exchange) throws Exception {
exchange.getContext().stopRoute("route-source");
}
});
from("direct:end")
.log("End");
That way, route-source is only started when route-config is completed. route-config and consequently route-source are restarted if new files are found in the config directory.
You can also place an "on completion" http://camel.apache.org/oncompletion.html in the first route that activates the second one.
Apache camel File will create a lock for the file that being processed. Any other File process on this file will not pool on if there is a lock (except if you put consumer.exclusiveReadLock=false)
source :
http://camel.apache.org/file.html => URI Options => consumer.exclusiveReadLock