I am trying to validate message with Apache Camel (ver 2.10.4) that is sent to a Virtual Topic in FuseESB (based on Apache ServiceMix, ver 7.1.0), using validator:xsd (message is XML in TextMessage), and when it fails the validation, I want to redirect the message to another topic, and stop processing, so do not send it to usual consumers. Because consumers will fail with invalid message.
I wanted to do validation with routing, so validating once, instead of doing it on multiple consumers.
Is this possible with Camel? and what would be the syntax?
my current approach is like this:
static final String ACTIVEMQ_TOPIC_PREFIX = "activemq:topic:";
static final String ACTIVEMQ_CONSUMER_PREFIX = "activemq:queue:Consumer.*.";
static final String TOPIC_ORDER_CREATED = "VirtualTopic.order.created";
static final String TOPIC_ORDER_CREATED_ERROR =
"VirtualTopic.order.created.error";
static final String DIRECT_ORDER_CREATED_ERROR = "direct:orderCreatedError";
from(DIRECT_ORDER_CREATED_ERROR)
.to(ACTIVEMQ_TOPIC_PREFIX + TOPIC_ORDER_CREATED_ERROR)
.log("Message sent to " + TOPIC_ORDER_CREATED_ERROR);
// validate order.created topic message
// before sending to consumer queues.
from(ACTIVEMQ_TOPIC_PREFIX + TOPIC_ORDER_CREATED)
.errorHandler(deadLetterChannel(DIRECT_ORDER_CREATED_ERROR))
.choice() // validation is enabled with property
.when(simple("${properties:" + PROP_VALIDATION_ENABLED + "} == true"))
.log("Validating order created body")
.to("validator:xsd/myxsd.xsd") // validate against xsd
.onException(ValidationException.class)
.handled(true)
.maximumRedeliveries(0)
.useOriginalMessage()
// if invalid send to error topic
.to(DIRECT_ORDER_CREATED_ERROR)
.stop()
.end()
.end()
.to(ACTIVEMQ_CONSUMER_PREFIX + TOPIC_ORDER_CREATED)
.log("Message sent to " + TOPIC_ORDER_CREATED);
I see "Validating order created body" and "Message sent to VirtualTopic.order.created.error" in the logs. On webconsole, I see a message enqueued in error topic for one message of main topic.
The problem is the consumer of VirtualTopic.order.created still gets the invalid message
Could you please help me to find the right syntax to intercept message before it goes to consumers of VirtualTopic?
Thanks
You can just use the deadLetterChannel, and have it use the original message, and then any errors gets handled and moved to the DLQ.
from(ACTIVEMQ_TOPIC_PREFIX + TOPIC_ORDER_CREATED)
.errorHandler(deadLetterChannel(DIRECT_ORDER_CREATED_ERROR).useOriginalMessage())
.choice() // validation is enabled with property
.when(simple("${properties:" + PROP_VALIDATION_ENABLED + "} == true"))
.log("Validating order created body")
.to("validator:xsd/myxsd.xsd") // validate against xsd
.end()
.to(ACTIVEMQ_CONSUMER_PREFIX + TOPIC_ORDER_CREATED)
.log("Message sent to " + TOPIC_ORDER_CREATED);
Also if you are using onException in the route, then you should put that in the top of the route, not in the middle.
Related
I am setting up a Camel Route with ackMode=NONE meaning acknowlegements are not done automatically. How do I explicitly acknowledge the message in the route?
In my Camel Route definition I've set ackMode to NONE. According to the documentation, I should be able to manually acknowledge the message downstream:
https://github.com/apache/camel/blob/master/components/camel-google-pubsub/src/main/docs/google-pubsub-component.adoc
"AUTO = exchange gets ack’ed/nack’ed on completion. NONE = downstream process has to ack/nack explicitly"
However I cannot figure out how to send the ack.
from("google-pubsub:<project>:<subscription>?concurrentConsumers=1&maxMessagesPerPoll=1&ackMode=NONE")
.bean("processingBean");
My PubSub subscription has an acknowledgement deadline of 10 seconds and so my message keeps getting re-sent every 10 seconds due to ackMode=NONE. This is as expected. However I cannot find a way to manually acknowledge the message once processing is complete and stop the re-deliveries.
I was able to dig through the Camel components and figure out how it is done. First I created a GooglePubSubConnectionFactory bean:
#Bean
public GooglePubsubConnectionFactory googlePubsubConnectionFactory() {
GooglePubsubConnectionFactory connectionFactory = new GooglePubsubConnectionFactory();
connectionFactory.setCredentialsFileLocation(pubsubKey);
return connectionFactory;
}
Then I was able to reference the ack id of the message from the header:
#Header(GooglePubsubConstants.ACK_ID) String ackId
Then I used the following code to acknowledge the message:
List<String > ackIdList = new ArrayList<>();
ackIdList.add(ackId);
AcknowledgeRequest ackRequest = new AcknowledgeRequest().setAckIds(ackIdList);
Pubsub pubsub = googlePubsubConnectionFactory.getDefaultClient();
pubsub.projects().subscriptions().acknowledge("projects/<my project>/subscriptions/<my subscription>", ackRequest).execute();
I think it is best if you look how the Camel component does it with ackMode=AUTO. Have a look at this class (method acknowledge)
But why do you want to do this extra work? Camel is your fried to simplify integration by abstracting away low level code.
So when you use ackMode=AUTO Camel automatically commits your successfully processed messages (when the message has successfully passed the whole route) and rolls back your not processable messages.
I would like to know if the below is expected behaviour for Camel idempotent consumer:
I have removeOnFailure=true for the route, which means basically when the exchange fails idempotent consumer should remove the Identifier from the repository. This brings up a very interesting scenario which allows duplicate on the exchange.
Suppose I have identifier=12345 and first attempt to execute the exchange was Succesfull which means identifier is added to idempotent repository. Next attempt to use same identifier i.e 12345 fails as this is caught as Duplicate Message (CamelDuplicateMessage). But at this point having removeOnFailure=true will remove the identifier from the repository which on next attempt will allow the exchange to go through successfully without catching the default message. Hence, creating a room for duplication on the exchange.
Can someone advise if this is expected behaviour or some bug?
Sample Route:
from("direct:Route-DeDupeCheck").routeId("Route-DeDupeCheck")
.log(LoggingLevel.DEBUG, "~~~~~~~ Reached to Route-DeDupeCheck: ${property.xref}")
.idempotentConsumer(simple("${property.xref}"), MemoryIdempotentRepository.memoryIdempotentRepository()) //TODO: To replace with Redis DB for caching
.removeOnFailure(true)
.skipDuplicate(false)
.filter(exchangeProperty(Exchange.DUPLICATE_MESSAGE).isEqualTo(true))
.log("~~~~~~~ Duplicate Message Found!")
.to("amq:queue:{{jms.duplicateQueue}}?exchangePattern=InOnly") //TODO: To send this to Duplicate JMS Queue
.throwException(new AZBizException("409", "Duplicate Message!"));
Your basic premise is wrong.
Next attempt to use same identifier i.e 12345 fails as this is caught
as Duplicate Message (CamelDuplicateMessage)
When there is a duplicated message, it is not considered as a failure. It is just ignored from further processing(unless you have skipDuplicate option set to true).
Hence the scenario what you just explained cannot occur what so ever.
It is very easy to test. Considering you have a route like this,
public void configure() throws Exception {
//getContext().setTracing(true); Use this to enable tracing
from("direct:abc")
.idempotentConsumer(header("myid"),
MemoryIdempotentRepository.memoryIdempotentRepository(200))
.removeOnFailure(true)
.log("Recieved id : ${header.myid}");
}
}
And a Producer like this
#EndpointInject(uri = "direct:abc")
ProducerTemplate producerTemplate;
for(int i=0, i<5,i++) {
producerTemplate.sendBodyAndHeader("somebody","myid", "1");
}
What you see in logs is
INFO 18768 --- [tp1402599109-31] route1 : Recieved id : 1
And just once.
I have an issue with Camel.
I am creating dynamic queues like this
for (Client client : clients) {
// Get the ids
String clientId = client.getId();
String queueId = "jms:sendRoute" + clientId;
// Generate the queues
from(queueId)
.routeId(queueId)
.errorHandler(defaultErrorHandler().maximumRedeliveries(-1).redeliveryDelay(1000 * 60 * 5)) // Retries indefinitively
.transacted()
.process(new MyProcessor(clientId, clientService))
.to("mock:end");
}
Which are called by this initial route
from("direct:availabilityRoute")
.routeId("availabilityRoute")
.unmarshal(bindy)
.split(body())
.process(new AProcessor(controler))
.split(body())
.process(new DBProcessor(controler))
.process(new RoutesProcessor(controler))
.recipientList(header("clientIds"), ",")
.to("mock:end");
The route processor generates the list of queue ids to which a essage is multicasted
ids.append("jms:sendRoute" + client.getId());
and Ids (a string) is put in the message header
I am using activeMQ
I don't understand the behaviour
When it starts, it creates 3 routes dynamically. They are named:
sendRoute1
sendRoute2
sendRoute3
(this is what I see in the admin console)
The list of ids generated by the processor is: jms:sendRoute1,jms:sendRoute2, jms:sendRoute3
The issue I have is that when the message is sent to the multiple route, when I put a breakpoint I see that the first message is consumed by sendRoute1, and that 2 new routes named jms:sendRoute2 and jms:sendRoute3 have been added.
The message has been sent to:
- sendRoute1
- jms:sendRoute2
- jms:sendRoute3
But I have no consumer on jsm:sendRoute2 and jms:sendRoute3 so the messages are not read.
Could somebody explain me why this happens and how I can solve this?
Regards
Gilles
When a message is multicasted to all the queues in scope
From a programming point of view, I have a very simple business case. However, I can't figure out how to implement it using Apache Camel... Well, I have 2 JMS queues: one to receive commands, another - to store large number of message which should be delivered to external system in a batches of 1000 or less.
Here is the concept message exchange algorithm:
upon receiving a command message in 1st JMS queue I prepare XML
message
Send the XML message to external SOAP Web Service to obtain a usertoken
Using the usertoken, prepare another XML message and send it to a REST service to obtain jobToken
loop:
4.1. aggregate messages from 2nd JMS queue in batches of 1000, stop aggregation at timeout
4.2. for every batch, convert it to CSV file
4.3. send csv via HTTP Post to a REST service
4.4. retain batchtoken assigned to each batch
using the jobtoken prepare XML message and send to REST service to commit the batches
using batchtoken check execution status of each batch via XML message to REST service
While looking at Camel I could create a sample project where I can model out the exchange 1-3, 5:
from("file:src/data?noop=true")
.setHeader("sfUsername", constant("a#fd.com"))
.setHeader("sfPwd", constant("12345"))
.to("velocity:com/eip/vm/bulkPreLogin.vm?contentCache=false")
.setHeader(Exchange.CONTENT_TYPE, constant("text/xml; charset=UTF-8"))
.setHeader("SOAPAction", constant("login"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://bulklogin") // send login
.to("xslt:com/eip/xslt/bulkLogin.xsl") //xslt transformation to retrieve userToken
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String body = (String) exchange.getIn().getBody();
String[] bodyParts = body.split(",");
exchange.getProperties().put("userToken", bodyParts[0]);
.....
}
})
.to("velocity:com/eip/vm/jobInsertTeamOppStart.vm")
.setHeader(Exchange.CONTENT_TYPE, constant("application/xml; charset=UTF-8"))
.setHeader("X-Session", property("userToken"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://scheduleJob") //schedule job
.to("xslt:com//eip/xslt/jobInfoTransform.xsl")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
String body = (String) exchange.getIn().getBody();
exchange.getProperties().put("jobToken",body.trim());
}
})
//add batches in a loop ???
.to("velocity:com/eip/vm/jobInsertTeamOppEnd.vm")
.setHeader(Exchange.HTTP_URI, simple("https://na15.com/services/async/job/${property.jobToken}"))
.setHeader(Exchange.CONTENT_TYPE, constant("application/xml; charset=UTF-8"))
.setHeader("X-ID-Session", property("userToken"))
.setHeader("CamelHttpMethod", constant("POST"))
.to("http4://closeJob") //schedule job
//check batch?
.bean(new SomeBean());
So, my question is:
How can I read messages from my 2nd JMS queue?
This doesn't strike me as a very good use-case for a single camel route. I think you should implement the main functionality in a POJO and use Camels Bean Integration for consuming and producing messages. This will result in much more easy to maintain code, and also for easier Exception handling.
See https://camel.apache.org/pojo-consuming.html
Hope this doesn't sound ridiculous, but how can I discard a message in Camel on purpose?
Until now, I sent them to the Log-Component, but meanwhile I don't even want to log the withdrawal.
Is there a /dev/null Endpoint in Camel?
You can use the message filter eip to filter out unwanted messages.
http://camel.apache.org/message-filter
There is no dev/null, component.
Also there is a < stop /> you can use in the route, and when a message hit that, it will stop continue routing.
And the closest we got on a dev/null, is to route to a log, where you set logLeve=OFF as option.
With credit to my colleague (code name: cayha)...
You can use the Stub Component as a camel endpoint that is equivalent to /dev/null.
e.g.
activemq:route?abc=xyz
becomes
stub:activemq:route?abc=xyz
Although I am not aware of the inner workings of this component (and if there are dangers for memory leaks, etc), it works for me and I can see no drawbacks in doing it this way.
one can put uri/mock-uri to the config using property component
<camelContext ...>
<propertyPlaceholder id="properties" location="ref:myProperties"/>
</camelContext>
// properties
cool.end=mock:result
# cool.end=result
// route
from("direct:start").to("properties:{{cool.end}}");
I'm a little late to the party but you can set a flag on the exchange and use that flag to skip only that message (by calling stop) if it doesn't meet your conditions.
#Override
public void configure() throws Exception {
from()
.process(new Processor() {
#SuppressWarnings("unchecked")
#Override
public void process(Exchange exchange) throws Exception {
exchange.setProperty("skip", false);
byte[] messageBytes = exchange.getIn().getBody(byte[].class);
if (<shouldNotSkip>) {
} else { //skip
exchange.setProperty("skip", true);
}
}
}).choice()
.when(exchangeProperty("skip").isEqualTo(true))
.stop()
.otherwise()
.to();
}
I am using activemq route and needs to send reply in normal cases, so exchange pattern is InOut. When I configure a filter in the route I find that even it does not pass message to next step, the callback is executed(sending reply), just same as the behavior when calling stop(). And it will send the same message back to reply queue, which is not desirable.
What I do is to change the exchange pattern to InOnly conditionally and stop if I want to filter out the message, so reply is not sent. MAIN_ENDPOINT is a direct:main endpoint I defined to include normal business logic.
from("activemq:queue:myqueue" + "?replyToSameDestinationAllowed=true")
.log(LoggingLevel.INFO, "Correlation id is: ${header.JMSCorrelationID}; will ignore if not null")
.choice()
.when(simple("${header.JMSCorrelationID} == null"))
.to(MAIN_ENDPOINT)
.endChoice()
.otherwise()
.setExchangePattern(ExchangePattern.InOnly)
.stop()
.endChoice()
.end();
Note that this message is also consumed and not in the queue anymore. If you want to preserve the message in the queue(not consuming it), you may just stop() or just filter() so the callback(sending reply which is the original message) works, putting the message back to the queue.
Using only filter() would be much simpler:
from("activemq:queue:myqueue" + "?replyToSameDestinationAllowed=true")
.log(LoggingLevel.INFO, "Correlation id is: ${header.JMSCorrelationID}; will ignore if not null")
.filter(simple("${header.JMSCorrelationID} == null"))
.to(MAIN_ENDPOINT);