We defined a route in Camel with split and aggregate functionality, but can't propagate the exception back to split after the aggregator. Which cause's the split to run even if we encounter an exception
below is the code which is not working
from("direct:MyRoute")
.routeId("MyRouteID")
.split().tokenize("\n", 1)
.streaming().stopOnException()
.choice()
.when(simple("${property.CamelSplitIndex} > 0"))
.unmarshal(domainDataFormat)
.choice()
.when(simple("${property.CamelSplitComplete}"))
.process(
new Processor()
{
#Override
public void process(Exchange exchange) throws Exception
{
exchange.getIn().getHeaders().put(Exchange.AGGREGATION_COMPLETE_ALL_GROUPS_INCLUSIVE, true);
}
}
)
.end()
.aggregate(myAggregationStrategy).constant(true) //if i comment this line split will be stop on exception
.threads().executorService(executorService)
.process(myProcessor).end()
.end();
Processor(myProcessor) from the above code is below:
counter.incrementAndGet(); //atomic counter
if(counter.get()==3)
{
exchange.setException(new RuntimeException());
throw new RuntimeCamelException();
}
But, the moment I remove the aggregate from the route, Split is able to stop the route on Exception.
Use the composite message processor EIP which is split + aggregate (fork/join) in the same unit of work.
See docs at: https://camel.apache.org/components/next/eips/composed-message-processor.html
And see the splitter only section where you can specify an aggregation strategy to the splitter to have it work together.
Related
I have the following route:
from("file:/home/tmp/test?move=.done")
.routeId("file")
.split(body().tokenize("\n"),new GroupedBodyAggregationStrategy())
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getMessage().setHeader("Test", "test");
System.out.println(exchange.getIn().getBody());
}
})
and the file that is being consumed is:
1234,56676,2345
234,213,412
124,423,5236
I would expect only one exchange to reach the processor after split and that its body contains all the above lines, however it behaves like I do not use an aggregation strategy at all. 3 exchanges reach the processor each with the body that corresponds to a single line. Also, it does not matter if I use GroupedBodyAggregationStrategy or a custom one, it is always the same. What am I doing wrong?
Give a look here enter link description here
What you look is :
from("file:/home/tmp/test?move=.done")
.routeId("file")
.split(body().tokenize("\n"),new GroupedBodyAggregationStrategy())
// instructions on each message from the split (.ie each row here)
.end()
// instructions on aggregated result
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getMessage().setHeader("Test", "test");
System.out.println(exchange.getIn().getBody());
}
})
I've faced with behavior that I can't understand. This issue happens when Split with AggregationStrategy is executed and during one of the iterations, an exception occurs. An exception occurs inside of Splitter in another route (direct endpoint which is called for each iteration). Seems like route execution stops just after Splitter.
Here is sample code.
This is a route that builds one report per each client and collects names of files for internal statistics.
#Component
#RequiredArgsConstructor
#FieldDefaults(level = PRIVATE, makeFinal = true)
public class ReportRouteBuilder extends RouteBuilder {
ClientRepository clientRepository;
#Override
public void configure() throws Exception {
errorHandler(deadLetterChannel("direct:handleError")); //handles an error, adds error message to internal error collector for statistic and writes log
from("direct:generateReports")
.setProperty("reportTask", body()) //at this point there is in the body an object of type ReportTask, containig all data required for building report
.bean(clientRepository, "getAllClients") // Body is a List<Client>
.split(body())
.aggregationStrategy(new FileNamesListAggregationStrategy())
.to("direct:generateReportForClient") // creates report which is saved in the file system. uses the same error handler
.end()
//when an exception occurs during split then code after splitter is not executed
.log("Finished generating reports. Files created ${body}"); // Body has to be List<String> with file names.
}
}
AggregationStrategy is pretty simple - it just extracts the name of the file. If the header is absent it returns NULL.
public class FileNamesListAggregationStrategy extends AbstractListAggregationStrategy<String> {
#Override
public String getValue(Exchange exchange) {
Message inMessage = exchange.getIn();
return inMessage.getHeader(Exchange.FILE_NAME, String.class);
}
}
When everything goes smoothly after splitting there is in the Body List with all file names. But when in the route "direct:generateReportForClient" some exception occurred (I've added error simulation for one client) than aggregated body just contains one less file name -it's OK (everything was aggregated correctly).
BUT just after Split after route execution stops and result that is in the body at this point (List with file names) is returned to the client (FluentProducer) which expects ReportTask as a response body.
and it tries to convert value - List (aggregated result) to ReportTask and it causes org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type
Why route breaks after split? All errors were handled and aggregation finished correctly.
PS I've read Camel In Action book and Documentation about Splitter but I haven't found the answer.
PPS project runs on Spring Boot 2.3.1 and Camel 3.3.0
UPDATE
This route is started by FluentProducerTemplate
ReportTask processedReportTask = producer.to("direct:generateReports")
.withBody(reportTask)
.request(ReportTask.class);
The problem is error handler + custom aggregation strategy in the split.
From Camel in Action book (5.3.5):
WARNING When using a custom AggregationStrategy with the Splitter,
it’s important to know that you’re responsible for handling
exceptions. If you don’t propagate the exception back, the Splitter
will assume you’ve handled the exception and will ignore it.
In your code, you use the aggregation strategy extended from AbstractListAggregationStrategy. Let's look to aggregate method in AbstractListAggregationStrategy:
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
List<V> list;
if (oldExchange == null) {
list = getList(newExchange);
} else {
list = getList(oldExchange);
}
if (newExchange != null) {
V value = getValue(newExchange);
if (value != null) {
list.add(value);
}
}
return oldExchange != null ? oldExchange : newExchange;
}
If a first exchange is handled by error handler we will have in result exchange (newExchange) number of properties set by Error Handler (Exchange.EXCEPTION_CAUGHT, Exchange.FAILURE_ENDPOINT, Exchange.ERRORHANDLER_HANDLED and Exchange.FAILURE_HANDLED) and exchange.errorHandlerHandled=true. Methods getErrorHandlerHandled()/setErrorHandlerHandled(Boolean errorHandlerHandled) are available in ExtendedExchange interface.
In this case, your split finishes with an exchange with errorHandlerHandled=true and it breaks the route.
The reason is described in camel exception clause manual
If handled is true, then the thrown exception will be handled and
Camel will not continue routing in the original route, but break out.
To prevent this behaviour you can cast your exchange to ExtendedExchange and set errorHandlerHandled=false in the aggregation strategy aggregate method. And your route won't be broken but will be continued.
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
Exchange aggregatedExchange = super.aggregate(oldExchange, newExchange);
((ExtendedExchange) aggregatedExchange).setErrorHandlerHandled(false);
return aggregatedExchange;
}
The tricky situation is that if you have exchange handled by Error Handler as not a first one in your aggregation strategy you won't face any issue. Because camel will use the first exchange(without errorHandlerHandled=true) as a base for aggregation.
In my camel route I want to send part of the output (json path is $._attachments) to another endpoint. This works. However, in a few corner cases it appears some of the json objects do not have this element. I guessed this would result in a CamelSplitSize being 0, and tried to test on that; however, the DefaultErrorHandler seems to kick in, rather than completing the route. What am I doing wrong here?
public class DiscoverAttachmentsFromCouchResponseRoute extends RouteBuilder {
#Override
public void configure() throws Exception
{
JsonPathExpression jsonPathExpression = new JsonPathExpression("$._attachments");
Predicate no_attachments = header("CamelSplitSize").isEqualTo("0");
errorHandler(
deadLetterChannel("broker1:queue:dlq.jsonobject.queue")
.maximumRedeliveries(3)
);
from("broker1:queue:input.jsonobject.queue")
.routeId("DiscoverAttachmentsFromCouchResponseRoute")
.threads(2)
.split(jsonPathExpression)
.marshal().json(JsonLibrary.Jackson,true)
.to("broker1:queue:with.attachments.queue")
.choice()
.when(no_attachments)
.log("No attachments found.")
.to("broker1:queue:no.attachments.queue")
.otherwise()
.log("A grand total of '${header.CamelSplitSize}' attachments were found.")
.endChoice();
}
}
Edit somethimes the answer is too obvious... I was looking at the wrong place. Why do I always seem to find an answer a few minutes after I asked it? Am direly in need of a rubber duck.
I changed the route a bit; Rather than handling the issue after the split, I should do it prior to the split:
.choice()
.when().jsonpath("$._attachments", true)
.to("vm://withattach")
.otherwise()
.to("vm://withoutattach")
.endChoice();
Then I only needed two new consumers from the vm://* and do the logic there. This seems to work. In jsonpath I can skip the exception which I cannot do in the JsonPathExpression nor in the split();
I'd like to split exchange message body (it's list of MyCustomClass object), process them (one by one) and them aggregate the all exchanges together. Split is ok, process one by one also ok, but I can't figure out how to aggregate them.
from("mysource")
.unmarshal(new ListJacksonDataFormat(MyClass.class))
.split().body()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
// process MyClass item
exchange.getIn().setBody(processedItem);
}
})
.to("destinationForProcessedItem")
.aggregate(new GroupedExchangeAggregationStrategy()) <== Seems like problem is here
.process(new Processor() {
// handle result of aggregation
})
I don't need complicated aggregation, just collect list of splitted Exchanges and handle them in the final processor.
Use the built-in aggregator in the splitter, see the composed message processor EIP pattern: https://camel.apache.org/components/latest/eips/composed-message-processor.html#_sample.
write like this
.aggregate(new AggregationStrategy() {
#Override
public Exchange aggregate(Exchange exchange, Exchange exchange1) {
//logic for aggregation using exchnage and exchange1
}
})
Hope this doesn't sound ridiculous, but how can I discard a message in Camel on purpose?
Until now, I sent them to the Log-Component, but meanwhile I don't even want to log the withdrawal.
Is there a /dev/null Endpoint in Camel?
You can use the message filter eip to filter out unwanted messages.
http://camel.apache.org/message-filter
There is no dev/null, component.
Also there is a < stop /> you can use in the route, and when a message hit that, it will stop continue routing.
And the closest we got on a dev/null, is to route to a log, where you set logLeve=OFF as option.
With credit to my colleague (code name: cayha)...
You can use the Stub Component as a camel endpoint that is equivalent to /dev/null.
e.g.
activemq:route?abc=xyz
becomes
stub:activemq:route?abc=xyz
Although I am not aware of the inner workings of this component (and if there are dangers for memory leaks, etc), it works for me and I can see no drawbacks in doing it this way.
one can put uri/mock-uri to the config using property component
<camelContext ...>
<propertyPlaceholder id="properties" location="ref:myProperties"/>
</camelContext>
// properties
cool.end=mock:result
# cool.end=result
// route
from("direct:start").to("properties:{{cool.end}}");
I'm a little late to the party but you can set a flag on the exchange and use that flag to skip only that message (by calling stop) if it doesn't meet your conditions.
#Override
public void configure() throws Exception {
from()
.process(new Processor() {
#SuppressWarnings("unchecked")
#Override
public void process(Exchange exchange) throws Exception {
exchange.setProperty("skip", false);
byte[] messageBytes = exchange.getIn().getBody(byte[].class);
if (<shouldNotSkip>) {
} else { //skip
exchange.setProperty("skip", true);
}
}
}).choice()
.when(exchangeProperty("skip").isEqualTo(true))
.stop()
.otherwise()
.to();
}
I am using activemq route and needs to send reply in normal cases, so exchange pattern is InOut. When I configure a filter in the route I find that even it does not pass message to next step, the callback is executed(sending reply), just same as the behavior when calling stop(). And it will send the same message back to reply queue, which is not desirable.
What I do is to change the exchange pattern to InOnly conditionally and stop if I want to filter out the message, so reply is not sent. MAIN_ENDPOINT is a direct:main endpoint I defined to include normal business logic.
from("activemq:queue:myqueue" + "?replyToSameDestinationAllowed=true")
.log(LoggingLevel.INFO, "Correlation id is: ${header.JMSCorrelationID}; will ignore if not null")
.choice()
.when(simple("${header.JMSCorrelationID} == null"))
.to(MAIN_ENDPOINT)
.endChoice()
.otherwise()
.setExchangePattern(ExchangePattern.InOnly)
.stop()
.endChoice()
.end();
Note that this message is also consumed and not in the queue anymore. If you want to preserve the message in the queue(not consuming it), you may just stop() or just filter() so the callback(sending reply which is the original message) works, putting the message back to the queue.
Using only filter() would be much simpler:
from("activemq:queue:myqueue" + "?replyToSameDestinationAllowed=true")
.log(LoggingLevel.INFO, "Correlation id is: ${header.JMSCorrelationID}; will ignore if not null")
.filter(simple("${header.JMSCorrelationID} == null"))
.to(MAIN_ENDPOINT);