Is it the designed and desired behavior that makes Camel routes to not execute onCompletion tasks in a pipeline with several SEDA queues? - apache-camel

I stumbled upon a problem with not working onCompletion between the routes that pass a message over SEDA queues.
The route configuration is similar to this simplified version:
from("direct:a")
.onCompletion().log("a - done").end()
.to("seda:b");
from("seda:b")
.onCompletion().log("b - done").end()
.to("seda:c");
from("seda:c")
.onCompletion().log("c - done").end()
.to("seda:d");
from("seda:d")
.onCompletion().log("d - done").end()
.to("mock:end");
With this configuration, I get only "d - done" logging.
I debugged the execution and noticed that the onCompletion handler from "a", "b", and "c" don't get executed because they are route-scoped and get attempted to be executed in the scope of the next route.
This happens because they get handed over from the initial exchange to an exchange prepared for the next route. It happens in the SedaProducer::addToQueue method with copy parameter defined as true, which makes the ::prepareCopy method being called, which in its turn calls ExchangeHelper.createCorrelatedCopy with handover defined as true.
It seems to me like a bug, because looking at the routes configuration I'd expect different behavior: all the onCompletion tasks get executed reporting on routes finalization. Though maybe I'm missing something here, and if this is the case then I would appreciate you guys helping me to find out the missing details.

The workaround I implemented is ugly but does the trick. Before sending to the next route (queue) we hand over completions to a holder exchange and after sending we hand over them back from the holder.
Here is a code example:
route
.process(exchange -> {
var holder = new DefaultExchange(exchange.getContext());
exchange.adapt(ExtendedExchange.class).handoverCompletions(holder);
exchange.setProperty(SYNC_HOLDER_PROPERTY, holder);
})
.to("seda://next")
.process(exchange -> exchange
.getProperty(SYNC_HOLDER_PROPERTY, ExtendedExchange.class)
.handoverCompletions(exchange));

Related

How to fix several input per route in Camel 3.X.X?

I have a route that looks like this:
from(URL_A)
.from(URL_B)
.to(URL_C)
.process(...)
// logging
.to(URL_D)
This route works perfectly in Camel 2.X.X but not in 3.7.X
The error message I get:
Only one input is allowed per route. Cannot accept input: From[direct:ABCD]
I checked the migration guide, but I cannot get how to migrate this sort of route.
Do you have any idea how to tackle it further?
I think you can use direct component: https://camel.apache.org/components/3.4.x/direct-component.html
For example:
from(URL_A)
.to(direct:collector)
from(URL_B)
.to(direct:collector)
from(direct:collector)
.to(URL_C)
.process(...)
// logging
.to(URL_D)
#Stepan Shcherbakov suggested a solution, below will be an enhancement of it:
String [] sources = {URL_A, URL_B}
for (String source : sources) {
from(source)
.to(direct:collector)
}
from(direct:collector)
.to(URL_C)
.process(...)
// logging
.to(URL_D)

How to dynamically return a from endpoint in apache camel DSL

Here is my code
from("google-pubsub:123:subscription1?maxMessagesPerPoll=3 & concurrentConsumers=5" ).routeId("myroute")
.process(new ProducerProcessor())
to("google-pubsub:123:topic1")
;
In my code above ,the from channel I want to make it generic.Basically it should be able to consume data from good-pubsub or may be from a file or from a JMS queue.Hence depending upon a parameter I want to return
a different from channel.Something like below
private RouteDefinition fromChannel(String parameter) {
if (parameter is "google" then
return from("google-pubsub:123:subscription1?maxMessagesPerPoll=3 & concurrentConsumers=5" )
if (parameter is "file" then
return from(/my/fileFolder/)).split(body().tokenize("\n")).streaming().parallelProcessing();
}
I tried this but I am getting null pointer exception in the fromChannel method.Please let me know if you have better ideas.
Rewrite based on comment
You can for example create a (static) template route for every input type and generate the routes based on a configured endpoint list.
I described such an endpoint configuration and route generation scenario in this answer.
Like this you can generate the split part for every file route and any other specialty for other route types.
All these input routes are routing at their end to a common processing route
.from(pubsubEndpoint)
.to("direct:genericProcessingRoute")
.from(fileEndpoint)
.split(body()
.tokenize("\n"))
.streaming()
.parallelProcessing()
.to("direct:genericProcessingRoute")
.from("direct:genericProcessingRoute")
... [generic processing]
.to("google-pubsub:123:topic1")
The multiple input (and output) routes around a common core route is called hexagonal architecture and Camel fits very well into this.

Options vs Query Parameters in Apache Camel

Apache camel components page have fields under option and query parameters but no clear path position for parameters , from samples I was able to make out they go along options .
I would like to know the difference between options and query parameters.
When an application with Apache Camel starts, it registers the routes in the Camel Context, and once the context is started, components present in from () and to () cannot be modified, for example:
String param = "a = xxxx & y = bbb";
...
to ("http4: //api.xxx.yy?" + stop)
...
It will only be evaluated at startup, so even if the value of the string in the param variable changes, the route will always use a=xxxx&y=bbb as the default since it has already been initialized in the context of Camel (you can see Camel recording the routes in the logs during application startup).
The options can undergo changes not only in construction, depending on the design of the component in question, but can also be exposed for configuration via starters using application.yml or application.properties or via java as in the example below:
application.properties
camel.component.http.http-configuration=br.com.pack.impl.MyHttpConfiguration
In java
HttpConfiguration config = new HttpConfiguration();
config.setProxyAuthMethod("Digest");
config.setProxyAuthUsername("myUser");
config.setProxyAuthPassword("myPassword");
HttpComponent http = context.getComponent("http", HttpComponent.class);
http.setHttpConfiguration(config);
from("direct:start")
.to("http4://www.google.com/search");
I hope it helped to clarify a little more
Options are used to configure Component and Query Parameters are used while creating endpoints.

PollingConsumer using smb protocol and pollEnrich()

Problem: The file is not consumed from the server
I am using
from("test")
.routeId("test")
.pollEnrich()
.simple("smb://myUrl?password=test&fileName=${in.headers.test}")
.aggregationStrategy((Exchange oldExchange, Exchange newExchange) -> {
//do things
return newExchange;
})
I have no error, I am sure that the url is ok, because when I am using the same url in the from(), the file gets consumed.
I don't understand what is happening here, I am using camel 2.24.0 and camel-extra:camel-jcifs:2.23.1. I have tried to use smb2 using the library from github.jborza.camel-smbj, still the same outcome.
I tried to debug, I can see in the GenericFileComponent class in the createEndpoint method, that the endpoint is correctly created, then I tried (in debug mode) to get the exchanges from my endpoint, I can get them successfully, further this will be a SmbEndpoint, when I try to get the exchanges from my smbEndpoint it returns exactly the needed file from the server, further a EventDrivenPollingConsumer is created for this endpoint, I had a look at it, is started (seems ok). When it hits the consumer.receive() from the PollEnricher it blocks, no file is consumed. I tried using a timeout, than returns null, so somehow cannot find the file, or the consumer is wrong, I honestly have no clue at this point.
I had a look here too: https://github.com/apache/camel/blob/b9a3117f19dd19abd2ea8b789c42c3e86fe4c488/core/camel-core/src/test/java/org/apache/camel/component/file/FileConsumePollEnrichFileTest.java
and I have played with delays
&consumer.initialDelay=100&consumer.delay=100&consumer.bridgeErrorHandler=true
Then I tried to implement with processor like here:
https://github.com/apache/camel/blob/b9a3117f19dd19abd2ea8b789c42c3e86fe4c488/core/camel-core/src/test/java/org/apache/camel/component/file/FileConsumePollEnrichFileUsingProcessorTest.java
The same result :(
At some point the file was consumed, suddenly, but this happened only once, I cannot understand this behavior.
Sounds that you have a readlock problem, can you find any files in .done with the same name as the file you try to consume?

camel-smooks returns null in body

I am using talend-ESB and want to parse EDI message to XML using smooks & I am getting null in body. The code looks as below.
from(
"file://D:/cimt/InvoiceEDI_Mapping/" + "?noop=true"
+ "&autoCreate=true" + "&flatten=false"
+ "&fileName=InDev_EDI_Msg.txt" + "&bufferSize=128")
.routeId("TestSmooksConfig_cFile_1")
.log(org.apache.camel.LoggingLevel.WARN,
"TestSmooksConfig.cLog_1", "${body}")
.id("TestSmooksConfig_cLog_1")
.to("smooks://EDI_Config.xml")
.to("log:TestSmooksConfig.cLog_2" + "?level=WARN")
.id("TestSmooksConfig_cLog_2");
}
My Talend route looks as below.
I used following set of external dependencies.
milyn-commons-1.7.0.jar
milyn-smooks-camel-1.7.0.jar
milyn-smooks-edi-1.7.0.jar
milyn-smooks-core-1.7.0.jar
jaxen-1.1.6.jar
milyn-edisax-parser-1.4.jar
Also, I see a strange behavior that, upon execution, I still see "starting" prior to cJavaDSLProcessor, which initially made me wonder if at all it gets executed. But later, when I intentionally made a mistake in EDI-Mapping, then the route was throwing errors, which kind of convinced me that it does parse the EDI message.
I did also search before posting this question here, and found a similar problem in this link
And I tried to lower my revision of org.milyn.* jars to 1.4.0, and got an exception that the route could not register smooks component. So I continued using 1.7.0 version of org.milyn.* jars.
For the benefit of others who might bump into similar issue, I 'assume' that the output of the smooks gets written into an Object of type StringResult.class. However, in my initial implementation, there was no such option and hence the output body was null.
Later, I tried alternative approach from http://smooks.org/guide where they used processor endpoint.Actually they had even made a statement that the data could be retrieved through exports element. The below code snippet helped to fix issue.
Smooks smooks = new Smooks("edi-to-xml-smooks-config.xml");
ExecutionContext context = smooks.createExecutionContext();
smooks.setExports(new Exports(StringResult.class));
SmooksProcessor processor = new SmooksProcessor(smooks, context);
from("file://input?noop=true")
.process(processor)
.to("mock:result");

Resources