In my Apache Camel application I have multiple conditions checking the existence of keys in a JSON. I want to reduce the boiler plate code, therefore I need to transform my Expressions to a Predicate.
My code with Expressions:
.choice()
.when().jsonpath("$.score", true).to("direct:b")
.when().jsonpath("$.points", true).to("direct:b")
.otherwise().to("direct:c");
See also: JSONPATH
My code with Predicates:
.choice()
.when(PredicateBuilder.or(jsonpath("$.score", true), jsonpath("$.points", true))).to("direct:b")
.otherwise().to("direct:c");
See also: PREDICATES
But this is not working, because there is no suppressExceptions parameter (see BuilderSupport#jsonpath). Unfortunately, there is also no exists mehod (see ValueBuilder).
How can I write a Predicate for checking the existence of a key in a JSON?
this code solve your problem .
.choice()
.when(PredicateBuilder.and(jsonpath("$[?(#.score)]"), jsonpath("$.score"))).to("direct:b")
.when(PredicateBuilder.and(jsonpath("$[?(#.points)]"), jsonpath("$.points"))).to("direct:b")
.otherwise().to("direct:c");
Related
is there a way to assign uid to a window function (such as apply(ApplyCustomFunction)) as we do for map/flatmap (or other) functions in Flink. The Flink version is 1.13.1.
I would like to specify the case with an example
DataStream<RECORD> outputDataStream = dataStream
.coGroup(otherDataStream)
.where(DATA::getKey)
.equalTo(OTHERDATA::getKey)
.window(TumblingProcessingTimeWindows.of(Time.seconds(2)))
.apply(new CoGroupFunction());
Thanks
CoGroupedStreams.WithWindow#apply(CoGroupFunction<T1,T2,T>) doesn't have the return type that's needed for setting a UID or per-operator parallelism (among other things). This was done in order to keep binary backwards compatibility, and can't be fixed before Flink 2.0.
You can work around this by using the (deprecated) with method instead of apply, as in
DataStream<RECORD> outputDataStream = dataStream
.coGroup(otherDataStream)
.where(DATA::getKey)
.equalTo(OTHERDATA::getKey)
.window(TumblingProcessingTimeWindows.of(Time.seconds(2)))
.with(new CoGroupFunction())
.uid("window");
The with method will be removed once it is no longer needed.
Use with() instead of apply(). It will be fixed in 2.0 version, how it sayed in documentation
Is there a way to do smth like this to work? I am talking about the condition inside when.
.choice()
.when(Exchange::isFailed)
.to(direct(URI_DEADLETTER))
I tried:
.when(method(Exchange.class, "isFailed"))
.when().exchange(Exchange::isFailed)
For the first solution an error is thrown and the second is not working.
I know that I can create a new class and a method inside, from here: How do i use java boolean condition in camel route?
And I read about the predicat here: http://www.davsclaus.com/2009/02/apache-camel-and-using-compound.html.
But without using a new class or predicat, is there a way that I can achieve this?
A lazy solution is to use Camel simple language (http://camel.apache.org/simple.html) which allows you to access anything (headers, properties, body, method, etc..) of current exchange
.choice()
.when( simple("${exception} != null") )
A more OO solution would be to use Camel Predicate (Builder):
Predicate condition1 = ...
Predicate condition2 = ...;
Predicate isFailed = PredicateBuilder.or(condition1, condition2);
.choice()
.when( isFailed )
As the title suggests, I want to move a file into a different folder after I am done writing DB records to to it.
I have already looked into several questions related to this: Apache camel file with doneFileName
But my problem is a little different since I am using split, stream and parallelProcessing for getting the DB records and writing to a file. I am not able to know when and how to create the done file along with the parallelProcessing. Here is the code snippet:
My route to fetch records and write it to a file:
from(<ROUTE_FETCH_RECORDS_AND_WRITE>)
.setHeader(Exchange.FILE_PATH, constant("<path to temp folder>"))
.setHeader(Exchange.FILE_NAME, constant("<filename>.txt"))
.setBody(constant("<sql to fetch records>&outputType=StreamList))
.to("jdbc:<endpoint>)
.split(body(), <aggregation>).streaming().parallelProcessing()
.<some processors>
.aggregate(header(Exchange.FILE_NAME), (o, n) -> {
<file aggregation>
return o;
}).completionInterval(<some time interval>)
.toD("file://<to the temp file>")
.end()
.end()
.to("file:"+<path to temp folder>+"?doneFileName=${file:header."+Exchange.FILE_NAME+"}.done"); //this line is just for trying out done filename
In my aggregation strategy for the splitter I have code that basically counts records processed and prepares the response that would be sent back to the caller.
And in my other aggregate outside I have code for aggregating the db rows and post that writing into the file.
And here is the file listener for moving the file:
from("file://<path to temp folder>?delete=true&include=<filename>.*.TXT&doneFileName=done")
.to(file://<final filename with path>?fileExist=Append);
Doing something like this is giving me this error:
Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot store file: <folder-path>/filename.TXT] org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: <folder-path>/filename.TXT
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:292)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)[209:org.apache.camel.camel-core:2.16.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_144]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.io.InputStream but has value: Total number of records discovered: 5
What am I doing wrong? Any inputs will help.
PS: Newly introduced to Apache Camel
I would guess that the error comes from .toD("file://<to the temp file>") trying to write a file, but finds the wrong type of body (String Total number of records discovered: 5 instead of InputStream.
I don't understand why you have one file-destinations inside the splitter and one outside of it.
As #claus-ibsen suggested try to remove this extra .aggregate(...) in your route. To split and re-aggregate it is sufficient to reference the aggregation strategy in the splitter. Claus also pointed to an example in the Camel docs
from(<ROUTE_FETCH_RECORDS_AND_WRITE>)
.setHeader(Exchange.FILE_PATH, constant("<path to temp folder>"))
.setHeader(Exchange.FILE_NAME, constant("<filename>.txt"))
.setBody(constant("<sql to fetch records>&outputType=StreamList))
.to("jdbc:<endpoint>)
.split(body(), <aggregationStrategy>)
.streaming().parallelProcessing()
// the processors below get individual parts
.<some processors>
.end()
// The end statement above ends split-and-aggregate. From here
// you get the re-aggregated result of the splitter.
// So you can simply write it to a file and also write the done-file
.to(...);
However, if you need to control the aggregation sizes, you have to combine splitter and aggregator. That would look somehow like this
from(<ROUTE_FETCH_RECORDS_AND_WRITE>)
.setHeader(Exchange.FILE_PATH, constant("<path to temp folder>"))
.setHeader(Exchange.FILE_NAME, constant("<filename>.txt"))
.setBody(constant("<sql to fetch records>&outputType=StreamList))
.to("jdbc:<endpoint>)
// No aggregationStrategy here so it is a standard splitter
.split(body())
.streaming().parallelProcessing()
// the processors below get individual parts
.<some processors>
.end()
// The end statement above ends split. From here
// you still got individual records from the splitter.
.to(seda:aggregate);
// new route to do the controlled aggregation
from("seda:aggregate")
// constant(true) is the correlation predicate => collect all messages in 1 aggregation
.aggregate(constant(true), new YourAggregationStrategy())
.completionSize(500)
// not sure if this 'end' is needed
.end()
// write files with 500 aggregated records here
.to("...");
I'm tasked with creating a Camel route using Camel version 2.20.0 that takes a line in from a CSV file uses a value from that line in the SQL statement where clause and merges the results and outputs them again. If I hardcode the identifier in the SQL statement it works fine, if I try and use a dynamic URI I get an error.
The route is:
from("file:///tmp?fileName=test.csv")
.split()
.tokenize("\n")
.streaming()
.parallelProcessing(true)
.setHeader("userID", constant("1001"))
//.enrich("sql:select emplid,name from employees where emplid = '1001'",
.enrich("sql:select name from employees where emplid = :#userID",
new AggregationStrategy() {
public Exchange aggregate(Exchange oldExchange,
Exchange newExchange) {...
As I said if I uncomment the line with the hardcoded 1001 it queries the db and works as expected. However using the ':#userID' syntax I get an Oracle error of:
java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route3 ] [route3 ] [file:///tmp?fileName=test.csv ] [ 43]
[route3 ] [log5 ] [log ] [ 2]
[route3 ] [setHeader2 ] [setHeader[userID] ] [ 0]
[route3 ] [enrich2 ] [enrich[constant{sql:select name from employees where emplid = :#userID] [ 40]
The table is clearly there because it works when the value is hardcoded so it's got something to do with passing in the dynamic value. I've tried lots of variations on how to pass that variable in, inside single quotes, using values from the body instead of headers, etc. and haven't found the working combination yet though I've seen lots of similar seemingly working examples.
I've turned trace on it appears the header is correctly set as well:
o.a.camel.processor.interceptor.Tracer : >>> (route3) setHeader[userID, 1001] --> enrich[constant{sql:select name from employees where emplid = :#userID}] <<< Pattern:InOnly, Headers:{CamelFileAbsolute=true, CamelFileAbsolutePath=/tmp/test.csv, CamelFileLastModified=1513116018000, CamelFileLength=26, CamelFileName=test.csv, CamelFileNameConsumed=test.csv, CamelFileNameOnly=test.csv, CamelFileParent=/tmp, CamelFilePath=/tmp/test.csv, CamelFileRelativePath=test.csv, userID=1001}, BodyType:String, Body:1001,SomeValue,MoreValues
What needs to change to make this work?
I should also note I've tried this approach, using various syntax options to refer to the header value, without any luck:
.enrich().simple("sql:select * from employees where emplid = :#${in.header.userID}").aggregate ...
From the docs:
From Camel 2.16 onwards both enrich and pollEnrich supports dynamic endpoints that uses an Expression to compute the uri, which allows to use data from the current Exchange. In other words all what is told above no longer apply and it just works.
As you are using 2.20, I think you may try this example:
from("file:///tmp?fileName=test.csv")
.split()
.tokenize("\n")
.streaming()
.parallelProcessing(true)
.setHeader("userID", constant("1001"))
//.enrich("sql:select emplid,name from employees where emplid = '1001'",
.enrich("sql:select name from employees where emplid = ':#${in.header.userID}'",
new AggregationStrategy() {
public Exchange aggregate(Exchange oldExchange,
Exchange newExchange) {...
Take a look at the Expression topic in docs for further examples.
To sum up, the expression could be:
"sql:select name from employees where emplid = ':#${in.header.userID}'"
EDIT:
Sorry, I've missed the :# suffix. You could see a unit test working here.
Just take care with the columns types. If it's a integer, you shouldn't need the quotes.
Cheers!
From the Camel docs:
pollEnrich or enrich does not access any data from the current
Exchange which means when polling it cannot use any of the existing
headers you may have set on the Exchange.
The recommended way of achieving what you want is to instead use the recipientList, so I suggest you read up on that.
Content Enricher
Recipient List
Edit:
As Ricardo Zanini rightly pointed out in his answer it is actually possible to achieve this with Camel-versions from 2.16 onwards. As the OP is using 2.20 my answer is invalid.
I will, however, keep my answer but want to point out that this is only valid if you're using an older version than 2.16.
I would like to achieve something like below:
from("direct:dataload")
.beanRef("headerUpdater")
.log("Log: " + simple("${in.header.contentType}").getText())
//.unmarshal().beanio(simple("${in.header.contentType}").getText(), "content")
.unmarshal(new BeanIODataFormat(
"file://C://Users//admr229//Documents//mappings.xml", "clients"))
.to("bean:headerFooterValidator")
.split(body())
.process(dataValidator).choice()
.when(header("error").isNotNull())
.to("seda:saveErrorsForReport").otherwise()
.to("seda:updateLive")
.end();
I have commented out the line which I cannot make. I wanted to pass dynamic values from previous endpoint's output to initialize beanio.
Only thing I can think of is using recipient list which will dynamically choose a predefined endpoint. Because, for my case, that endpoint will have unmarshall with beanio, unlike something like "activemq:queue:test", which is purely text.
I hope I have made my question clear. Please let me know if you need any further details.
I am using camel 2.15.2
You can use the data format component [1] where you can specify beanio as the data format, and build the uri dynamic [2]
[1] - http://camel.apache.org/dataformat-component.html
[2] - http://camel.apache.org/how-to-use-a-dynamic-uri-in-to.html
you can use do something like this but, again this not dynamic I guess as it need to have properties set before brining up the context.
.unmarshal().beanio(mapping, streamName)