Camel 2.13.0
I am attempting to consume a json string containing multiple records and produce an output file for each record.
public void configure() {
from("file:data/input")
// my bean splits the input string and returns a json string via Gson
.bean(new com.camel.Tokenizer(), "getSentences")
.split(new JsonPathExpression("$[*]"))
.convertBodyTo(String.class)
.to("file:data/output");
}
}
Sample input:
"This is string #1. This is string #2."
If I run the code above as-is I get a single output file containing "This is string #2.". If I remove the .split() call I get a single output file containing json as expected:
[
{
"token": "This is string #1."
},
{
"token": "This is string #2."
}
]
How can I achieve two output files representing both lines of data?
It occurred to me that perhaps the split was working correctly and the second output file was overwriting the first. According to the documentation, the default behavior if CamelFileName is not set is to create a unique generated ID but I do not experience this. In my case the output file name always matches the input file name.
How can I get unique file name within each folder?
Thanks!
Finally stumbled upon the proper search terms and came across the following helpful post: Camel: Splitting a collection and writing to files
The trick is to use a bit of logic in the .to() to achieve unique output file names:
.to("file:data/sentence_q?fileName=${header.CamelSplitIndex}.txt");
The JsonPathExpression works like a charm and no need for a Processor() or unmarshal(), as I'd tried previously.
Related
I am trying to write a GUI that will display the name of the sketch it was generated from using a simple text() command. However, I am running into trouble getting any of the general JS solutions to work for me. Many solutions I have found use the filename reserved word but that does not seem to be reserved in Processing 3.5.4. I have also tried parsing the strings using a similar method to what can be found here. I am very new to processing and this is only my 2nd attempt at using Processing.
Any advice would be greatly appreciated
You can get the path (as a string) to the sketch with sketchPath().
From there you could either parse the string (pull off everything after the last slash) to get the sketch name, or you can use sketchFile() to get a reference to the file itself and get the name from there:
String path = sketchPath();
File file = sketchFile(path);
String sketchName = file.getName();
println(sketchName);
You could combine this all into one line like so:
String sketchName = sketchFile(sketchPath()).getName();
I'm new to perfecto mobile device automation. I came across a name field which returns value in two lines in the application. Like below
School Name : ABCD
INTERNATIONAL
I found the locator which is Xpath as //*[text()="ABCD INTERNATIONAL"], this returns value in two lines which I confirmed when I ran the code and asserted the value. So basically the error is like below
expected [ABCD INTERNATIONAL] but found [ABCD
INTERNATIONAL]
My feature file is like
Feature: Verify the school Details
Scenario : Verify the school Name
Given User logs into Application
When User verifies the school name as "ABCD INTERNATIONAL"
Then user logs of
Step Defenition:
#When("User verifies the school name as {string}")
public void a_User_verifies_the_school_name_as (String Name) throws Exception {
String School_Name = utility.getText(appiumDriver, "//*[text()="ABCD INTERNATIONAL"]);
Assert.assertEquals(School_Name, Name);
}
Your help in sorting out this will mean a lot.
I think you have to modify your actual and expected string by using replaceAll() method.
Like replaceAll("\n",""); and replaceAll(" ",""). then compare both in assertion.
If it is not working then you have to use pattern matching by regular expression.
As the title suggests, I want to move a file into a different folder after I am done writing DB records to to it.
I have already looked into several questions related to this: Apache camel file with doneFileName
But my problem is a little different since I am using split, stream and parallelProcessing for getting the DB records and writing to a file. I am not able to know when and how to create the done file along with the parallelProcessing. Here is the code snippet:
My route to fetch records and write it to a file:
from(<ROUTE_FETCH_RECORDS_AND_WRITE>)
.setHeader(Exchange.FILE_PATH, constant("<path to temp folder>"))
.setHeader(Exchange.FILE_NAME, constant("<filename>.txt"))
.setBody(constant("<sql to fetch records>&outputType=StreamList))
.to("jdbc:<endpoint>)
.split(body(), <aggregation>).streaming().parallelProcessing()
.<some processors>
.aggregate(header(Exchange.FILE_NAME), (o, n) -> {
<file aggregation>
return o;
}).completionInterval(<some time interval>)
.toD("file://<to the temp file>")
.end()
.end()
.to("file:"+<path to temp folder>+"?doneFileName=${file:header."+Exchange.FILE_NAME+"}.done"); //this line is just for trying out done filename
In my aggregation strategy for the splitter I have code that basically counts records processed and prepares the response that would be sent back to the caller.
And in my other aggregate outside I have code for aggregating the db rows and post that writing into the file.
And here is the file listener for moving the file:
from("file://<path to temp folder>?delete=true&include=<filename>.*.TXT&doneFileName=done")
.to(file://<final filename with path>?fileExist=Append);
Doing something like this is giving me this error:
Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot store file: <folder-path>/filename.TXT] org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: <folder-path>/filename.TXT
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:292)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)[209:org.apache.camel.camel-core:2.16.2]
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)[209:org.apache.camel.camel-core:2.16.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_144]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.io.InputStream but has value: Total number of records discovered: 5
What am I doing wrong? Any inputs will help.
PS: Newly introduced to Apache Camel
I would guess that the error comes from .toD("file://<to the temp file>") trying to write a file, but finds the wrong type of body (String Total number of records discovered: 5 instead of InputStream.
I don't understand why you have one file-destinations inside the splitter and one outside of it.
As #claus-ibsen suggested try to remove this extra .aggregate(...) in your route. To split and re-aggregate it is sufficient to reference the aggregation strategy in the splitter. Claus also pointed to an example in the Camel docs
from(<ROUTE_FETCH_RECORDS_AND_WRITE>)
.setHeader(Exchange.FILE_PATH, constant("<path to temp folder>"))
.setHeader(Exchange.FILE_NAME, constant("<filename>.txt"))
.setBody(constant("<sql to fetch records>&outputType=StreamList))
.to("jdbc:<endpoint>)
.split(body(), <aggregationStrategy>)
.streaming().parallelProcessing()
// the processors below get individual parts
.<some processors>
.end()
// The end statement above ends split-and-aggregate. From here
// you get the re-aggregated result of the splitter.
// So you can simply write it to a file and also write the done-file
.to(...);
However, if you need to control the aggregation sizes, you have to combine splitter and aggregator. That would look somehow like this
from(<ROUTE_FETCH_RECORDS_AND_WRITE>)
.setHeader(Exchange.FILE_PATH, constant("<path to temp folder>"))
.setHeader(Exchange.FILE_NAME, constant("<filename>.txt"))
.setBody(constant("<sql to fetch records>&outputType=StreamList))
.to("jdbc:<endpoint>)
// No aggregationStrategy here so it is a standard splitter
.split(body())
.streaming().parallelProcessing()
// the processors below get individual parts
.<some processors>
.end()
// The end statement above ends split. From here
// you still got individual records from the splitter.
.to(seda:aggregate);
// new route to do the controlled aggregation
from("seda:aggregate")
// constant(true) is the correlation predicate => collect all messages in 1 aggregation
.aggregate(constant(true), new YourAggregationStrategy())
.completionSize(500)
// not sure if this 'end' is needed
.end()
// write files with 500 aggregated records here
.to("...");
I am passing a JSON array object in the HTTP POST as
[{"level":"INFO","data": "Test 1"},{"level":"INFO","data": "Test 2"}]
This message is seen as 1 object/log message in SumoLogic. How can I tell SumoLogic to consider each JSON object as an independent object and show 2 log messages instead of one?
I believe this can't be done with the json operator. But, have a look at the docs for the "parse regex" operator. There's an option called "multi" which creates a new message for each match of the regex. In your case, something like this might do the trick:
parse regex "\{?<fieldname>.*?\}" multi
I didn't try this in the product itself, but here is Regex101 link to play with the regex.
I believe the actual answer to this is to not send your logs as an array. Instead include each json object in your body with a '\n' at the end for Sumo to consider these as individual log messages.
{"level":"INFO","data": "Test 1"}\n
{"level":"INFO","data": "Test 2"}\n
I am creating an elixir project to search for patterns in files.
I want to store those patterns a config files to allow for easy changes in the app.
My first idea is storing those files as exs files in the config folder in the mix project.
So, the questions are:
Is there any easy way to store the config in the files a a keyword list?
How would I load it in the app?
I see there are modules like File to read the file, but is there no standard way to parse keyword lists in elixir? I was thinking something similar as the yml files in Rails.
You can read keyword lists stored in a *.exs file, using Mix.Config.read(path). For writing Elixir terms to a *.exs file, you can use Inspect.Algebra.to_doc(%Inspect.Opts{pretty: true}) and write the resulting string content to a file using File.write. It's not as well formatted as if you did it by hand, but it's definitely still readable.
If you don't mind using Erlang terms, you can read and write those easily using :file.consult(path) and :file.write_file(:io_lib.fwrite('~p.\n', [config]), path) respectively.
Using Code.eval_file
Adding another option, is to evaluate the file as a code file, using Code.eval_file and get in return the result as an elixir construct.
Config file config1.ex:
%{configKey1: "configValue1", configKey2: "configValue2"}
Reading the file:
{content, _} = Code.eval_file("config1.ex")
*evaluating a code file has security consideration needs to take in mind.
Regarding using Mix.Config.read! in #bitwalker correct answer
the config file needs to be in a specific format of:
[
appName: [key1: "val1", key2: "val2"]
]
In the Mix.Config.read code, it try to validate the contents and expect a main keyword list [ {}, {}.. ] which includes keys that has value of type keyword list also.
The code is not long:
def validate!(config) do
if is_list(config) do
Enum.all?(config, fn
{app, value} when is_atom(app) ->
if Keyword.keyword?(value) do
true
else
raise ArgumentError,
"expected config for app #{inspect app} to return keyword list, got: #{inspect value}"
end
_ ->
false
end)
else
raise ArgumentError,
"expected config file to return keyword list, got: #{inspect config}"
end
end
We can circumvent and use a first key which is not atom, and then the validate stops but does not throw:
[
{"mockFirstKey", "mockValue"},
myKey1: "myValue1",
myKey2: "myValue2"
]