I'm currently creating some route and for one of them I have a problem.
Usually I have a data file and then a done file which have the same name prefixed by "ACK" and this works perfectly with camel and the doneFileName option.
But for one of my route I have to work with a different situation, I still receive two files but they have the same typology, it's like: MyFILE-{{timestamp}}. The data file contains the data, and the done file contains just "done".
So I need something to check the content of the file, and if it's juste "done" then process the other file.
Is there a way to handle this with camel?
The most pragmatic solution I see is to write an "adapter script" (bash or whatever you have at your disposal) that peeks into every file with a timestamp in its name.
If the file content is "done":
Lookup the other "MyFILE-{{timestamp}}" (the data file) and rename it to "MyFILE"
Rename the done file to "MyFILE.done"
Camel can then import the data file using the standard done-file-option. Because both files are renamed to something without a timestamp, the peek-script ignores them after renaming.
Related
I have an URL like
url = "file:D:/inputFolder?move=D:/outputFolder". we are making this url dynamically.
I want to rename the file while moving, So I made it something like this
url = "file:D:/inputFolder?move=D:/outputFolder&fileName=abc.txt". But I think move and fileName do not work together, it is not renaming.
Is there any alternative to do it? Please remember I want with "move" only.
I cannot use .setHeader(..) also.
Thanks,
Hy,
as far as I understand you, your trying to move the file in one single uri.
That is not really how camel works.
The idea of camel is to have a "consumer" and a "producer", where the consumer loads data (e.g. your file) and the producer puts the data somewhere (e.g. save the file into a folder)
That being said, here is what worked for me with a java route:
from("file:/home/chris/temp/camel/in")
.to("file:/home/chris/temp/camel/out/?fileName=test.txt");
The from part configures the folder where camel looks for new files. A few notes on that:
The file component checks the folder each 0.5 sec for new files. This can be changed with the delay parameter
The option noop configures, if the file is being moved or copied. By default it is set to false, which means it is moved
In the to part you configure, where the file is supposed to be moved. Here you can use the fileName parameter to rename the file.
Be careful with this though, because setting an option in the uri directly does make it "static".
What I mean by that is, that the only way of changing the parameter is by completely reconfiguring the route or by restarting it, where neither is something you would want to do normally.
Note 1:
Moving all files that are put into one folder into the same file always overrides the previous file by default.
You could, for example, use the fileExists parameter to always just append the content of the file: fileExists=Append (See camel file docu for details)
Note 2:
There is an option in the file component to not "move" the file, but copy, rename and delete it, which sometimes is necessary, when you want to move it onto a different drive and a simple copy does not work.
Also see the docu for the camel file component for details on that.
Note 3:
You can have multiple to() statements in the same route to have the file moved to multiple locations. For example:
from("file:/home/chris/temp/camel/in")
.to("file:/home/chris/temp/camel/out/?fileName=test.txt")
.to("smtp:....");
Hope I could help you and answer you question.
Greets
Chris
Two possible ways to achieve your goal.
Use both "consumer" and "producer"
Using this way, you are free to control where and how your destination can be set and has great freedom to control filename with the use of a processor/bean.
from("file:D:/inputFolder")
.to("file:D:/outputFolder?fileName=abc.txt")
Use "consumer" only
Using this way, you are treating your work as source data control. This can be use when your file is going to move within same drive. The drawback is the filename rename pattern is limited (refer to camel file language)
from("file:D:/inputFolder?move=${file:parent}/../outputFolder/abc.txt")
source_dir Have files like: ABC_02022018_162301.CSV, ABC_02022018_231801.CSV, controlFile
<route id="Test">
<from uri="file:source_dir?include=ABC_.*\.CSV&doneFileName=controlFile&delete=true&readLock=changed&readLockTimeout=20000&readLockCheckInterval=5000&eadLockMinLength=0"/>
<log message="${file:name}"/>
**destination directory **
</route>
I am looking here is, route has to check controlFile for every main file. If any main file doesn't have control file, main file shouldn't move from source folder.
In my above code, camel only once checking for control file existence in source folder and moving all the files to destination folder. Can anyone please help on this?
According to the Camel docs (section 'Using 'done' Files'), in order to have one doneFile per main file you need to specify dynamic doneFile names:
it is more common to have one done file per target file. This means
there is a 1:1 correlation. To do this you must use dynamic
placeholders in the doneFileName option. Currently Camel supports the
following two dynamic tokens: file:name and file:name.noext which must
be enclosed in ${}
if you don't, then Camel will consume all files and then delete the doneFile unless noop=true
Short answer: if you define doneFileName dynamically like this "doneFileName=${file:name.noext}.trig" then each file you want to transfer must have a copy with .trig extension.
If you define doneFileName statically like "doneFileName=files.trig" then every file will be moved when files.trig appears.
So you have to think of a unique doneFileName for each file instead of "controlFile" and set it dynamically. Easiest way is to use the actual fileName and add some extension.
Examples:
Imagine this files: file1.txt, file2.txt
and this route from:
from("ftp://admin#localhost/from?password=admin"
+ "&fileName=${file:name}"
+ "&doneFileName=${file:name.noext}.done")
In this case file1.txt will be moved only if file1.done is also present in the same folder. file2.txt will be moved only if file2.done is also present in the same folder.
Now imagine this route:
from("ftp://admin#localhost/from?password=admin"
+ "&fileName=${file:name}"
+ "&doneFileName=controlFile");
In this case both files file1.txt and file2.txt will be moved when files.done file is present in the same folder.
When I use a file consumer
<from uri="file:in?move=$simple{file:name}-transfered&include=^demo_keys\.ks$&sortBy=file:name" />
the file(s) are renamed to xxx-transfered (as I expected and stated in the doc) after processing.
But when I use the same with pollEnrich (for just one file)
<pollEnrich>
<simple>file://in?fileName=demo_keys.ks2&move=${camelId}-uploaded&sendEmptyMessageWhenIdle=true&maxMessagesPerPoll=1&delay=8000</simple>
</pollEnrich>
the file is not renamed after processing, instead moved into a newly created sub-directory with the original name.
How can I rename the pollEnrich processed file, achieve the same behaviour as a normal file consume?
I've tested it with v2.17.2 and v2.18.0
Thanks!
I think this may be a bug in Camel; try using a file language expression (e.g. ${file:name}) instead of ${camelId} just to be sure but the official documentation is pretty much clear in this case - it should interpret the value as a file name instead of directory.
I guess you should report a bug in Camel's JIRA.
I have to read a file in my shell script. I was using PL/SQL's UTL_FILE to open the file.
But I have to do a new change which will append timestamp to the file.
e.g import.data file becomes import_20152005101200.data
Now timestamp is the time at which file arrive at the server.
Since the file name changed I can't use the old way of file accessing.
I came up with below solution:
UTL_FILE.FOPEN ('path','import_${file_date}.data','r');
To achieve this I have to get filename and trim it using SUBSTR to get timestamp and pass to file_date variable.
However I am not able to find how to access filename in a particular path. I can use basename. But My file name keeps changing because of timestamp.
Any help/ alternate ideas are welcome.
PL/SQL isn't a good tool to solve this problem; UTL_FILE doesn't have any tools to list all the files in a folder.
A better solution is to define a stored procedure which uses UTL_FILE and pass the file name to process as an argument to the procedure. That way, you use the shell (which has many powerful commands and tools to examine folders and files) or a script language like Python to determine which file to process.
I have a directory with files likes this:
inbox/
data.20130813T1921.json
data.20130818T0123.json
data.20130901T1342.json
I'm using Apache Camel 2.11 and on process start, I only want to process one file: the latest. The other files can actually be ignored. Alternatively, the older files can be deleted once a new file has been processed.
I'm configuring my component using the following, but it obviously doesn't do what I need:
file:inbox/?noop=true
noop does keep the last file, but also all other files. On startup, Camel processes all existing files, which is more than I need.
What is the best way to only process the latest file?
You can use the sorting and then sort by name, and possible need to reverse it so the latest is first / last. You can try it out to see which one you need. And then set maxMessagesPerPoll=1 to only pickup one file. And you need to set eagerMaxMessagesPerPoll=false to allow to sort before limiting the number of files.
You can find details at: http://camel.apache.org/file2. See the section Sorting using sortBy for the sorting.
An alternative would be to still using the sorting to ensure the latest file is last. Then you can use the aggregator EIP to aggregate all the files, and use org.apache.camel.processor.aggregate.UseLatestAggregationStrategy as the aggregation strategy to only keep the last (which would be the latest file). Then you can instruct the file endpoint to delete=true to delete the files when done. You would then also need to configure the aggregator to completionFromBatchConsumer=true.
The aggregator eip is documented here: http://camel.apache.org/aggregator2