I have a requirement to read a file continuously from an FTP location and write to some topic(for eg:Kafka) using Apache Camel. I am able to read the file in the startup and write to the topic. Is there a way in Camel to read the file continuously from a folder whenever a new file come in to that location?
Related
Fuse/Camel newbie here. I'm trying to automate a manual process where .done files are downloaded from an FTP host, then renamed "fileout.txt", and finally an AS/400 program is executed on that file.
However, the department hosting the AS/400 program doesn't have resources to update their programming. The solution I'm working toward is to have Camel download one file at a time, save it as "fileout.txt", then execute a JT400 program to process it. Individually those steps work but I'm left with one problem.
What I pray you, dear reader, can help me with is "How can I stop Camel after downloading just one file?
(since overwriting, appending, or downloading multiple files won't work for the following step)".
How can I stop Camel after downloading just one file?
You can set following parameters in FTP consumer
maxMessagesPerPoll=1 (Limit number of message to be download in single batch)
delay=500000 (Increase the time interval between each poll, so you have time to stop the route)
Then, your ftp route can trigger an asynchronous message (maybe wireTap component) to another route to trigger controlBus component to stop the ftp route by route id.
I'm trying to automate a manual process where .done files are downloaded from an FTP host, then renamed fileout.txt, and finally an AS/400 program is executed on that file
Other than stop/start your route, you may try pollEnrich component with FTP usage. Using pollEnrich, you can trigger FTP consumer once when needed if you know the target file name already.
This question is related to this problem: Programmatically read a Queue's parameters
Is there a way to read queue.xml file's content programmatically on App Enigne? As far as I know all operations related to filesystem are prohibited on GAE.
The prohibited functions are related to the writing process in the file system (because does not exists in the sandbox) but the reading functions are available w/o problems.
the new File(); object set the root in you war folder (or webapp if Maven project), so you can open any file under that folder.
You can try to create new File("WEB-INF/queue.xml") and then read it with the common ways to read an xml
I am using camel 2.10.7 with great success from servicemix to feed files from the local file system to my application.
The files shall remain on the file system, hence I use a configuration like this one.
from uri="file:../ange-data/vessels?noop=true&idempotentKey=${file:name}-${file:modified}"
This works great if I touch/update a file on the file system.
Only issue remains: how can I then in my Java code detect that a file has been removed from the file system by some other person or process?
Could not find any hint by studying the manual pages http://camel.apache.org/file-language.html or http://camel.apache.org/file2.html - but I believe it should be possible to get a message on file deletion?
You would need to use Java 7 nio2 which has a file watcher api where you can get notifications when files are added/removed etc.
Search the web / SO for details on this api, for example
http://docs.oracle.com/javase/tutorial/essential/io/notification.html
Mule flow must download big and huge XML files to process. When provider writes a file in input folder, in the same time File endpoint component can download incomplete file or provider cannot complete write process for some reasons. Does Mule API provide handling the situation?
Please advise.
The file connector doesn't download data, are you mixing up with the HTTP connector? Or do you mean the Mule file transport picks-up a file while it is still being copied by the file producer?
If the latter, the best option is to write to file into a temporary location then move it to the Mule pick-up folder because moving a file is an atomic operation.
Alternatively, you can use the fileAge attribute on the file connector to configure Mule to only pick-up files that are older than the specified age. This can work only if you have an idea of the maximum time it takes the file writer to write the file.
I am using Mule 3.2 and I am moving files from one location to another location. The error/problem is that Mule keeps on processing the same files again and again and do not deleted them.
The console displays:
org.mule.transport.file.FileMessageReceiver: Lock obtained on file:
My config file is below:
<flow name="File-FTP-Bridge">
<file:inbound-endpoint path="${outbound.input.path}"
moveToDirectory="${outbound.input.backup.path}">
<file:filename-wildcard-filter
pattern="*.msg" />
</file:inbound-endpoint>
<ftp:outbound-endpoint user="${outbound.ftp.user}"
password="${outbound.ftp.password}" host="${outbound.ftp.host}"
path="${outbound.ftp.path}" port="${outbound.ftp.port}"
outputPattern="#[header:originalFilename]">
</ftp:outbound-endpoint>
</flow>
I could not find the root cause for this problem. Thanks in advance.
Your file endpoint misses a pollingFrequency attributes, which means it uses the default of 1000ms. This makes Mule poll files way faster than the FTP endpoint can process them. Try for example:
pollingFrequency="10000"
If this is not good enough because the FTP upload has unpredictable performances (so Mule still retries a file that is being uploaded), then if your files are small enough to fit in memory, try adding:
<object-to-byte-array-transformer />
between your inbound and outbound endpoint. This loads the file in-memory and moves it right away to outbound.input.backup.path, before trying the FTP upload. Of course, if the FTP upload fails, you'll have to move the file back to outbound.input.path...