I Am having two servers which poll to same ftp location. when a file is placed in that location both the servers are picking the files. But I need only one server to pick the file and process and delete.I am using Camel 2.16.2 Version. Is there any way to solve this issue?
Camel FTP uses most of camel-File2 API internally. All, camel file the options are inherited.
There are many strategies to avoid parallel processing of same file.
Use preMove, moveFailed, readLock, readLockRemoveOnCommit Camel-File options.
For instance you could use below parameters.
ftp://{{input.folder}}?readLock=rename&preMove=.inprogress&moveFailed=.error
Related
We are using apache camel sftp 2.25.4 as a poller (Jsch) to read xml files. There is two spring boot 2.6.10 applications (same application for redundancy) reading from the same sftp folder 'inbound/orders' with the configuration:
sftp://user#localhost:2222/inbound/orders?preMove=$simple{file:parent}/.processing_$simple{sys.hostname}/$simple{file:onlyname}
When either of the application is shutdown for maintenance(graceful shutdown) the preMove file becomes orphaned. Is there a way to ensure camel fully consumes this 'preMove' file before shutting down the route?
I expect some may suggest an idempotent component to handle this which is something that we are considering but (dare I say) trying to avoid the overhead of a cache lookup during the read operation (we consider this a tier 0 service so need to avoid any dependencies).
I have tried other styles of control such as the markerFile and rename strategy but none seem to work as well as the preMove. The preMove works really well.
I have zero experience with ETL.
Whenever a file(a .csv) is moved into a specific folder, it should be uploaded to SalesForce I don't know how to get this automated flow.
I hope I was clear enough.
I gotta use the opensource version, any helpful links or resources will be appreciated.
Thank you in advance
You could definitely use Talend Open Studio for ESB : this studio contains 'Routes' functionalities : you'll be able to use a cFile component, which will check your folder for new files, and raise an event that will propagate throughout the route to a designed endpoint (for example a salesForce API). Talend ESB maps Apache Camel components , which are well documented.
Check about Routes with Talend for ESB, it should do the trick.
We have tFileExists component, you can use that and configure to check the file.
Also you have tFileWait component, where you can defile the frame of the arrival of he files and the number of iterations it has to check the file.
But i would suggest id you have any scheduling tool, use file watcher concept and then use talend job to upload the file to a specific location.
Using talend itself to check the file arrival is not a feasible way as the jobs has be in running state continuously which consumes more java resource
I have gone through google but I didn't find anything like how to execute sftp commands using Apache camel could any one refer the example or some link relevant
to the example. any help is greatly appreciated.
The camel-ftp component (and all the other Camel components) are intended for exchanging data (messages) between systems. They are not really intended for a Java client library with a free API you can do all sorts of thing with.
So if you want to do some special things on the FTP server then maybe use the Java FTP client API directly, and not the camel-ftp component.
That said the camel-ftp component has a download=false option you may be able to use to consume the FTP files metadata (name, size etc). But again the consumer is intended for keep monitoring the FTP server and download new files. So its not a perfect fit for your use-case.
I need to process 1000 files after downloading from ftp , the downloading part is done using apache camel, can I also break the processing of files into sub tasks using camel. like multiple processes that camel is handling for me
you can always use the threads() API to enable concurrent processing on a route...
from("file://downloaded").threads(10).to(...);
Is it possible to use apache camel is a cluster when using the file compoment?
For example the following rout:
from("file://C:/temp/camel/in?moveFailed=/temp/camel/error&delay=10000&delete=true")
.to("file://C:/temp/camel/out")
Will camel know it file has already been taken by one member of the cluster?
What if both camel instances take the file at the same time?
Thanks
Yes read the docs about the read lock option, where you have different options to setup this.