Is it possible to use apache camel is a cluster when using the file compoment?
For example the following rout:
from("file://C:/temp/camel/in?moveFailed=/temp/camel/error&delay=10000&delete=true")
.to("file://C:/temp/camel/out")
Will camel know it file has already been taken by one member of the cluster?
What if both camel instances take the file at the same time?
Thanks
Yes read the docs about the read lock option, where you have different options to setup this.
Related
I am using apache camel as integration framework in my microservice. I am deploying it in a Kubernetes cloud as multiple pods. I had written a route for reading file from a directory and write to another. But I am facing an issue as the different pods are picking same file. I need to avoid that. I only want any of the pod to pick the file and process but currently all the pods are picking and processing the file. Can someone help with this. Please suggest some examples available in GitHub or any other.
Thanks in advance.
Camel recently introduced some interesting clustering capabilities - see here.
In your particular case, you could model a route which is taking the leadership when starting the directory polling, preventing thereby other nodes from picking the (same or other) files.
Set it up is very easy and all you need is to prefix singleton
endpoints according to the master component syntax:
master:namespace:delegateUri
This would result in something like this:
from("master:mycluster:file://...")
.routeId("clustered-route")
.log("Clustered file polling !");
I have zero experience with ETL.
Whenever a file(a .csv) is moved into a specific folder, it should be uploaded to SalesForce I don't know how to get this automated flow.
I hope I was clear enough.
I gotta use the opensource version, any helpful links or resources will be appreciated.
Thank you in advance
You could definitely use Talend Open Studio for ESB : this studio contains 'Routes' functionalities : you'll be able to use a cFile component, which will check your folder for new files, and raise an event that will propagate throughout the route to a designed endpoint (for example a salesForce API). Talend ESB maps Apache Camel components , which are well documented.
Check about Routes with Talend for ESB, it should do the trick.
We have tFileExists component, you can use that and configure to check the file.
Also you have tFileWait component, where you can defile the frame of the arrival of he files and the number of iterations it has to check the file.
But i would suggest id you have any scheduling tool, use file watcher concept and then use talend job to upload the file to a specific location.
Using talend itself to check the file arrival is not a feasible way as the jobs has be in running state continuously which consumes more java resource
I Am having two servers which poll to same ftp location. when a file is placed in that location both the servers are picking the files. But I need only one server to pick the file and process and delete.I am using Camel 2.16.2 Version. Is there any way to solve this issue?
Camel FTP uses most of camel-File2 API internally. All, camel file the options are inherited.
There are many strategies to avoid parallel processing of same file.
Use preMove, moveFailed, readLock, readLockRemoveOnCommit Camel-File options.
For instance you could use below parameters.
ftp://{{input.folder}}?readLock=rename&preMove=.inprogress&moveFailed=.error
We have a clustered environment where same camel ftp endpoint is installed on multiple fuse instances. I want message(file) to be consumed by only one fuse instance. I am planning to achieve this by implementing IdempotentRepository interface with database persistence. Want to make sure that this approach will work or there is a better way to do it?
If you don't want to depend on a database for doing this you could try with Hazelcast. Hazelcast is a distributed data cache that can be used as a idempotent repository without external dependencies. Also, Apache Camel provides a class for doing this. There is a nice tutorial explaining how to do it: Hazelcast Idempotent Repository Tutorial
My purpose is to not to read same files again and again from a shared directory by different applications.
Is there a sharable idemponent repository for multiple applications which use camel file component?
THNX
Yes read about the idempotentRepository on the Camel file doc page: http://camel.apache.org/file2
You can use a shared repository such as a JDBC based, so you can use a database table,
or write your own repository implementation.
You can also read more here: http://camel.apache.org/idempotent-consumer.html