We are using apache camel sftp 2.25.4 as a poller (Jsch) to read xml files. There is two spring boot 2.6.10 applications (same application for redundancy) reading from the same sftp folder 'inbound/orders' with the configuration:
sftp://user#localhost:2222/inbound/orders?preMove=$simple{file:parent}/.processing_$simple{sys.hostname}/$simple{file:onlyname}
When either of the application is shutdown for maintenance(graceful shutdown) the preMove file becomes orphaned. Is there a way to ensure camel fully consumes this 'preMove' file before shutting down the route?
I expect some may suggest an idempotent component to handle this which is something that we are considering but (dare I say) trying to avoid the overhead of a cache lookup during the read operation (we consider this a tier 0 service so need to avoid any dependencies).
I have tried other styles of control such as the markerFile and rename strategy but none seem to work as well as the preMove. The preMove works really well.
Related
I am using apache camel as integration framework in my microservice. I am deploying it in a Kubernetes cloud as multiple pods. I had written a route for reading file from a directory and write to another. But I am facing an issue as the different pods are picking same file. I need to avoid that. I only want any of the pod to pick the file and process but currently all the pods are picking and processing the file. Can someone help with this. Please suggest some examples available in GitHub or any other.
Thanks in advance.
Camel recently introduced some interesting clustering capabilities - see here.
In your particular case, you could model a route which is taking the leadership when starting the directory polling, preventing thereby other nodes from picking the (same or other) files.
Set it up is very easy and all you need is to prefix singleton
endpoints according to the master component syntax:
master:namespace:delegateUri
This would result in something like this:
from("master:mycluster:file://...")
.routeId("clustered-route")
.log("Clustered file polling !");
I have an Apache Camel Route that uses LevelDB as the aggregation repository. My problem is that when the Camel Context is started the LevelDBAggregationRepository is automatically started by Camel even if the Camel Route it is used in is off and not started.
Is there a way of preventing this?
Why is this important for me? I want my application to be highly available, so I want to share the same LevelDB between nodes. But the LevelDBAggregationRepository unfortunately does not support using multiple processes at a time, and I have no SQL DB available for the JDBC Aggregation Repository.
So, my current solution attempt is to use a route policy that ensures that only one node at a time has the Camel Route enabled (determined by leader election with Zookeeper). However, when I start a second node with the route turned off, its Camel Context tries to launch the LevelDB anyway and then all hell breaks loose.
I Am having two servers which poll to same ftp location. when a file is placed in that location both the servers are picking the files. But I need only one server to pick the file and process and delete.I am using Camel 2.16.2 Version. Is there any way to solve this issue?
Camel FTP uses most of camel-File2 API internally. All, camel file the options are inherited.
There are many strategies to avoid parallel processing of same file.
Use preMove, moveFailed, readLock, readLockRemoveOnCommit Camel-File options.
For instance you could use below parameters.
ftp://{{input.folder}}?readLock=rename&preMove=.inprogress&moveFailed=.error
I need to process 1000 files after downloading from ftp , the downloading part is done using apache camel, can I also break the processing of files into sub tasks using camel. like multiple processes that camel is handling for me
you can always use the threads() API to enable concurrent processing on a route...
from("file://downloaded").threads(10).to(...);
I was wondering what optimisations I could make to my server to better it's performance at handling file uploads/downloads.
At the moment I am thinking Apache2 may not be the best HTTP server for this?
Any suggestions or optimisations I could make on my server?
My current set up is an Apache2 HTTP server with PHP dealing with the file uploads which are currently stored in a folder out of the web root and randomly assigned a name which is stored in a MySQL database (along with more file/user information).
When a user wants to download a file, I use the header() function to force the download and readfile() to output the file contents.
You are correct that this is inefficient, but it's not Apache's fault. Serving the files with PHP is going to be your bottleneck. You should look into X-Sendfile, which allows you to tell Apache (via a header inserted by PHP) what file to send (even if it's outside the DocRoot).
The increase in speed will be more pronounced with larger files and heavier loads. Of course an even better way to increase speed is by using a CDN, but that's overkill for most of us.
Using X-Sendfile with Apache/PHP
http://www.jasny.net/articles/how-i-php-x-sendfile/
As for increasing performance with uploads, I have no particular knowledge. In general however, I believe each file upload would "block" one of your Apache workers for a long time, meaning Apache has to spawn more worker processes for other requests. With enough workers spawned, a server can slow noticeably. You may look into Nginx, which is an event-based, rather than process-based, server. This may increase your throughput, but I admit I have never experimented with uploads under Nginx.
Note: Nginx uses the X-Accel-Redirect instead of X-Sendfile.
http://wiki.nginx.org/XSendfile