I am currently working on a Camel based test harness application, that processes groups of files from multiple folders and compares with the files present in the local repository.
Is there any way to change the folder location from the end point in Camel route dynamically? I want to use a single route for polling files from the multiple folders.
According dynamic change endpoint camel, use following procedure:
stop the route
remove the route
change the endpoint
add the route
start the route
Related
Thanks to JMX (java console), I try to restart a route with a file component consumer endpoint.
from("file:<some dir>?noop=true")
I am using the wiretap pattern to record the intermediate data transformation through other files endpoint.
On first start of the camel application, everything is fine, and all the files already present in the input directory are polled and processed.
But when I try to restart the route thanks to jmx, nothing happens.
I try to manually removed .camel directory - created by I guess the default FileIdempotentRepository - before restarting the route, in vain.
I also tried to change the kind of IdempotentRepository with a MemoryIdempotentRepository :
from("file:<somedir>?noop=true").idempotentConsumer(header("CamelFileName"), MemoryIdempotentRepository.memoryIdempotentRepository(1000))
Even if I trigger the clear() operation of this MemoryIdempotentRepository before restarting the route in java console, nothing is polled from the input directory after restarting.
If I add a file, it works. Everything behaves like if there is a persistent history of the files already polled once.
I wonder if the use of the option "noop=true" creates an unmanaged idempotent repository I cannot control with jmx.
If true, the file is not moved or deleted in any way. This option is
good for readonly data, or for ETL type requirements. If noop=true,
Camel will set idempotent=true as well, to avoid consuming the same
files over and over again.
Any idea ?
(i am using camel-core 2.21)
I found the solution to my issue.
I made a bad use of idempotentConsumer; I needed to initialize the endpoint idempotent consumer inside the endpoint URI parameters list.
First, create an entry in a bean registry:
registry.put("myIdempotentRepository", MemoryIdempotentRepository.memoryIdempotentRepository(1000));
Then, refer to this idempotentRepository in the endpoint:
from("file:<somedire>noop=true&initialDelay=10&delay=1000&idempotentRepository=#myIdempotentRepository")
By doing this, GenericFileEndPoint:
will not create a default idempotentRepository
will add the idempotentRepository given in options of the endpoint to the services of the camel context. This means that it will be possible to manage it thanks to JMX
I think it would be useful to be allowed to manage the default idempotent repository in the FileEndPoint class of camel-core.
I have written an application in CakePHP 3.x and there is a form to upload files.
At present, files are being uploaded to WWW_ROOT.'files' which is /app/webroot/files/ path.
To store files separately from core application, I created a subdomain cdn.example.com whose path is like /home/user/example.com/cdn.example.com/.
since, uploading files requires absolute path, how can I get absolute path of the subdomain cdn.example.com same as $_SERVER['DOCUMENT_ROOT']; from example.com?
First check if your server allow to write files from main domain to subdomain.
You can't get the absolute path of a different domain of the one you deployed your application, you should store the path in a variable or in constant.
If you really need to store files in another domain/subdomain and any path is blocked you should consider to reupload the files via FTP script.
I have two separate routes running in an application and I want to control the total amount of work in flight across the entire path.
Route 1: Gzipped file on SFTP --> unzip --> local directory
Route 2: local directory --> process stuff --> Kafka
If route 2 has problems or falls behind in its work, I don't want route 1 to fill up the local directory. How can I limit the total number of files sitting in local dir waiting to be processed?
(if it was a single route I may be able to throttle() easier, but are there other options to look at the overall picture of multiple routes?)
You can implement a custom RoutePolicy where you check the number of files in that directory and if its bigger than X then suspend the route, and resume it again if its lower than X.
See more details at the Camel docs: http://camel.apache.org/routepolicy.html
You can look at the existing ThrottlingInflightRoutePolicy how its implemented for inspiration.
I have a camel route consuming from an FTP server and storing any files it consumes to a directory with move=.dealtWith. However, the number of files in this .dealtWith directory can quickly become unmanageable for users to view, so I would like to move the file to a .dealtWith/{the_date} directory. Is there a way to specify this functionality in camel without bringing the route down?
Use Camel Simple Expression Language
ftp:url?move=.dealtWith/${date:now:yyyy-MM-dd}/${header.CamelFileName}
I've defined the following camel route:
RouteBuilder rb = new RouteBuilder() {
#Override
public void configure() throws Exception {
from("sftp://myhost//path/to/files/")
.to("log:loggingCategory?level=INFO")
.to("file:///tmp/");
}
};
When I start the context with this route camel does connect and it downloads the files. My problem is that camel repeats downloading the same files until the context is shut down. Why does the FTP2 component do this and how can I stop it?
I've included version 2.10.4 of camel-core and camel-ftp via maven.
The Indempotent Consumer does the trick. Docs of the FTP2 component refer to the File2 component as "as all the options there also applies for this component". There is a parameter "indempotent=true" that activates usage of an LRUCache:
Option to use the Idempotent Consumer EIP pattern to let Camel skip
already processed files. Will by default use a memory based LRUCache
that holds 1000 entries. If noop=true then idempotent will be enabled
as well to avoid consuming the same files over and over again.
My complete source definition now looks like this:
from("sftp://myhost//path/to/files/?username=user&password=secret&idempotent=true")
From the camel ftp2 documentation:
The FTP consumer will by default leave the consumed files untouched on
the remote FTP server. You have to configure it explicitly if you want
it to delete the files or move them to another location. For example
you can use delete=true to delete the files, or use move=.done to move
the files into a hidden done sub directory.
To delete the file, change the route to
from("sftp://myhost//path/to/files?delete=true")
Ensure that the connected user has required permissions.