We're experiencing a strange problem.
We have a file component monitoring a folder. This works perfectly if the path is either
a) myrelativepath - which is relative to the Karaf installation where the camel route is run; or
b) /tst/mypath - which reads from a folder from the root
If I set log level to DEBUG I see the logs of it polling based on my interval.
However, if I set the path to be:
/mnt/windowsshare - which is a mounted windows share.
I get nothing in the logs, I don't see the poll, and it doesn't pick up any files. apparently the route is started though.
Interestingly, I have another camel route which writes a file to that location (a subfolder called inbound) and it writes file with no problem.
Any ideas?
I can get perhaps more logs tomorrow, but this is only happening in this environment where we have a windows share. And the share seems to be fine.
For testing we have run Camel as root and as root on the commandline we have tested reading the files (via vi) and all is ok.
Any suggestions for things to look at?
Basically make sure you don't have too many files in the antexlcude...polling is logarithmic and a fraction more makes polling very slow.
Needs more code analysis and VM introspection to understand why.
Related
I have been playing around with Apache module development and got the module working. However, I ran into the issue of where to hook properly to get all the data I need.
I am making a simple caching module that needs to hook at the beginning of the request and check if the file for this URL exists on disk and if it does then serve that file and stop content generation of Apache.
Currently, the module still continues to go into content generation mode. Let's say I have a long-running PHP script that takes 5s to generate. I would to omit calling the script altogether and just serve the static file from disk.
Furthermore, if the local file does not exist, I would like Apache to execute content generation (actually executes the PHP script) and before sending that data to the client I would like to have a proper hook that somehow gets this data and saves it to a local file.
I have tried ap_hook_fixups, ap_hook_handler and APR_HOOK_LAST, APR_HOOK_LAST and all the variations but no luck.
It always executes at the start of the request.
I also do not want to use any existing Apache modules. I want this to be a self-contained module.
Is there a way to do this kind of thing?
Based on the information you have provided, it sounds like you want your module to execute First and not last.
From what I understand of your issue, you want to make sure the file that would potentially be generated is already on disk or not, and if it is, serve that file, instead of allowing your php script to serve it.
In this case, you'll want to use APR_HOOK_FIRST or APR_HOOK_REALLY_FIRST
Then, assuming your file is on disk, you serve your file, and at the end of your module's work do a return OK;
If the file does not exist, do a return DECLINED
The DECLINED tells Apache that your module should not be the handler for this request and will continue down the list of modules till it finds something that will.
The goal here is to get your module to run before the php module does, to prevent your generation code from running, and fall back onto the php module if the requested file was not found.
Note:
The APR_HOOK_? priorities are just numbers -10 to 30
You should be able to fine tune this if you find your module executing a little too soon, like before mod_ssl for example.
Also, I am terrible at following documents, but the official Apache module development docs are amazingly assembled. Please try to use them, if you have,
I have spent the last 6 months messing around with Apache module development, working on a telegram bot.
I have had to do this song and dace a few times now.
Camel sftp is unable to poll more than 2 levels deep when the java code runs on Linux, but it works fine on Windows.
For example, polling files from
sftp://user#domain:22/folder1/folder2?...
works on both Unix and Windows. But, when I use something like
sftp://user#domain:22/folder1/folder2/folder3?...,
the route is always started yet the route running on Unix doesn't get the files in folder 3.
Route: route22 started and consuming from:sftp://user#domain:22/folder1/folder2/folder3?...
The sftp is to the same Unix machine and the same paths are used.
I have tried with stepwise true and false, as well as with recursive.
Could anyone shed some light on this please?
The problem was caused by a quartz trigger (attached to the route) that became corrupted. That happend because of a camel bug that makes camel unable to reconcile triggers when running in cluster mode if they fail due to database reasons.
Fuse/Camel newbie here. I'm trying to automate a manual process where .done files are downloaded from an FTP host, then renamed "fileout.txt", and finally an AS/400 program is executed on that file.
However, the department hosting the AS/400 program doesn't have resources to update their programming. The solution I'm working toward is to have Camel download one file at a time, save it as "fileout.txt", then execute a JT400 program to process it. Individually those steps work but I'm left with one problem.
What I pray you, dear reader, can help me with is "How can I stop Camel after downloading just one file?
(since overwriting, appending, or downloading multiple files won't work for the following step)".
How can I stop Camel after downloading just one file?
You can set following parameters in FTP consumer
maxMessagesPerPoll=1 (Limit number of message to be download in single batch)
delay=500000 (Increase the time interval between each poll, so you have time to stop the route)
Then, your ftp route can trigger an asynchronous message (maybe wireTap component) to another route to trigger controlBus component to stop the ftp route by route id.
I'm trying to automate a manual process where .done files are downloaded from an FTP host, then renamed fileout.txt, and finally an AS/400 program is executed on that file
Other than stop/start your route, you may try pollEnrich component with FTP usage. Using pollEnrich, you can trigger FTP consumer once when needed if you know the target file name already.
I was playing with a modified version of https://github.com/libfuse/libfuse/blob/master/example/passthrough.c and I mounted it in a folder in my home directory (home/joao/mnt). Without realizing that I was running it already, I re-ran the fuse programm and my computer stopped working. I rebooted the system and then I saw my home folder without most of the files that ithad before. Is there anything I can do to restore my old state?
I'm no way an expert, but I'm afraid you've lost everything unless you have a backup of your system... I can only give a hint. In passthrough.c it is said that
This file system mirrors the existing file system hierarchy of the
system, starting at the root file system. This is implemented by just "passing through" all requests to the corresponding user-space libc functions. Its performance is terrible.
So, if I understood it right, every change you're applying to mounted passthrough FS affects your home dir. Or, since it mirrors your home dir, you could just make a pretty recursion by mounting it there and thus break something.
We have an integration system based on Camel v2.16.1 that runs on a Jboss v6 Linux platform. There are multiple interfaces running simultaneously each with a different polling rate.
We are intermittently experiencing 'Cannot rename file' issue with Camel failing to backup to the 'done' folder successfully processed and transmitted files from the FTP source. Restarting the camel application fixes the issue.
Basically, at regular intervals triggered by a quartz scheduler, the route:
picks up files from a source via FTP,
processes them, smooks + xsl transformations
delivers the generated flat file to an endpoint via FTP.
If multiple files are read from the source directory, then all the files are appended together in a temporary file before being processed.
The Camel FTP configuration uses the following URL:
ftp://xxxx/export?antInclude=dsciord_*.dat&inProgressRepository=#warehouseIntegrationIdempotentRepository&preMove=in_progress_bpo/$simple{date:now:yyyyMMddHHmm}/$simple{file:name}&move=done&consumer.bridgeErrorHandler=true
read files dsciord_*.dat from /export directory
use custom inprogressRepository to store the read filename into a local db (this was done to prevent contention issue with a second cluster node, however, currently only a single node is live. This option is unnecessary and can be removed speeding up the process).
move files to an in_progress_bpo/201609061522 directory, where the subdirectory is created based on the date_timestamp.
move them to the in_progress_bpo/201609061522/done subdirectory once successfully processed.
In vast majority of cases the route works with no issues, however, sometimes the file(s) cannot be moved to the done folder (see error below). Even in this case, the route can sometimes continue successfully at the next polling cycle, however, in other cases the route enters a state when even if the quartz scheduler triggers the poll, the route fails to detect any files in the source /export directory even when there ARE files there.
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file: RemoteFile[in_progress_bpo/201609060502/dsciord_3605752.dat] to: RemoteFile[in_progress_bpo/201609060502/done/dsciord_3605752.dat]
Notes: We are using
a single instance of a ConsumerTemplate to handle our interfaces.
a custom inprogressRepository to store the file names read.
Obviously, there must be a system locking the source files and this is causing the Camel route to stop processing further files.
Any ideas/suggestions on debugging/resolving this issue would be greatly appreciated. The issues that I read through the camel-users forum seem to deal with Windows-related deployments, sometimes Smooks failing to close the input stream. I've check and we don't use the
org.milyn.templating.xslt.XslTemplateProcessor#bypass method where Smooks fails to close the underlying input stream.
Finally I have been able to reproduce/identify the issue.
Given that we are using a relative path to move the processed files into once successfully ftp-ed to the destination servers:
../../../u/4gl_upload/warehouse_integration_2/trs-server/export/in_progress_bpo/201609081030/done
However, for some reason instead of traversing the via correct path to move the processed files the camel consumer creates a new subdirectory tree starting from the current working directory and this could be quite long as follows. Hence the problem. It doesn’t know where it is and it doesn’t reset itself.
/u/4gl_upload/warehouse_integration_2/trs-server/u/4gl_upload/warehouse_integration_2/trs-server/export/in_progress_bpo/201609081030
This was reproduced with the option stepwise=false, which means it traverses the subdirectories in a single step instead of step wise.
Still don’t know what best solution is.