Iam trying to automate the testing of a java bundle,Which will process once a file is dropped in a particular folder.
can we drop a file to a folder location automatically using camel,or at a set period of time(not intervals)?
is this possible purely by camel or should we incorporate other frameworks?
sure, you can use the camel-file component to produce (create files somewhere) and consume (read/process files from somewhere) and optionally control the initial/polling delays easily with attributes...
here is a simple example of consuming->processing->producing
from("file://inputdir").process(<dosomething>).to("file://outputdir")
alternatively, you could periodically produce a file and drop it somewhere
from("timer://foo?fixedRate=true&period=60000").process(<createFileContent>").to("file://inputdir");
Although camel could do this by creating a timer endpoint, then setting the file content and writing to a file endpoint, my answer would be to simply use a bash script. No camel needed here.
Pseudo bash script:
while [ true ]
do
cp filefrom fileto
pauze 10s
done
Related
I need to invoke a Data Factory V2 pipeline when a file is placed in a blob container.
I have tried using Powershell to check if the file is present, the issue I have there is that if the file is not there, and tell me its not there, I then place the file in the container and Powershell will still tell me its not there, though perhaps if it reruns the variable will get a fresh value and tell ites there? Maybe there is a way around that? If yes, I can then use the result to invoke the pipeline with the Powershell script. Am I along the right lines here?
Other option will be to write a t-sql query that will give a true/false result if the row condition is met, but I am not sure how I can use this result within/against DFv2. In the IF condition module?
Tried a Logic app but it was kind of useless. It would be great if I could get some suggestions in some ways to trigger the pipeline on the arrival of the file in the blob container, there is more than one way to skin a cat, so open to any and all ideas. Thank you.
This is now available as an event trigger with ADF V2 as announced in this bog post on June 21, 2018.
Current documentation on how to set it up is available here: Create a trigger that runs a pipeline in response to an event.
From the documentation:
As soon as the file arrives in your storage location and the corresponding blob is created, this event triggers and runs your Data Factory pipeline. You can create a trigger that responds to a blob creation event, a blob deletion event, or both events, in your Data Factory pipelines.
There is a note to be wary of:
This integration supports only version 2 Storage accounts (General purpose).
Event triggers can be one, or both of:
Microsoft.Storage.BlobCreated
Microsoft.Storage.BlobDeleted
With firing conditions from the following:
blobPathBeginsWith
blobPathEndsWith
The documentation also provides the following examples of event trigger firing conditions over blobs:
Blob path begins with('/containername/') – Receives events for any blob in the container.
Blob path begins with('/containername/foldername') – Receives events for any blobs in the containername container and foldername folder.
Blob path begins with('/containername/foldername/file.txt') – Receives events for a blob named file.txt in the foldername folder under the containername container.
Blob path ends with('file.txt') – Receives events for a blob named file.txt at any path.
Blob path ends with('/containername/file.txt') – Receives events for a blob named file.txt under container containername.
Blob path ends with('foldername/file.txt') – Receives events for a blob named file.txt in foldername folder under any container.
I'm using Solr 6.5 to index files from multiples ftp files into multiples cores (having one core for each type of document, like audio file, image, software, video and documents).
The situation is that I'm doing this to populate an app that in its front end has a social networking approach in which every user can add new tags or modify other metadata without restriction.
So when I execute again data import handler to add new files to my application, it erase the index that previosly was modified for the user and set up with the data-config default configuration.
My question: is there a way to tell DIH, if the id exists, continues without importing and just adds the files which don't have an id in the index?
If this is not possible, can I do something similar in a different way?
Thanks for everything!
Sounds like you are doing a full import with default settings. One of them is clean, which defaults to true and deletes the whole index before the import.
Try setting it to false and also look at preImportDeleteQuery and postImportDeleteQuery for even more precision.
How to create log file with custom file name in cakephp 2.5.1.
I am written following code
$this->log("Data : ".print_r($this->data,true),'debug');
It create's debug.log file in tmp/log folder but when I write custom name like
$this->log("Data : ".print_r($this->data,true),'data');
This code do not create any log file. My Question is how to create log file with custom name?
Same answer as for the other question: Start reading the manual. A huge part of the profession is to read documentation.
Changed in version 2.5.
CakeLog does not auto-configure itself anymore. As a result log files
will not be auto-created anymore if no stream is listening. Make sure
you got at least one default stream set up, if you want to listen to
all types and levels. Usually, you can just set the core FileLog class
to output into app/tmp/logs/:
I have a scenario where multiple files will be dropped at a location. I am interested in reading two files out of it as a pair (e.g., 913.xml and 913.pdf). I want to pass both files to a Java component which will merge them.
I know how to merge them but I am not sure how to pass two files at once to my Java component
I can think of two options:
Configure the file inbound endpoint to select say the XML files and in your component use the MuleClient to request the other file,
Configure the file inbound endpoint to select both XML and PDF files and add a collection-aggregator configured with an expression-message-info-mapping that uses the originalFilename property in a correlation ID expression that extracts the file name sans extension from it. Don't forget to set the MULE_CORRELATION_GROUP_SIZE property to 2 before the aggregator.
1)be.file backend has a feature - messages added before _be_file_set_filename() are not lost and will be appended to log later.
2)_be_file_set_filename opens file immediately => if no entries was added, emty file will be created
Question: how to suppress creating empty file (create file only when first log entry appended) and save feature 1) ?
Use the flag PANTHEIOS_BE_FILE_F_DELETE_IF_EMPTY, described here.
My guess is that the file is created synchronously to avoid the less-desirable situation where a log file cannot be created while a process is running. If you really want that, maybe add the functionality and supply back to the project?
I've solved this problem by writing custom backend - wrapper over be.file