I have a requirement to delete files which do not get a .done file within 24 hours of the time that they drop. Is there a way to do this with Camel, Spring Xml, so that I can keep all of this processing in one bundle? So, I would need to find a file with no matching .done file, and where CamelFileLastModified is 24 hours earlier than the current time. Finding the file with no matching .done file is my concern
You can use a custom file filter, where you check for whether a .done file exists or not. And also do the age check.
The custom filter is just java code so you can write a little bit of code to see if file with .done exists or not. And as well check that header if the age is > 24h
There is a filter option you can use: http://camel.apache.org/file2
Related
I am attempting to understand the 2020q1 data set found here: https://www.sec.gov/dera/data/financial-statement-data-sets.html,
and am using the reference documentation inside the 2020q1 folder as a “readme” file. The reference documentation specifies that
within the Presentation (pre) data set, the “report” field is a numeric (integer) whose “value refers to the “R file” as posted
on the EDGAR Web site.” I have found no such file after extensive search, and am left with no method of interpreting the “report”
field and all associated data. Please link to the appropriate R File or guide me in the right direction for assistance if possible. Thanks!
So a point of clarification upfront, cause this confused me as well, the "R file" in question is not a script file of the R language. Instead, it simply seems to be a report file that holds the formatted data.
So after digging deeper into the readme, I found the following detail in the description for the SUB.txt data.
Note: To access the complete submission files for a given filing, please see the SEC EDGAR website. The SEC website folder HTTP(s)://www.sec.gov/Archives/edgar/data/{cik}/{accession}/ will always contain all the data sets for a given submission. To assemble the folder address to any filing referenced in the SUB data set, simply substitute {cik} with the cik field and replace {accession} with the adsh field (after removing the dash character). The following sample SQL Query provides an example of how to generate a list of addresses for filings contained in the SUB data set:
· select name,form,period, 'http(s)://www.sec.gov/Archives/edgar/data/' + ltrim(str(cik,10))+'/' + replace(adsh,'-','')+'/'+instance as url from SUBM subm order by period desc, name
Therefore, it looks like we have to correlate each "adsh" submission ID with the "cik" company ID in order to get the link we are looking for.
Doing this for the first entry of pre.txt, we get an adsh value of "0001032208-20-000006". I simply searched through sub.txt with notepad and found its associated cik of "1032208" which belongs to "SEMPRA ENERGY". Therefore, we generate the following link: http://www.sec.gov/Archives/edgar/data/1032208/000103220820000006
From there, we find a directory of files associated with the given submission. Inside is a collection of files with the prefix of "R". Simply clicking on them will open them in your browser, using the "report" and "line" fields, we can then correlate which file we want. Notice that we can add "/R{number}.htm" at the end of the link we generated to find this folder to get a given report number.
If you know what you are looking for, doing this by hand with "ctr+f" find functionality should be fine. Otherwise, you may want to open these docs in excel to generate the links for you.
First off, sorry for the lack of code; verbiage is the only thing that I can think of using to describe the problem
I have a link which when I run it using:
webbrowser.open('http://www.MyLink&ticker=IBM')
automatically downloads IBM data for me. The file is automatically named download.csv
The next time I run this it tries to use the same name and therefore I get a file download(1).csv and so on.
There are 2 things I wish to accomplish.
Open the file download.csv and rename it ticker_Date.csv where ticker would be IBM or GE, etc and date is the previous biz day. This way I avoid file names download(1) download(2) etc and can actually have the file name associated with the ticker (IBM for example) and the trade date the data is from
Not open 20 web browser pages if I request 20 different tickers by closing the webbrowser after the download is complete.
I tried:
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
I don't seem to be able to get this to work. To be clear, when I enter the url a download automatically takes place. All of the resources I have seen have referenced "downloading" a file with a "file_Name", but this is automatically done in my case. I don't have a file name per-say
thanks for any help
Iam trying to automate the testing of a java bundle,Which will process once a file is dropped in a particular folder.
can we drop a file to a folder location automatically using camel,or at a set period of time(not intervals)?
is this possible purely by camel or should we incorporate other frameworks?
sure, you can use the camel-file component to produce (create files somewhere) and consume (read/process files from somewhere) and optionally control the initial/polling delays easily with attributes...
here is a simple example of consuming->processing->producing
from("file://inputdir").process(<dosomething>).to("file://outputdir")
alternatively, you could periodically produce a file and drop it somewhere
from("timer://foo?fixedRate=true&period=60000").process(<createFileContent>").to("file://inputdir");
Although camel could do this by creating a timer endpoint, then setting the file content and writing to a file endpoint, my answer would be to simply use a bash script. No camel needed here.
Pseudo bash script:
while [ true ]
do
cp filefrom fileto
pauze 10s
done
How to create log file with custom file name in cakephp 2.5.1.
I am written following code
$this->log("Data : ".print_r($this->data,true),'debug');
It create's debug.log file in tmp/log folder but when I write custom name like
$this->log("Data : ".print_r($this->data,true),'data');
This code do not create any log file. My Question is how to create log file with custom name?
Same answer as for the other question: Start reading the manual. A huge part of the profession is to read documentation.
Changed in version 2.5.
CakeLog does not auto-configure itself anymore. As a result log files
will not be auto-created anymore if no stream is listening. Make sure
you got at least one default stream set up, if you want to listen to
all types and levels. Usually, you can just set the core FileLog class
to output into app/tmp/logs/:
I have to serve captcha image (in Coldfusion) without saving permanent files per request, so I am using cfimage without a source destination and the action set to captcha.
This returns a file like this:
From what I've tested, these files are seem to be temporary they become unavailable after around 5min. This fits my need, but I've been trying to find documentation regarding CFFileServlet, how it works, etc... with no luck...
Any thoughts?
Thanks, Phill
Not documented AFAIK, and yes they get deleted around 5 minutes after created.