Strange timestamp duplication when renaming and recreating a file - file

I'm trying to rename a log file named appname.log into the form appname_DDMMYY.log for archiving purposes and recreate an empty appname.log for further writing. When doing this in Windows 7 using C++ and either WinAPI or Qt calls (which may be the same internally) the newly created .log file strangely inherits the timestamps (last modified, created) from the renamed file.
This behaviour is also observable when renaming a file in Windows Explorer and creating a file with the same name quickly afterwards in the same directory. But it has to be done fast. After clicking on "new Text File" the timestamps are normal but after renaming they change to the timestamps the renamed file had or still has.
Is this some sort of Bug? How can I rename a file and recreate it shortly afterwards without getting the timestamps messed up?

This looks like it is by design, perhaps to try to preserve the time for "atomic saving." If an application does something like (save to temp, delete original, rename temp to original) to eliminate the risk of a mangled file, every time you saved a file the create time would increase. A file you have been editing for years would appear to have been created today. This kind of save pattern is very common.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724320(v=vs.85).aspx
If you rename or delete a file, then restore it shortly thereafter, Windows searches the cache for file information to restore. Cached information includes its short/long name pair and creation time.
Notice that modification time is not restored. So after saving the file appears to have been modified and the creation time is the same as before.
If you create "a-new" and rename it back to "a" you get the old creation time of "a". If you delete "a" and recreate "a" you get the old creation time of "a".

This behaviour is called "File Tunneling". File Tunneling is allow "...to enable compatibility with programs that rely on file systems being able to hold onto file meta-info for a short period of time". Basically backward compatibility for older Windows systems that use a "safe save" function that involved saving a copy of the new file to a temp file, deleting the original and then renaming the temp file to the original file.
Please see the following KB article: https://support.microsoft.com/en-us/kb/172190 (archive)
As a test example, create FileA, rename FileA to FileB, Create FileA again (within 15 seconds) and the creation date will be the same as FileB.
This behaviour can be disabled in the registry as per the KB article above. This behaviour is also quite annoying when "forensicating" Windows machines.
Regards
Adam B

Here's a simple python script that repro's the issue on my Windows7 64bit system:
import time
import os
def touch(path):
with open(path, 'ab'):
os.utime(path, None)
touch('a')
print " 'a' timestamp: ", os.stat('a').st_ctime
os.rename('a', 'a-old')
time.sleep(15)
touch('a')
print "new 'a' timestamp: ", os.stat('a').st_ctime
os.unlink('a')
os.unlink('a-old')
With the sleep time ~15 seconds I'll get the following output:
'a' timestamp: 1436901394.9
new 'a' timestamp: 1436901409.9
But with the sleep time <= ~10 seconds one gets this:
'a' timestamp: 1436901247.32
new 'a' timestamp: 1436901247.32
Both files... created 10 seconds apart have the time created-timestamp!

Related

Date in NLog file name and limit the number of log files

I'd like to achieve the following behaviour with NLog for rolling files:
1. prevent renaming or moving the file when starting a new file, and
2. limit the total number or size of old log files to avoid capacity issues over time
The first requirement can be achieved e.g. by adding a timestamp like ${shortdate} to the file name. Example:
logs\trace2017-10-27.log <-- today's log file to write
logs\trace2017-10-26.log
logs\trace2017-10-25.log
logs\trace2017-10-24.log <-- keep only the last 2 files, so delete this one
According to other posts it is however not possible to use date in the file name and archive parameters like maxArchiveFiles together. If I use maxArchiveFiles, I have to keep the log file name constant:
logs\trace.log <-- today's log file to write
logs\archive\trace2017-10-26.log
logs\archive\trace2017-10-25.log
logs\archive\trace2017-10-24.log <-- keep only the last 2 files, so delete this one
But in this case every day on the first write it moves the yesterday's trace to archive and starts a new file.
The reason I'd like to prevent moving the trace file is because we use Splunk log monitor that is watching the files in the log folder for updates, reads the new lines and feeds to Splunk.
My concern is that if I have an event written at 23:59:59.567, the next event at 00:00:00.002 clears the previous content before the log monitor is able to read it in that fraction of a second.
To be honest I haven't tested this scenario as it would be complicated to set up as my team doesn't own Splunk, etc. - so please correct me if this cannot happen.
Note also I know that it is possible to directly feed Splunk other ways like via network connection, but the current setup for Splunk at our company is reading from log files so it would be easier that way.
Any idea how to solve this with NLog?
When using NLog 4.4 (or older) then you have to go into Halloween mode and make some trickery.
This example makes hourly log-files in the same folder, and ensure archive cleanup is performed after 840 hours (35 days):
fileName="${logDirectory}/Log.${date:format=yyyy-MM-dd-HH}.log"
archiveFileName="${logDirectory}/Log.{#}.log"
archiveDateFormat="yyyy-MM-dd-HH"
archiveNumbering="Date"
archiveEvery="Year"
maxArchiveFiles="840"
archiveFileName - Using {#} allows the archive cleanup to generate proper file wildcard.
archiveDateFormat - Must match the ${date:format=} of the fileName (So remember to correct both date-formats, if change is needed)
archiveNumbering=Date - Configures the archive cleanup to support parsing of filenames as dates.
archiveEvery=Year - Activates the archive cleanup, but also the archive file operation. Because the configured fileName automatically ensures the archive file operation, then we don't want any additional archive operations (Ex. avoiding generating extra empty files at midnight).
maxArchiveFiles - How many archive files to keep around.
With NLog 4.5 (Still in BETA), then it will be a lot easier (As one just have to specify MaxArchiveFiles). See also https://github.com/NLog/NLog/pull/1993

Replay a file-based data stream

I have a live stream of data based on files in different formats. Data comes over the network and is written to files in certain subdirectories in a directory hierarchy. From there it is picked up and processed further. I would like to replay e.g. one day of this data stream for testing and simulation purposes. I could duplicate the data stream for one day to a second machine and „record“ it this way, by just letting the files pile up without processing or moving them.
I need something simple like a Perl script which takes a base directory, looks at all contained files in subdirectories and their creation time and then copies the files at the same time of the day to a different base directory.
Simple example: I have files a/file.1 2012-03-28 15:00, b/file.2 2012-03-28 09:00, c/file.3 2012-03-28 12:00. If I run the script/program on 2012-03-29 at 08:00 it should sleep until 09:00, copy b/file.2 to ../target_dir/b/file.2, then sleep until 12:00, copy c/file.3 to ../target_dir/c/file.3, then sleep until 15:00 and copy a/file.1 to ../target_dir/a/file.1.
Does a tool like this already exist? It seems I’m missing the right search keywords to find it.
The environment is Linux, command line preferred. For one day it would be thousands of files with a few GB in total. The timing does not have to be ultra-precise. Second resolution would be good, minute resolution would be sufficient.

How do you set the modification time of a file in a ClearCase vob?

I have several files that I rsync'd over to the vob and they all have times 40 minutes in the future.
I tried touch, and all that does is maintain the time 40 minutes into the future from when I touch.
I guess that ClearCase is in charge of setting the modification time and is overriding touch.
Is there another way? Is there a way to tell ClearCase to stop messing up the file time?
What option did you use when adding those files to source control?
As explained in this help page:
To preserve the modification time of the file being checked in, use the -ptime option.
If you omit the -ptime option, the modification time of the new version is set to the checkin time.
The mkelem man page adds:
On some UNIX and Linux platforms, it is important that the modification time be preserved for archive files (libraries) created by ar(1) (and perhaps updated with ranlib(1)).
The link editor, ld(1), generates an error message if the modification time does not match a time recorded in the archive itself. Be sure to use this option, or (more reliably) store archive files as elements of a user-defined type, created with the mkeltype –ptime command. This causes –ptime to be invoked when the element is checked in.
Unless you remove those files and re-create them, I don't think you can change the "Created on" time.

Get last modified file without enumerating all files

In c#, given a folder path, is there a way to get the last modified file without getting all files?
I need to quickly find folders that have been updated after a certain time and if the file that was last modified is before the time, i want to skip the folder entirely.
I noticed that folder's last modified time does not get updated when one of its file get updated so this approach does't work.
No, this is why windows comes with indexing to speed up searching. The NTFS file system wasn't designed with fast searching in mind.
In any case you can monitor file changes which is not difficult to do. If it is possible to allow your program to run in the background and monitor changes then this would work. If you needed past history you could do an initial scan only once and then build up your hierarchy from their. As long as your program is always being ran then it should have a current snapshot and not have to do the slow scan.
You can also use the Window Search itself to find the files. If indexing is available then it's probably as fast as you'll get.
Try this.
DirectoryInfo di = new DirectoryInfo("strPath");
DateTime dt = di.LastWriteTime;
Then you should use
Directory.EnumerateFiles(strPath, "*.*", SearchOption.TopDirectoryOnly);
Then loop the above collection and get FileInfo() for each file.
I don't see a way how can you get the modified date of a file w/o getting reference to FileInfo() on that file.
I don't think FileInfo will get this file as far as I know.

Filesystem links on a FAT32 formatted storage

I know FAT32, as well as FAT16/12 neither support symbolic links nor hard-links. However I came up with this idea:
The FAT specification describes that every file is associated with a directory-entry. In my understanding, one could say that a file-entry in a directory somehow or other points to the file's content.
So, how can I define two directory-entries which point to the same file-content? Or, what could prevent me from doing so?
Use case: I have a USB mass storage device for my car radio, and I want to use directories as playlists since the radio software doesn't support playlists. So it isn't important to me how Windows behaves when doing this.
This should work for simple issues. I.e. it works as a hack / workaround and I don't know what happens if you rename / move / remove files. So, you should not do this on your main hdd.
I edited the directory-entries manually using a hex editor. I modified clusters as well as file-sizes and successfully faked hardlinks. My car-radio and even Windows (7, 64Bit) have no problems with playing back the original and "hard-linked" mp3-Files I used.
When I'm opening the device again in the hex-editor none of my modifications are changed back (See chkdsk issue in answer #1 - but as far as I know chkdsk has to be started manually, anyways.
What you are talking about ("two directory-entries which are pointing to the same file-content") are hard links. chkdsk will report them as cross-links and break them, "repairing" the files (in fact making the copies).
MichaelPh posted instructions on SuperUser:
https://superuser.com/a/486829/51237
It's possible to use Disk Probe (on XP only, I've yet to get it to write the changes on Win7) to modify the cluster a FAT Directory references. This method can be used to redirect the DCIM folder (or a subfolder) to point to the folder used by a different scan device.
Whether this is a good idea or not is a different matter and you use this at your own risk.
Insert the Eye-Fi card either in it's USB Card Reader or directly into an SSD slot and note the drive letter it's installed as (assumed to be F:\ for simplicity)
Ensure all Windows Explorer windows for the card and sub-directories are closed.
Run Disk Probe
Select Drives->Logical Volume
In the Open Logical Volume dialog double-click F:\ in the Logical Volumes list
Click the Set Active button for the Handle F: has been selected as. You can leave the handle as read-only for now.
Select Tools->Search Sectors...
Check Exhaustive Search, enter DCIM in Enter characters to search for and Search
You should find a match (mine is at 8192). Select No on the "Found match..." dialog to cancel the rest of the search.
Select Sectors->Read and increase Number of Sectors to at least two so that the whole directory table is included.
Find DCIM in the ASCII on the right of the Disk Probe screen, this is the start of the FAT entry for the directory. Make a note of the hex value of the 27th byte of the record (each entry is 32bytes), this is the directory cluster reference. This value is require to revert the DCIM directory back to normal use if required.
Find the entry for the directory you want to redirect DCIM to and again make a note of the 27th byte in the record.
Go back to the 27th byte of the DCIM record and change it to the value noted in step 11.
Select Sectors->Write and then click Write it on the Write Sector dialog. A warning will come up if you opened the sectors as read-only. Yes to overwrite if you're happy to make the change.
Opening the DCIM directory in Windows Explorer will now show the contents of the target directory.

Resources