Here is a simple C program for illustration:
#include <windows.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
//MoveFile(argv[0], "dst.exe");
getchar();
return 0;
}
make an test.exe from code above.
Now execute test.exe, the test.exe hangs there due to getchar(), then I can cut and paste this exe freely.
But when I uncomment that MoveFile(argv[0], "dst.exe");, I was hoping it could move itself to dst.exe, it turns out to have a dst.exe, while program.exe is still there, just like CopyFile() does.
From what I know, in Windows, when exe is running I can rename it, move it, but not deleting it, that MoveFile() behaves as a combination of CopyFile() and DeleteFile()
And also see this from Microsoft doc MoveFileEx.
BOOL WINAPI MoveFileEx(
_In_ LPCTSTR lpExistingFileName,
_In_opt_ LPCTSTR lpNewFileName,
_In_ DWORD dwFlags
);
dwFlags has an option MOVEFILE_COPY_ALLOWED
the file is to be moved to a different volume, the function simulates the move by using the CopyFile and DeleteFile functions.
If the file is successfully copied to a different volume and the original file is unable to be deleted, the function succeeds leaving the source file intact.
This value cannot be used with MOVEFILE_DELAY_UNTIL_REBOOT.
Further confirming my guess, and I tested with MoveFileEx() with option MOVEFILE_REPLACE_EXISTING , recompiled the program, run it, now MoveFileEx() just returned as fail, not even dst.exe generated.
But I can definitely cut and paste that exe while running, MoveFileEx() should does so, why???
If they can't, what should I do to make it just like cut and paste.
If the target destination is on the same volume, MoveFile just updates the corresponding directory entries. The file's MFT record is not changed, its index remained the same, its contents is not touched. Because the file is not affected at all, you can move it within the same directory (i.e. rename) or within the same volume even if the file is in use (note: this is true for files being executed; in general, this is true only if the file was open with FILE_SHARE_DELETE).
If the target directory is on another volume, the system needs to copy it (will fail if the file is open in exclusive mode) and delete it on the old volume (will fail unconditionally if the file is in use).
Cut&paste works ok within the same volume and does not on different volumes. The reason is that the file clipboard operations use a different technique than the text ones.
When you select a text and press Ctrl-X, the text string is moved to an allocated global memory block and the block is passed to Windows. The program does not own it anymore. The text is physically in the Windows Clipboard and you can paste it as many times as you wish.
When you press Ctrl-X on a file, it is not moved to the Clipboard. The Clipboard will receive a file descriptor, which contains info about the file and the requested operation (this technique is known as delayed rendering). When you press Ctrl-C, the Clipboard will simply ask the object owner (i.e. Windows Explorer) to perform the requested operation. And the Explorer will perform it using the very same MoveFile.
Note that you can paste a cut file only once because the first Ctrl-C will invalidate the descriptor in the Clipboard. A copied file can be pasted several times.
Related
I am developing a command line application in C (linux environment) to edit a particular file format. This file format is a plain XML file, which is compressed, then encrypted, then cryptographically signed.
I'd like to offer an option to the user to edit this kind of file in an easy way, without the hassle of manualy extracting the file, editing it, and then compressing, encrypting and signing it.
Ideally, when called, my application should do the following:
Open the encrypted/compressed file and extract it to a temporary location (like /tmp)
Call an external text editor like nano or sublime-text or gedit depending on which is installed and maybe the user preferences. Wait until the user have edited the file and closed the text editor.
Read the modified temporary file and encrypt/compress it, replacing the old encrypted/compressed file
How can I achieve point no. 2?
I thought about calling nano with system() and waiting for it to return, or placing an inotify() on the temp file to know when it is modified by the graphical text editor.
Which solution is better?
How can i call the default text editor of the user?
Anything that can be done in a better way?
First, consider not writing an actual application or wrapper yourself, which calls another editor, but rather writing some kind of plugin for some existing editor which is flexible enough to support additional formats and passing its input through decompression.
That's not the only solution, of course, but it might be easier for you.
With your particular approach, you could:
Use the EDITOR and/or VISUAL command-line variables (as also pointed out by #KamilCuk) to determine which editor to use.
Run the editor as a child process so that you know when it ends execution, rather than having to otherwise communicate with it. Being notified of changes to the file, or even to its opening or closing, is not good enough, since the editor may make changes multiple files, and some editors don't even keep the file open while you work on it in them.
Remember to handle the cases of the editor failing to come up; or hanging; or you getting some notification to stop waiting for the editor; etc.
Call an external text editor like nano or sublime-text or gedit depending on which is installed and maybe the user preferences. Wait until the user have edited the file and closed the text editor.
Interesting question. One way to open the xml file with the user's default editor is using the xdg-open, but it doesn't give the pid of the application, in which user will edit the file.
You can use xdg-mime query default application/xml to find out the .desktop file of the default editor, but then you have to parse this file to figure out the executable path of the program - this is exactly how xdg-open actually works, in the search_desktop_file() function the line starting with Exec= entry is simply extracted from the *.desktop to call the editor executable and pass the target file as argument... What I am trying to say, is, after you find the editor executable, you can start it, and wait until it's closed, and then check if the file content has been changed. Well, this looks like a lot of unnecessary work...
Instead, you can try a fixed well-known editor, such as gedit, to achieve the desired workflow. You can also provide user a way (i.e. a prompt or config file) to set a default xml editor, i.e. /usr/bin/sublime_text, which then can be used in your programm on next run.
However, the key is here to open an editor that blocks the calling process, until user closes the editor. After the editor is closed, you can simply check if the file has been changed and if so, perform further operations.
To find out, if the file contents have been modified, you can use the stat system call to get the inode change time of the file, before you open the file, and then compare the timestamp value with the current one once it is closed.
i.e.:
stat -c %Z filename
Output: 1558650334
Wrapping up:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
void execute_command(char* cmd, char* result) {
FILE *fp;
fp = popen(cmd, "r");
fscanf (fp, "%s" , result);
}
int get_changetime(char* filename) {
char cmd[4096];
char output[10];
sprintf(cmd, "stat -c %%Z %s", filename);
execute_command(cmd, output);
return atoi(output);
}
int main() {
char cmd[4096];
char* filename = "path/to/xml-file.xml";
uint ctime = get_changetime(filename);
sprintf(cmd, "gedit %s", filename);
execute_command(cmd, NULL);
if (ctime != get_changetime(filename)) {
printf("file modified!");
// do your work here...
}
return 0;
}
I have a situation where I submitted jobs that have been running for five days but due to a bug introduced all the work could be lost. I made a 'system' call to compress the data file and then remove the original uncompressed file that could be as big as 4G. So I have this in the C code
strcpy(command,"data"); ////I should added a forward slash here "data/"
sprintf(command,"%scompress -c -i %s -o %s",command,name,out_name);
system(command);
remove(name); /////This is the problem
The bug is in the sprintf line, in which what I wanted to do was to call a program in data/compress, but due to the missing '/' the system command fails. And thus the data produced is not compressed AND then immediately the original file is DELETED leaving me with nothing! If it was compressed it would have been OK.
There are currently five running jobs in such a state. I need to divert this behavior somehow so that I don't lose five days work. I am thinking to create a fake script named 'datacompress' in the current directory to change the behavior of the running program. Can I do this or are there better options, if at all?
You can make datacompress a symbolic link to data/compress. Oops, this won't work unless the process's $PATH includes ..
Another option: remove the user's write permission to the directory containing name. This will cause the remove() function to fail.
If your system has Access Control Lists, remove the process's delete permission on the uncompressed file.
While you're trying to come up with a solution, you can suspend the process with:
kill -STOP <pid>
Create hard links (not symbolic links) to the data files:
ln datafile datafile.bkp
When the program removes the original datafile, the file's contents will remain under the .bkp filename.
And then fix the program to check error status of important things like the compress command.
I have three processes designed to run constantly in both Linux and Mac OS X environments. One process (the Downloader) downloads and stores a local copy of a large XML file every 30 seconds. Two other processes (the Workers) use the stored XML file for input. Each Worker starts and runs at random times. Since the XML file is big, it takes a long time to download. The Workers also take a long time to read and parse it.
What is the safest way to setup the processes so the Downloader doesn't clobber the stored file while the Workers are trying to read it?
For Linux and Mac OS X machines that use inode based file systems, use temporary files to store the data while its being downloaded (and is an incomplete state). Once the download is complete, move the temporary file into its final location with an atomic action.
For a little more detail, there are two main things to watch out for when one process (e.g. Downloader) writes a file that's actively read by other processes (e.g. Workers):
Make sure the Workers don't try to read the file before the Downloader has finished writing it.
Make sure the Downloader doesn't alter the file while the Workers are reading it.
Using temporary files accommodates both of these points.
For a more specific example, when the Downloader is actively pulling the XML file, have it write to a temporary location (e.g. 'data-storage.tmp') on the same device/disk* where the final file will be stored. Once the file is completely downloaded and written, have the Downloader move it to its final location (e.g. 'data-storage.xml') via an atomic (aka linearizable) rename command like bash's mv.
* Note that the reason the temporary file needs to be on the same device as the final file location is to ensure the inode number stays the same and the rename can be done atomically.
This methodology ensures that while the file is being downloaded/written the Workers won't see it since it's in the .tmp location. Because of the way renaming works with inodes, it also make sure that any Worker that opened the file continues to see the old content even if a new version of the data-storage file is put in place.
Downloader will point 'data-storage.xml' to a new inode number when it does the rename, but the Worker will continue to access 'data-storage.xml' from the previous inode number thereby continuing to work with the file in that state. At the same time, any Worker that opens a new copy 'data-storage.xml' after Downloader has done the rename will see contents from the new inode number since it's now what is referenced directly in the file system. So, two Workers can be reading from the same filename (data-storage.xml) but each will see a different (and complete) version of the contents of the file based on which inode the filename was pointed to when the file was first opened.
To see this in action, I created a simple set of example scripts that demonstrate this functionality on github. They can also be used to test/verify that using a temporary file solution works in your environment.
An important note is that it's the file system on the particular device that matters. If you are using a Linux or Mac machine but working with a FAT file system (for example, a usb thumb drive), this method won't work.
im working in a code that detects changes in a file (a log file) then its process the changes with the help of fseek and ftell. but if the file get deleted and changed (with logrotate) the program stops but not dies, because it not detect more changes (even if the file is recreated). fseek dont show errors and eiter ftell.
how i can detect that file deletion? maybe a way to reopen the file with other FILE *var and comparing file descriptor. but how i can do that. ?
When a file gets deleted, it is not necessarily erased from your disk. In your case the program still has a handle to the old file. The old file handle will not get you any information about its deletion or replacement with another file.
An easy way to detect file deletion and recreation is using stat(2) and fstat(2). They give you a struct stat which contains the inode for the file. When a file is recreated (and still open) the files (old open and recreated) are different and thus the inodes are different. The inode field is st_ino. Yes, you need to poll this unless you wish to use Linux-features like inotify.
You can periodically close the file and open it again, that way you will open the newly created one. Files actually get deleted when there is no handle to the file (open file descriptor is a handle), you are still holding the old file.
On windows, you could set callbacks on the modifications of the FS. Here are details: http://msdn.microsoft.com/en-us/library/aa365261(VS.85).aspx
I recently ran out of disk space on a drive on a FreeBSD server. I truncated the file that was causing problems but I'm not seeing the change reflected when running df. When I run du -d0 on the partition it shows the correct value. Is there any way to force this information to be updated? What is causing the output here to be different?
In BSD a directory entry is simply one of many references to the underlying file data (called an inode). When a file is deleted with the rm(1) command only the reference count is decreased. If the reference count is still positive, (e.g. the file has other directory entries due to symlinks) then the underlying file data is not removed.
Newer BSD users often don't realize that a program that has a file open is also holding a reference. The prevents the underlying file data from going away while the process is using it. When the process closes the file if the reference count falls to zero the file space is marked as available. This scheme is used to avoid the Microsoft Windows type issues where it won't let you delete a file because some unspecified program still has it open.
An easy way to observe this is to do the following
cp /bin/cat /tmp/cat-test
/tmp/cat-test &
rm /tmp/cat-test
Until the background process is terminated the file space used by /tmp/cat-test will remain allocated and unavailable as reported by df(1) but the du(1) command will not be able to account for it as it no longer has a filename.
Note that if the system should crash without the process closing the file then the file data will still be present but unreferenced, an fsck(8) run will be needed to recover the filesystem space.
Processes holding files open is one reason why the newsyslog(8) command sends signals to syslogd or other logging programs to inform them they should close and re-open their log files after it has rotated them.
Softupdates can also effect filesystem freespace as the actual inode space recovery can be deferred; the sync(8) command can be used to encourage this to happen sooner.
This probably centres on how you truncated the file. du and df report different things as this post on unix.com explains. Just because space is not used does not necessarily mean that it's free...
Does df --sync work?