The question is related with implementing file copy in linux:
1. Create a new file
2. Copying data to this file
3. If user want to stop before copying finished, this new file should be deleted
Do I need to call fsync before closing this file and then unlink it?
Because closing file will not do fsync, does unlink will not delete the file until all dirty data of this file finished written back to disc?
I want to make sure the file is really unlinked at the time unlink() command returned.
If user stops it before copying, Simply call fsync() and unlink(). It will sync data in to disk and deletes your file
Related
Assuming a plain text file, foo.txt, and two processes:
Process A, a shell script, overwrites the file in regular intervals
$ echo "example" > foo.txt
Process B, a C program, reads from the file in regular intervals
fopen("foo.txt", "r"); getline(buf, len, fp); fclose(fp);
In the C program, keeping the FILE* fp open after the initial fopen(), doing a rewind() and reading again does not seem to reflect the changes that have happened to the file in the meantime. Is the only way to see the updated contents by doing an fclose() and fopen() cycle, or is there a way to re-use the already opened FILE handle, yet reading the most recently written data?
For context, I'm simply trying to find the most efficient way of doing this.
On Unix/Linux, when you create a file with a name which already existed, the old file is not deleted or altered in any way. A new file is created and the directory is updated to point at the new file instead of the old one.
The old file will continue to exist as long as some directory entry points at it (Unix file systems allow the same file to be pointed to by multiple directories) or some program has an open file handle to the file, which is more relevant to your question.
As long as you don't close fp, it continues to refer to the original file, even if that file is no longer referenced by the filesystem. When you close fp, the file will get garbage collected automatically, and the next time you open foo.txt, you'll get a file descriptor for whatever file happens to have that name at that point in time.
In short, with the shell script you indicate, your C program must close and reopen the file in order to see the new contents.
Theoretically, it would be possible for the shell script to overwrite the same file without deleting it, but (a) that's tricky to get right; (b) it's prone to race conditions; and (c) closing and reopening the file is not that time-consuming. But if you did that, you would see the changes. [Note 1]
In particular, it's common (and easy) to append to an existing file, and if you have a shell script which does that, you can keep the file descriptor open and see the changes. However, in that case you would normally have already read to the end of the file before the new data was appended, and the standard C library treats the feof() indicator as sticky; once it gets set, you will continue to get an EOF indication from new reads. If you suspect that some process will be writing more data to the file, you should reset the EOF indication with fseek(fp, 0, SEEK_CUR); before retrying the read.
Notes
As #amadan points out in a comment, there are race conditions with echo text > foo.txt as well, although the window is a bit shorter. But you can definitely avoid race conditions by using the idiom echo text > temporary_file; mv -f temporary_file foo.txt, because the rename operation is atomic. Of course, that would definitely require you to close and reopen the file. But it's a good idea, particularly if the contents being written are long or critical, or if new files are created frequently.
Here is the setup: I have a shared file (lets call it status.csv) that is read by many processes (lets call them consumers) in a read-only fashion. I have one producer that periodically updates status.csv by creating a temp file, writing data to it and using the C function discussed here:
http://www.gnu.org/software/libc/manual/html_node/Renaming-Files.html
to rename the temp file (effectively overwrite) to status.csv so that the consumers can process the new data. It want to try and guarantee (as much as possible in the Linux world) that the consumers won't get a malformed/corrupted/half-old/half-new status.csv file (I want them to get either all of the old data or all of the new). I can't seem to guarantee this by reading the description of rename: it seems to guarantee that the rename action itself is atomic but I want to know if a consumer already has the status.csv file open, he will continue to read the same file as it was when it was opened, even if the file is renamed/overwritten by the producer in the middle of this reading operation.
I attempted to prototype this thinking that the consumers will get some type of error or a half old/half new file but it seems to always be in the state it was when it was open by the consumer even if renamed/overwritten multiple times.
BTW, these processes are running on the same machine (RHEL 6).
Thanks!
In Linux and similar systems, if a process has a file open and the file is deleted, the file itself remains undeleted until all processes close it. All that happens immediately is that the directory entry is deleted so that it cannot be opened again.
The same thing happens if rename is used to replace an open file. The old file descriptor still keeps the old file open. However, new opens will see the new file.
Therefore, for your consumers to see the new file, they must close and reopen the file.
Note: your consumers can discover if the file has been replaced by using the stat(2) call. If either the st_dev or st_ino entries (or both) have changed, then the file has been replaced and must be closed and reopened. This is how tail -F works.
In my program (on Mac OS X), I opened the file using following code.
int fd;
fd = open(filename, O_RDWR);
Program to delete the file is as follows:
unlink(filename);
In my case, I have same file which is opened and deleted. I observed the following:
After opening the file, I can delete it using this program and even by using rm command.
After deleting the file, read and write operations are working on the file without any problem.
I would like to know the reason behind this. How to prevent rm command or unlink(2) system call from deleting the file which is being opened?
You can't stop unlink(2) from unlinking a file which it has permission to unlink (i.e. it has write access to the directory).
unlink is not called unlink because nobody could think of a better name. It's called that because that is what it does; it unlinks the file from the directory. (A directory is just a collection of links; i.e. it associates names with the location of the corresponding data.) It does not delete the file; the file is garbage collected by the filesystem when there are no longer any links to it.
Open file descriptors are not the only way to keep links to files. Another, quite common, way is to use the link(1) command without the -s option. This creates "hard" links. If a file has several hard links, then removing one of the links (with unlink(2)) does just that -- it removes one of the links.
The rm command has a possibly more confusing name, but it, too, only removes the name, not the file. The file exists as long as someone has a link to it, including a running process.
First, rm command is calling unlink(2)
Then, unlinking an opened file is a normal thing to do on Linux or others Unixes (e.g. MacOSX). It is the canonical way to get temporary files (like tmpfile(3) probably does).
You should understand what inodes are, and realize that a file is not its name or file path, but essentially an inode. A file can have zero, one, or several file paths or names (one can add more with the link(2) syscall, provided all the names sit in the same filesystem). Directory entries associate names to inodes.
So there is no (POSIX-ly portable) way to prohibit I/O on open-ed files without any names.
For some opened file, the kernel has reference counters to its inode, and keep that inode till all processes having open(2)-ed it did close(2) it or have terminated.
See also inode(7) and credentials(7).
It's a normal Situation in UNIX SYSTEM. when you rm or unlink an opened file. UNIX system just mark a flag , and won't really delete the file desception. until the file is closed. and it will be really deleted in the file system.
It's protection to help the daemon work fine.
A link is a name associated to some file (a file is basically unamed). Note that a file could have different names (try ln).
unlink() removes one of this association to a file. If you remove the last link to a file, this just makes you unable to access the file by a name. But, this doesn't mean that the file is unusable, as a file could have been opened and his currently read/written by some application.
A file is removed if and only if :
- there is no link on it
- it is not currently opened by any application
Consider the following scenario: I am opening a tar file (say abc.tar.gz), writing the data, and before closing the file descriptor, I am trying to extract the same file.
I am unable to do so. But if I extract the file after having closing the fd, it works fine.
I wonder what could be the reason.
All files has a position where data is read or written. After writing to the file, the position is at the end. Trying to read will attempt to read from that position. You have to change the position to the beginning of the file with a function like lseek.
Also, you did open the file in both read and write mode?
Edit
After reading your comments, I see you do not actually read the file from inside your program, but from an external program. Then it might be as simple as you not flushing the file to disk, which happens automatically when closing a file. You might want to check the fsync function for that, or possible the sync function.
im working in a code that detects changes in a file (a log file) then its process the changes with the help of fseek and ftell. but if the file get deleted and changed (with logrotate) the program stops but not dies, because it not detect more changes (even if the file is recreated). fseek dont show errors and eiter ftell.
how i can detect that file deletion? maybe a way to reopen the file with other FILE *var and comparing file descriptor. but how i can do that. ?
When a file gets deleted, it is not necessarily erased from your disk. In your case the program still has a handle to the old file. The old file handle will not get you any information about its deletion or replacement with another file.
An easy way to detect file deletion and recreation is using stat(2) and fstat(2). They give you a struct stat which contains the inode for the file. When a file is recreated (and still open) the files (old open and recreated) are different and thus the inodes are different. The inode field is st_ino. Yes, you need to poll this unless you wish to use Linux-features like inotify.
You can periodically close the file and open it again, that way you will open the newly created one. Files actually get deleted when there is no handle to the file (open file descriptor is a handle), you are still holding the old file.
On windows, you could set callbacks on the modifications of the FS. Here are details: http://msdn.microsoft.com/en-us/library/aa365261(VS.85).aspx