Suppose I have an open file. How can I detect when the file is changed by another program in the background. Some text editors can detect and update the open file if it is changed by another process.
I'm specifically asking for this with C under Linux(this seems to be OS dependent).
If you don't want to poll the file using stat, and don't mind being Linux-specific, then you can use the inotify API. Your kernel needs to be 2.6.13 or newer and glibc 2.4 or newer (which they will be if you're targeting anything from the past 2 or 3 years). The API basically gives you a file descriptor that you can poll or select, and read to get information about modified files. If your application is interactive, like an editor, then it will typically have some sort of event loop that calls select or poll, and can watch your inotify file descriptor for events.
Using inotify is generally preferable stat, because you get notifications immediately and you don't waste time and disk I/O polling when the file isn't changing. The downside is that might not work over NFS or other networked file systems, and it's not portable.
This page at IBM Developerworks gives some example C code, and the man page is the definitive reference.
use stat function. Example in the page.
Text editors I've seen on Windows and Linux have done it the same way: they don't check to see whether the file has actually changed, they just looking at the file's stat mtime.
Related
I want to build file sync software. Is there any way to get exact file changes (or at least changes size) with kernel systems like I-notify or others?
EDIT:
I'm interested in the following scenario with I-notify:
When getting IN_MODIFY event on a file I want retrieve in some way changed lines of the file (some kind of a file diff format). Are there any linux kernel tools to achieve this?
Even if there were such a kernel feature, it would not work in practice. You see, most editors modify files by creating a copy, then renaming it over the original one. This way the user is assured of getting either the old contents or the new contents, never a mix between the two.
The only real option is to take snapshots of the file (at e.g. when file is closed when it was open for writing, or when the file is replaced with a new one), and compare the snapshots, to find which part was changed.
Comparing two versions of a file to see which part of it was changed is itself a difficult question, as it definitely depends on the file format. For source code, unified diffs work well, but for other types (including plain text files that are not line-oriented), it's not that simple.
Could you please refine your question? The inotify API on Linux does monitor such changes, and similar changes such as if a file was open, if a file inside a directory (or the directory itself) was moved and file deletions etc.
For more, see here:
(http://man7.org/linux/man-pages/man7/inotify.7.html)
EDIT:
I believe I misread the question the first time around, if I did, yes such programs exist and the inotify API is the primary one existing within the Linux kernels. See the above link for a comprehensive guide on the different functions it provides.
What is the best way to create an "atomic" snapshot of file contents in Linux? Emphasis is not on performance, but on getting contents as a whole.
I may think of using sendfile(2) (since 2.6.33) or splice(2), but neither have any indication of operation atomicity. Both are run in the kernel-space entirely, but at least sendfile(2) implies it's using mmap(2) and mmap gives no guarantees that writes to the same mmaped (as MAP_SHARED) region in other processes won't be visible even with MAP_PRIVATE (probably they will, because that are the same pages).
Taking that this functions are writing with performance in mind and sendfile(2) is optimized to be used with DMA, I may only assume that they just copy memory in some background kernel thread and it's quite possible that other operations may also affect the data being copied.
So the only possible solution I see is to place a read lease with fcntl(2) (FD_SETLEASE) and copy file as normal, but if someone opens it for writing, either try to "rush" it (very reliable, I know) and beat the timer, or just give up and try later. Is that correct?
So the only possible [filesystem-independent] solution I see is to place a read lease with fcntl(2) (FD_SETLEASE) and copy file as normal, but if someone opens it for writing, either try to "rush" it (very reliable, I know) and beat the timer, or just give up and try later. Is that correct?
Almost; there is also fanotify. Plus, as mentioned in a comment, there are some filesystem-specific options, and some possibilities only available in certain configurations.
The lease break timer is configurable, /proc/sys/fs/lease_break_time in seconds, and the default is 45 seconds.
"Just give up and try later" is also a bit defeatist; you do have ways to monitor when the snapshot might work. Consider placing an inotify IN_CLOSE_WRITE and IN_CLOSE_NOWRITE watch on the file, and try the snapshot whenever you receive such an event.
fanotify:
For a few years now, I've been monitoring the progress of Linux fanotify, in the hopes that it would grow enough features that it could be used for automagic file versioning. Essentially, whenever someone opens the file with write permissions, the current file would be snapshot to temporary storage, marked with some metadata (timestamp, real human user (backtracked through sudo/su), and so on). When that descriptor is closed, another snapshot is taken, and a helper thread/process diffs the two, annotating the changes (or even pushing it to git).
It is limited to local filesystems, but with 2.6.37 and later kernels (including 3.x), the interface is sufficient for specific files, or an entire mount. In your case, the fanotify interface allows similar features to file leases, except for local filesystems only, but you can simply deny any accesses during the snapshot. (One can argue whether that is a good idea at all, especially if the file to be snapshotted is a system or configuration file; many programmers overlook error checking, because "some files just have to be always accessible, or your system is broken".)
As far as my change monitoring goes, fanotify should now have all sufficient features, but only if an entire mount is monitored. I was hoping to monitor configuration files on multi-admin clusters, but those files reside on the same mount as all system libraries and binaries do, so the monitoring causes considerable overhead. So much so, that it seems more appropriate to just modify SSH configuration, console configuration (getty etc.), sudo configuration, and possibly su, to always include a dynamic library that interposes file access syscalls, and basically does the versioning on behalf of the user. This way service binaries are not affected, only user actions are monitored.
This might work under some circumstances:
(Optional) Do something to prevent new processes to open the file:
a/ rename the file
b/ restrict file permissions
Find all existing file readers/writers via lsof and kill -STOP them
Do your snapshot
kill -CONT all readers/writers
(Optional) Restore action 1.
I would like to open an anonymous file that is what would be the result under linux of opening a file an unlink it or using memfd_create, but none of these seem to be available under windows (you could make delete a file work, but it's name doesn't seem to be removed until the file is closed). Getting a file descriptor that isn't backed with something visible in the file system.
Is there a way to achieve this under windows? Preferably I'd like it to never appear in the file system.
The reason why I want this is because I need a FILE* to be sent as an argument to a function that expects that (and I don't want it to clobber the file system). Changing the libraries does not look like a feasible option (besides the libraries has to work on other OSes as well - so they can't rely on windows specific abstractions anyway).
The most reasonably close to memfd_create you have in Windows are Memory-Mapped files. MSDN article about it here: http://go.microsoft.com/fwlink/?linkid=180801
But basically, the CreateFileMapping/OpenFileMapping API calls.
This does not use the physical disk (unless it needs the memory paged on disk) for it, but as far as I know, neither does memfd_create
No, there is no such thing as an anonymous file in Windows.
(Of course, that does not necessarily mean that you cannot have a FILE * that does what you need; for example, I like Ross's suggestion of using a named pipe.)
I want to create a project to lock file and folders in ubuntu by face detection through opencv using C language. Can you please let me know it is possible and how can i do it.
Can't help you with the opencv part, but "lock file and folders" could mean a few things:
You want to change permissions of files so that a given user/group can/cannot
access them. If this is the case, you want the chmod function.
See man 2 chmod. Seems like this is probably what you're after?
Usually, "file locking" on Linux refers to a means to prevent other processes from accessing a file without changing permissions via either:
Mandatory file locking via lockf (or fcntl).
Advisory file locking via flock.
If file locking is what you're after, here are the "see also" documents referred to by the man pages on lockf and/or flock:
https://www.kernel.org/doc/Documentation/filesystems/mandatory-locking.txt
https://www.kernel.org/doc/Documentation/filesystems/locks.txt
Note: Others have indicated you might want to use the C++ API for opencv. All of these functions should work just fine from C++ too.
I'm writing a TFTP server program for university, which needs exclusive access to the files it opens for reading. Thus it can be configured that if a file is locked by another process that it waits for the file to become unlocked.
Is there any way on Win32 to wait for a file become unlocked without creating a handle for it first?
The reason I ask, is that if another process calls CreateFile() with a dwShareMode that is incompatible to the one my process uses, I won't even be able to get a file handle to use for waiting on the lock using LockFileEx().
Thanks for your help in advance!
If you take a look at the Stack Overflow questions What Win32 API can be used to find the process that has a given file open? and SYSTEM_HANDLE_INFORMATION structure, you will find links to code that can be used to enumerate processes and all open handles of each running process. This information can be used to obtain a HANDLE to the process that has the file open as well as its HANDLE for the file. You would then use DuplicateHandle() to create a copy of the file HANDLE, but in the TFTP process' handle table. The duplicated HANDLE could then be used by the TFTP process with LockFileEx().
This solution relies on an internal function, NtQuerySystemInformation(), and an undocumented system information class value that can be used to enumerate open handles. Note that this feature of NtQuerySystemInformation() "may be altered or unavailable in future versions of Windows". You might want to use an SEH handler to guard against access violations were that to happen.
As tools from MS like OH and Process Explorer do it, it is definitely possible to get all the handles opened by a process. From there to wait on what you'd like the road is still long, but it is a beginning :)
If you have no success with the Win32 API, one place to look at is for sure the NT Native API http://en.wikipedia.org/wiki/Native_API
You can start from here http://msdn.microsoft.com/en-us/library/windows/desktop/ms724509%28v=vs.85%29.aspx and see if it works with the SystemProcessInformation flag.
Look also here for a start http://nsylvain.blogspot.com/2007/09/how-list-all-open-handles.html
The native API is poorly documented, but you can find resources online (like here http://www.osronline.com/article.cfm?id=91)
As a disclaimer, I should add that the Native API is somehow "internal", and therefore subject to change on future versions. Some functions, however, are exposed also publicly in the DDK, at kernel level, so the likelihood of these functions to change is low.
Good luck!