I'm working on a Linux/C application with strict timing requirements. I want to open a directory for reading without blocking on I/O (i.e. succeed only if the information is immediately available in cache). If this request would block on I/O I would like to know so that I can abort and ignore this directory for now. I know that open() has a non-blocking option O_NONBLOCK. However, it has this caveat:
Note that this flag has no effect for regular files and
block devices; that is, I/O operations will (briefly)
block when device activity is required, regardless of
whether O_NONBLOCK is set.
I assume that a directory entry is treated like a regular file. I don't know of a good way to prove/disprove this. Is there a way to open a directory without any I/O blocking?
You could try using COPROC command in linux to run a process in background. Maybe it could work for you.
Related
It's possible to change UID/GID of current process as it runs programatically with setresgid/setresuid which affects future files access rights.
However what happens to already opened or memory mapped files? Are they still accessible for i/o operations like read/write? I'm asking more in context of "not explicit" i/o operations performed by libraries, for example sqlite database or other libraries that operate on files more internally. Files opened in DIRECT_IO mode sound even more uncertain in this aspect.
When you open a file, your ability to do so is determined by your effective uid and gid at the time you open the file.
When you change your effective uid or gid, it has no effect on any open file descriptors that you may have.
In most cases, if you have a valid file descriptor, that's all you need to read or write the resource that descriptor is connected to. The fact that you hold the valid file descriptor is supposed to be all the proof you need that you have permission to read/write the underlying resource.
When you read or write using an ordinary file descriptor, no additional authorization checks are performed. This is partly for efficiency (because those authentication checks would be expensive to perform each time), and partly so that -- this may be exactly what you are trying to do -- you can open a privileged resource, downgrade your process's privileges, and continue to access the open resource.
Bottom line: Yes, it's entirely possible for a process to use an open file descriptor to read or write a file which (based on its current uid/gid) it would not be able to open.
Footnote: What I've said is true for ordinary Unix file descriptors connected to ordinary resources (files, devices, pipes, network streams, etc.). But as #Mark Plotnick reminds in a comment, some file descriptors and underlying resources are "different" -- NFS and Linux /proc files are two examples. For those, it's possible for additional checks to be performed at the time of read/write.
I need to update a log file according to the messages produced by two different modules which may be running simultaeously.
So is it possible to open and write a file simultaneously in two programs?
Sys Spec: SLES 11 x86_64.
You can do one of the following:
Use flock() (or a similar mechanism) to synchronize the writes on the open file descriptors (as already answered).
Use open() and close() (or similar) repeatedly on systems that support (or even enforce) exclusive open().
Depend on buffered output to send out log lines uninterrupted. This is often used with stderr logging, as a possible race condition isn't usually a problem here.
Use a logging service and only open() the file there. Other processes communicate with the logging service via IPC. You can use a custom logging service or a tool like syslog or journald. Both of them AFAIK support logging from non-root processes as well.
I would personally prefer the last option because its design is the cleanest one and it doesn't depend so much on OS-specific behavior. If your application consists of multiple processes started by the main process, then the main process may perform as the logging service as well and create pipes before spawning the child processes. If the processes are started separately, you can have a separate service that listens on a TCP/IP socket or (if your system supports it) a local domain socket.
Yes. A file can be opened by several processes/programs simulatneously. Multiple processes/programs can read & write in a file simultaneously but the end result of writing in the same file at the same time may be undefined. So it is better to use locks.
On Linux you can use: flocks
I'm doing some basic file reading using open, read, and close (Files are opened with access mode O_RDONLY).
When it comes time to close the file, I can't think of a good way to handle a possible file close error to make sure that the file is closed properly.
Any suggestions?
From my experience close will succeed even when it fails. There are several reasons for this.
I suspect that one of the big reasons close started to fail on some operating systems was AFS. AFS was a distributed file system from the '80s with interesting semantics - all your writes were done to a local cache and your data was written to the server when you close the file. AFS also was cryptographically authenticated with tokens that expired after a time. So you could end up in an interesting situation where all the writes you did to a file were done while your tokens were valid, but close which actually talked to the file server could be done with expired tokens meaning that all the data you wrote to the local cache was lost. This is why close needed to communicate to the user that something went wrong. Most file editors handle this correctly (emacs refuses to mark the buffer as not dirty, for example), but I've rarely seen other applications that can handle this.
That being said, close can't really fail anyway. close is implicit during exit, exec (witch close-on-exec file descriptors) and crashes that dump core. Those are situations where you can't fail. You can't have exit or a crash fail just because closing the file descriptor failed. What would you do when exit fails? Crash? What if crashing fails? Where do we run after that? Also, since almost no one checks for errors in close if failing was common you'd end up with file descriptor leaks and information leaks (what if we fail to close some file descriptor before spawning an unprivileged process?). All this is too dangerous for an operating system to allow so all operating systems I've looked at (*BSD, Linux, Solaris) close the file descriptor even if the underlying filesystem close operation has failed.
In practice this means that you just call close and ignore any errors it returns. If you have a graceful way of handling it failing like editors do you can send a message to the user and let the user resolve the problem and reopen the file, write down the data and try to close again. Don't do anything automatically in a loop or otherwise. Errors in close are beyond your control in an application.
I guess the best thing to do is to retry a couple of times, with some short delay between attempts, then log the problem and move on.
The manual page for close() mentions EINTR as being a possible error, which is why re-trying can help.
If your program is about to exit anyway, I wouldn't worry too much about this type of error checking, since any resources you've allocated are going to be de-allocated by the operating system anyway (on most/typical desktop/server platforms, that is).
What ways are there available, for exclusively opening a device file (say, the display frame buffer)?
[Info: I already know about flock() & friends, which have an effect only when the other applications are also using it (in other words: open() will succeed but flock() will fail if already locked) --> but still the device handle retrieved from open() can be used to write to the display..]
What about cases when I want to enforce such an exclusive access on a device files? How would such an enforcement be possible?
From fcntl(2):
To make use of mandatory locks, mandatory locking must be enabled
both on the filesystem that contains the file to be locked, and on
the file itself.
...also, you need to enable CONFIG_MANDATORY_FILE_LOCKING in the kernel.
Mandatory locking is enabled on a filesystem using
the "-o mand" option to mount(8), or the MS_MANDLOCK flag for
mount(2). Mandatory locking is enabled on a file by disabling group
execute permission on the file and enabling the set-group-ID permis‐
sion bit (see chmod(1) and chmod(2)).
Mandatory locking is not specified by POSIX. Some other systems also
support mandatory locking, although the details of how to enable it
vary across systems.
So, as you request a posix-compliant solution, the answer is: no, there is not such a feature in the POSIX standard.
try lockf() : apply, test or remove a POSIX lock on an open file
To open a device you should use open system call in linux and check the list of available devices for example /dev/ttyUSB0 or /dev/ttyS0 etc. and open it and you will get a descriptor to write and read on to the device is you open a device to communicate.
To know the further details follow the link :
http://www.firmcodes.com/lower-level-file-handling-in-linux/
If you want to get exclusive access to a device, create a lock file in /var/lock. The process that can create the lock file with open("my_device.lock", O_CREAT|O_EXCL, 0777) gets access to the device, the other processes have to wait. After the process is done using the device, it closes the file.
Such a lock is only advisory and doesn't guarantee that no other process (that you are not aware of) accesses the device.
Looking for information regarding the advantages and disadvantages of both fcntl and lockf for file locking. For example which is better to use for portability? I am currently coding a linux daemon and wondering which is better suited to use for enforcing mutual exclusion.
What is the difference between lockf and fcntl:
On many systems, the lockf() library routine is just a wrapper around fcntl(). That is to say lockf offers a subset of the functionality that fcntl does.
Source
But on some systems, fcntl and lockf locks are completely independent.
Source
Since it is implementation dependent, make sure to always use the same convention. So either always use lockf from both your processes or always use fcntl. There is a good chance that they will be interchangeable, but it's safer to use the same one.
Which one you chose doesn't matter.
Some notes on mandatory vs advisory locks:
Locking in unix/linux is by default advisory, meaning other processes don't need to follow the locking rules that are set. So it doesn't matter which way you lock, as long as your co-operating processes also use the same convention.
Linux does support mandatory locking, but only if your file system is mounted with the option on and the file special attributes set. You can use mount -o mand to mount the file system and set the file attributes g-x,g+s to enable mandatory locks, then use fcntl or lockf. For more information on how mandatory locks work see here.
Note that locks are applied not to the individual file, but to the inode. This means that 2 filenames that point to the same file data will share the same lock status.
In Windows on the other hand, you can actively exclusively open a file, and that will block other processes from opening it completely. Even if they want to. I.e., the locks are mandatory. The same goes for Windows and file locks. Any process with an open file handle with appropriate access can lock a portion of the file and no other process will be able to access that portion.
How mandatory locks work in Linux:
Concerning mandatory locks, if a process locks a region of a file with a read lock, then other processes are permitted to read but not write to that region. If a process locks a region of a file with a write lock, then other processes are not permitted to read nor write to the file. What happens when a process is not permitted to access the part of the file depends on if you specified O_NONBLOCK or not. If blocking is set it will wait to perform the operation. If no blocking is set you will get an error code of EAGAIN.
NFS warning:
Be careful if you are using locking commands on an NFS mount. The behavior is undefined and the implementation widely varies whether to use a local lock only or to support remote locking.
Both interfaces are part of the POSIX standard, and nowadays both interfaces are available on most systems (I just checked Linux, FreeBSD, Mac OS X, and Solaris). Therefore, choose the one that fits better your requirements and use it.
One word of caution: it is unspecified what happens when one process locks a file using fcntl and another using lockf. In most systems these are equivalent operations (in fact under Linux lockf is implemented on top of fcntl), but POSIX says their interaction is unspecified. So, if you are interoperating with another process that uses one of the two interfaces, choose the same one.
Others have written that the locks are only advisory: you are responsible for checking whether a region is locked. Also, don't use stdio functions, if you want the to use the locking functionality.
Your main concerns, in this case (i.e. when "coding a Linux daemon and wondering which is better suited to use for enforcing mutual exclusion"), should be:
will the locked file be local or can it be on NFS?
e.g. can the user trick you into creating and locking your daemon's pid file on NFS?
how will the lock behave when forking, or when the daemon process is terminated with extreme prejudice e.g. kill -9?
The flock and fcntl commands behave differently in both cases.
My recommendation would be to use fcntl. You may refer to the File locking article on Wikipedia for an in-depth discussion of the problems involved with both solutions:
Both flock and fcntl have quirks which
occasionally puzzle programmers from
other operating systems. Whether flock
locks work on network filesystems,
such as NFS, is implementation
dependent. On BSD systems flock calls
are successful no-ops. On Linux prior
to 2.6.12 flock calls on NFS files
would only act locally. Kernel 2.6.12
and above implement flock calls on NFS
files using POSIX byte range locks.
These locks will be visible to other
NFS clients that implement
fcntl()/POSIX locks.1 Lock upgrades
and downgrades release the old lock
before applying the new lock. If an
application downgrades an exclusive
lock to a shared lock while another
application is blocked waiting for an
exclusive lock, the latter application
will get the exclusive lock and the
first application will be locked out.
All fcntl locks associated with a file
for a given process are removed when
any file descriptor for that file is
closed by that process, even if a lock
was never requested for that file
descriptor. Also, fcntl locks are not
inherited by a child process. The
fcntl close semantics are particularly
troublesome for applications which
call subroutine libraries that may
access files.
I came across an issue while using fcntl and flock recently that I felt I should report here as searching for either term shows this page near the top on both.
Be advised BSD locks, as mentioned above, are advisory. For those who do not know OSX (darwin) is BSD. This must be remembered when opening a file to write into.
To use fcntl/flock you must first open the file and get its ID. However if you have opened the file with "w" the file will instantly be zeroed out. If your process then fails to get the lock as the file is in use elsewhere, it will most likely return, leaving the file as 0kb. The process which had the lock will now find the file has vanished from underneath it, catastrophic results normally follow.
To remedy this situation, when using file locking, never open the file "w", but instead open it "a", to append. Then if the lock is successfully acquired, you can then safely clear the file as "w" would have, ie. :
fseek(fileHandle, 0, SEEK_SET);//move to the start
ftruncate(fileno((FILE *) fileHandle), 0);//clear it out
This was an unpleasant lesson for me.
As you're only coding a daemon which uses it for mutual exclusion, they are equivalent, after all, your application only needs to be compatible with itself.
The trick with the file locking mechanisms is to be consistent - use one and stick to it. Varying them is a bad idea.
I am assuming here that the filesystem will be a local one - if it isn't, then all bets are off, NFS / other network filesystems handle locking with varying degrees of effectiveness (in some cases none)