I thought fsync() does fflush() internally, so using fsync() on a stream is OK. But I am getting an unexpected result when executed under network I/O.
My code snippet:
FILE* fp = fopen(file, "wb");
/* multiple fputs() calls like: */
fputs(buf, fp);
...
...
fputs(buf.c_str(), fp);
/* get fd of the FILE pointer */
fd = fileno(fp);
#ifndef WIN32
ret = fsync(fd);
#else
ret = _commit(fd);
fclose(fp);
But it seems _commit() is not flushing the data (I tried on Windows and the data was written on a Linux exported filesystem).
When I changed the code to be:
FILE* fp = fopen(file, "wb");
/* multiple fputs() calls like: */
fputs(buf, fp);
...
...
fputs(buf.c_str(), fp);
/* fflush the data */
fflush(fp);
fclose(fp);
it flushes the data.
I am wondering if _commit() does the same thing as fflush(). Any inputs?
fflush() works on FILE*, it just flushes the internal buffers in the FILE* of your application out to the OS.
fsync works on a lower level, it tells the OS to flush its buffers to the physical media.
OSs heavily cache data you write to a file. If the OS enforced every write to hit the drive, things would be very slow. fsync (among other things) allows you to control when the data should hit the drive.
Furthermore, fsync/commit works on a file descriptor. It has no knowledge of a FILE* and can't flush its buffers. FILE* lives in your application, file descriptors live in the OS kernel, typically.
The standard C function fflush() and the POSIX system call fsync() are conceptually somewhat similar. fflush() operates on C file streams (FILE objects), and is therefore portable.
fsync() operate on POSIX file descriptors.
Both cause buffered data to be sent to a destination.
On a POSIX system, each C file stream has an associated file descriptor, and all the operations on a C file stream will be implemented by delegating, when necessary, to POSIX system calls that operate on the file descriptor.
One might think that a call to fflush on a POSIX system would cause a write of any data in the buffer of the file stream, followed by a call of fsync() for the file descriptor of that file stream. So on a POSIX system there would be no need to follow a call to fflush with a call to fsync(fileno(fp)). But is that the case: is there a call to fsync from fflush?
No, calling fflush on a POSIX system does not imply that fsync will be called.
The C standard for fflush says (emphasis added) it
causes any unwritten data for [the] stream to be delivered to the host environment to be written to the file
Saying that the data is to be written, rather than that is is written implies that further buffering by the host environment is permitted. That buffering by the "host environment" could include, for a POSIX environment, the internal buffering that fsync flushes. So a close reading of the C standard suggests that the standard does not require the POSIX implementation to call fsync.
The POSIX standard description of fflush does not declare, as an extension of the C semantics, that fsync is called.
fflush() and fsync() can be used to try and ensure data is written to the storage media (but it is not always be possible):
first use fflush(fp) on the output stream (fp being a FILE * obtained from fopen or one of the standard streams stdout or stderr) to write the contents of the buffer associated with the stream to the OS.
then use fsync(fileno(fp)) to tell the OS to write its own buffers to the storage media.
Note however that fileno() and fsync() are POSIX functions that might not be available on all systems, notably Microsoft legacy systems where alternatives may be named _fileno(), _fsync() or _commit()...
I could say that for simplicity:
use fsync() with not streaming files (integer file descriptors)
use fflush() with file streams.
Also here is the help from man:
int fflush(FILE *stream); // flush a stream, FILE* type
int fsync(int fd); // synchronize a file's in-core state with storage device
// int type
To force the commitment of recent changes to disk, use the sync() or fsync() functions.
fsync() will synchronize all of the given file's data and metadata with the permanent storage device. It should be called just before the corresponding file has been closed.
sync() will commit all modified files to disk.
I think below document from python (https://docs.python.org/2/library/os.html) clarifies it very well.
os.fsync(fd) Force write of file with filedescriptor fd to disk. On
Unix, this calls the native fsync() function; on Windows, the MS
_commit() function.
If you’re starting with a Python file object f, first do f.flush(),
and then do os.fsync(f.fileno()), to ensure that all internal buffers
associated with f are written to disk.
Availability: Unix, and Windows starting in 2.2.3.
Related
I have multiple threads attempting to log to the same file.
Each thread has a FILE * that points to the file. The FILE *s were opened in append ('a') mode and are using line buffering.
Opening multiple FILE * to the same file within the same process is implementation defined according to ANSI C.
Would anyone happen to know the implementation specific behaviour for MacOS, FreeBSD and Linux, specifically whether each FILE * will have its own line buffer, and whether there's any chance of lost or interleaved writes.
MacOS, FreeBSD and Linux are all POSIX systems. As such each FILE* will have its own user-space buffer (or none if you disable it), and once that buffer is flushed it will be written to the underlying file descriptor. POSIX guarantees that append opened file descriptor writes are atomic, thus no data will be lost. As long as your data isn't split across multiple flushes it won't interleave with each other either.
I have a pendrive at /dev/sdc.
I want to write a simple file-system to it. How would I open this as file? It is possible to open with fopen?
I've tried to clear it but no bytes are returned.
while (fgetc(device) != EOF) {
fputc(0, device);
}
Although it is possible that you could open the device file with fopen() and use C stdio functions to manipulate it, it would probably be more appropriate for your purpose to open it with POSIX open() (thus obtaining a file descriptor rather than a FILE *) and to thereafter operate on it with the POSIX file-descriptor-based I/O functions, such as write() and read(). These provide an inherently binary interface to the file, without any library-level buffering other than what you provide yourself, and on Unix-like operating systems they are more conventional than C stdio functions for low-level operations such as you are proposing.
Is there any reason (other than syntactic ones) that you'd want to use
FILE *fdopen(int fd, const char *mode);
or
FILE *fopen(const char *path, const char *mode);
instead of
int open(const char *pathname, int flags, mode_t mode);
when using C in a Linux environment?
First, there is no particularly good reason to use fdopen if fopen is an option and open is the other possible choice. You shouldn't have used open to open the file in the first place if you want a FILE *. So including fdopen in that list is incorrect and confusing because it isn't very much like the others. I will now proceed to ignore it because the important distinction here is between a C standard FILE * and an OS-specific file descriptor.
There are four main reasons to use fopen instead of open.
fopen provides you with buffering IO that may turn out to be a lot faster than what you're doing with open.
fopen does line ending translation if the file is not opened in binary mode, which can be very helpful if your program is ever ported to a non-Unix environment (though the world appears to be converging on LF-only (except IETF text-based networking protocols like SMTP and HTTP and such)).
A FILE * gives you the ability to use fscanf and other stdio functions.
Your code may someday need to be ported to some other platform that only supports ANSI C and does not support the open function.
In my opinion the line ending translation more often gets in your way than helps you, and the parsing of fscanf is so weak that you inevitably end up tossing it out in favor of something more useful.
And most platforms that support C have an open function.
That leaves the buffering question. In places where you are mainly reading or writing a file sequentially, the buffering support is really helpful and a big speed improvement. But it can lead to some interesting problems in which data does not end up in the file when you expect it to be there. You have to remember to fclose or fflush at the appropriate times.
If you're doing seeks (aka fsetpos or fseek the second of which is slightly trickier to use in a standards compliant way), the usefulness of buffering quickly goes down.
Of course, my bias is that I tend to work with sockets a whole lot, and there the fact that you really want to be doing non-blocking IO (which FILE * totally fails to support in any reasonable way) with no buffering at all and often have complex parsing requirements really color my perceptions.
open() is a low-level os call. fdopen() converts an os-level file descriptor to the higher-level FILE-abstraction of the C language. fopen() calls open() in the background and gives you a FILE-pointer directly.
There are several advantages to using FILE-objects rather raw file descriptors, which includes greater ease of usage but also other technical advantages such as built-in buffering. Especially the buffering generally results in a sizeable performance advantage.
fopen vs open in C
1) fopen is a library function while open is a system call.
2) fopen provides buffered IO which is faster compare to open which is non buffered.
3) fopen is portable while open not portable (open is environment specific).
4) fopen returns a pointer to a FILE structure(FILE *); open returns an integer that identifies the file.
5) A FILE * gives you the ability to use fscanf and other stdio functions.
Unless you're part of the 0.1% of applications where using open is an actual performance benefit, there really is no good reason not to use fopen. As far as fdopen is concerned, if you aren't playing with file descriptors, you don't need that call.
Stick with fopen and its family of methods (fwrite, fread, fprintf, et al) and you'll be very satisfied. Just as importantly, other programmers will be satisfied with your code.
If you have a FILE *, you can use functions like fscanf, fprintf and fgets etc. If you have just the file descriptor, you have limited (but likely faster) input and output routines read, write etc.
open() is a system call and specific to Unix-based systems and it returns a file descriptor. You can write to a file descriptor using write() which is another system call.
fopen() is an ANSI C function call which returns a file pointer and it is portable to other OSes. We can write to a file pointer using fprintf.
In Unix:
You can get a file pointer from the file descriptor using:
fP = fdopen(fD, "a");
You can get a file descriptor from the file pointer using:
fD = fileno (fP);
Using open, read, write means you have to worry about signal interaptions.
If the call was interrupted by a signal handler the functions will return -1
and set errno to EINTR.
So the proper way to close a file would be
while (retval = close(fd), retval == -1 && ernno == EINTR) ;
I changed to open() from fopen() for my application, because fopen was causing double reads every time I ran fopen fgetc . Double reads were disruptive of what I was trying to accomplish. open() just seems to do what you ask of it.
open() will be called at the end of each of the fopen() family functions. open() is a system call and fopen() are provided by libraries as a wrapper functions for user easy of use
Depends also on what flags are required to open. With respect to usage for writing and reading (and portability) f* should be used, as argued above.
But if basically want to specify more than standard flags (like rw and append flags), you will have to use a platform specific API (like POSIX open) or a library that abstracts these details. The C-standard does not have any such flags.
For example you might want to open a file, only if it exits. If you don't specify the create flag the file must exist. If you add exclusive to create, it will only create the file if it does not exist. There are many more.
For example on Linux systems there is a LED interface exposed through sysfs. It exposes the brightness of the led through a file. Writing or reading a number as a string ranging from 0-255. Of course you don't want to create that file and only write to it if it exists. The cool thing now: Use fdopen to read/write this file using the standard calls.
opening a file using fopen
before we can read(or write) information from (to) a file on a disk we must open the file. to open the file we have called the function fopen.
1.firstly it searches on the disk the file to be opened.
2.then it loads the file from the disk into a place in memory called buffer.
3.it sets up a character pointer that points to the first character of the buffer.
this the way of behaviour of fopen function
there are some causes while buffering process,it may timedout. so while comparing fopen(high level i/o) to open (low level i/o) system call , and it is a faster more appropriate than fopen.
I'm doing a small project in C after quite a long time away from it. These happen to include some file handling. I noticed in various documentation that there are functions which return FILE * handles and others which return (small integer) descriptors. Both sets of functions offer the same basic services I need so it really does not matter I use.
But I'm curious about the collection wisdom: is it better to use fopen() and friends, or open() and friends?
Edit Since someone mentioned buffered vs unbuffered and accessing devices, I should add that one part of this small project will be writing a userspace filesystem driver under FUSE. So the file level access could as easily be on a device (e.g. a CDROM or a SCSI drive) as on a "file" (i.e. an image).
It is better to use open() if you are sticking to unix-like systems and you might like to:
Have more fine-grained control over unix permission bits on file creation.
Use the lower-level functions such as read/write/mmap as opposed to the C buffered stream I/O functions.
Use file descriptor (fd) based IO scheduling (poll, select, etc.) You can of course obtain an fd from a FILE * using fileno(), but care must be taken not to mix FILE * based stream functions with fd based functions.
Open any special device (not a regular file)
It is better to use fopen/fread/fwrite for maximum portability, as these are standard C functions, the functions I've mentioned above aren't.
The objection that "fopen" is portable and "open" isn't is bogus.
fopen is part of libc, open is a POSIX system call.
Each is as portable as the place they come from.
i/o to fopen'ed files is (you must assume it may be, and for practical purposes, it is) buffered by libc, file descriptors open()'ed are not buffered by libc (they may well be, and usually are buffered in the filesystem -- but not everything you open() is a file on a filesystem.
What's the point of fopen'ing, for example, a device node like /dev/sg0, say, or /dev/tty0... What are you going to do? You're going to do an ioctl on a FILE *? Good luck with that.
Maybe you want to open with some flags like O_DIRECT -- makes no sense with fopen().
fopen works at a higher level than open ....
fopen returns you a pointer to FILE stream which is similar to the stream abstraction that you read in C++
open returns you a file descriptor for the file opened ... It does not provide you a stream abstraction and you are responsible for handling the bits and bytes yourself ... This is at a lower level as compared to fopen
Stdio streams are buffered, while open() file descriptors are not. Depends on what you need. You can also create one from the other:
int fileno (FILE * stream) returns the file descriptor for a FILE *, FILE * fdopen(int fildes, const char * mode) creates a FILE * from a file descriptor.
Be careful when intermixing buffered and non-buffered IO, since you'll lose what's in your buffer when you don't flush it with fflush().
Yes. When you need a low-level handle.
On UNIX operating systems, you can generally exchange file handles and sockets.
Also, low-level handles make for better ABI compatibility than FILE pointers.
read() & write() use unbuffered I/O. (fd: integer file descriptor)
fread() & fwrite() use buffered I/O. (FILE* structure pointer)
Binary data written to a pipe with write() may not be able to read binary data with fread(), because of byte alignments, variable sizes, etc. Its a crap-shoot.
Most low-level device driver code uses unbuffered I/O calls.
Most application level I/O uses buffered.
Use of the FILE* and its associated functions
is OK on a machine-by-machine basis: but portability is lost
on other architectures in the reading and writing of binary data.
fwrite() is buffered I/O and can lead to unreliable results if
written for a 64 bit architecture and run on a 32bit; or (Windows/Linux).
Most OSs have compatibility macros within their own code to prevent this.
For low-level binary I/O portability read() and write() guarantee
the same binary reads and writes when compiled on differing architectures.
The basic thing is to pick one way or the other and be consistent about it,
throughout the binary suite.
<stdio.h> // mostly FILE* some fd input/output parameters for compatibility
// gives you a lot of helper functions -->
List of Functions
Function Description
───────────────────────────────────────────────────────────────────
clearerr check and reset stream status
fclose close a stream
fdopen stream open functions //( fd argument, returns FILE*) feof check and reset stream status
ferror check and reset stream status
fflush flush a stream
fgetc get next character or word from input stream
fgetpos reposition a stream
fgets get a line from a stream
fileno get file descriptor // (FILE* argument, returns fd)
fopen stream open functions
fprintf formatted output conversion
fpurge flush a stream
fputc output a character or word to a stream
fputs output a line to a stream
fread binary stream input/output
freopen stream open functions
fscanf input format conversion
fseek reposition a stream
fsetpos reposition a stream
ftell reposition a stream
fwrite binary stream input/output
getc get next character or word from input stream
getchar get next character or word from input stream
gets get a line from a stream
getw get next character or word from input stream
mktemp make temporary filename (unique)
perror system error messages
printf formatted output conversion
putc output a character or word to a stream
putchar output a character or word to a stream
puts output a line to a stream
putw output a character or word to a stream
remove remove directory entry
rewind reposition a stream
scanf input format conversion
setbuf stream buffering operations
setbuffer stream buffering operations
setlinebuf stream buffering operations
setvbuf stream buffering operations
sprintf formatted output conversion
sscanf input format conversion
strerror system error messages
sys_errlist system error messages
sys_nerr system error messages
tempnam temporary file routines
tmpfile temporary file routines
tmpnam temporary file routines
ungetc un-get character from input stream
vfprintf formatted output conversion
vfscanf input format conversion
vprintf formatted output conversion
vscanf input format conversion
vsprintf formatted output conversion
vsscanf input format conversion
So for basic use I would personally use the above without mixing idioms too much.
By contrast,
<unistd.h> write()
lseek()
close()
pipe()
<sys/types.h>
<sys/stat.h>
<fcntl.h> open()
creat()
fcntl()
all use file descriptors.
These provide fine-grained control over reading and writing bytes
(recommended for special devices and fifos (pipes) ).
So again, use what you need, but keep consistent in your idioms and interfaces.
If most of your code base uses one mode , use that too, unless there is
a real reason not to. Both sets of I/O library functions are extremely reliable
and used millions of times a day.
note-- If you are interfacing C I/O with another language,
(perl, python, java, c#, lua ...) check out what the developers of those languages
recommend before you write your C code and save yourself some trouble.
usually, you should favor using the standard library (fopen). However, there are occasions where you will need to use open directly.
One example that comes to mind is to work around a bug in an older version of solaris which made fopen fail after 256 files were open. This was because they erroniously used an unsigned char for the fd field in their struct FILE implementation instead of an int. But this was a very specific case.
fopen and its cousins are buffered. open, read, and write are not buffered. Your application may or may not care.
fprintf and scanf have a richer API that allows you to read and write formatted text files. read and write use fundamental arrays of bytes. Conversions and formatting must be hand crafted.
The difference between file descriptors and (FILE *) is really inconsequential.
Randy
It turns out this whole misunderstanding of the open() versus fopen() stems from a buggy I2C driver in the Linux 2.6.14 kernel on an ARM. Backporting a working bit bashed driver solved the root cause of the problem I was trying to address here.
I'm trying to figure out an issue with a serial device driver in Linux (I2C). It appears that by adding timed OS pauses (sleeps) between writes and reads on the device things work ... (much) better.
Aside: The nature of I2C is that each byte read or written by the master is acknowledged by the device on the other end of the wire (slave) - the pauses improving things encourage me to think of the driver as working asynchronously - something that I can't reconcile with how the bus works. Anyhoo ...
I'd either like to flush the write to be sure (rather than using fixed duration pause), or somehow test that the write/read transaction has completed in an multi-threaded friendly way.
The trouble with using fflush(fd); is that it requires 'fd' to be stream pointer (not a file descriptor) i.e.
FILE * fd = fopen("filename","r+");
... // do read and writes
fflush(fd);
My problem is that I require the use of the ioctl(), which doesn't use a stream pointer. i.e.
int fd = open("filename",O_RDWR);
ioctl(fd,...);
Suggestions?
I think what you are looking for may be
int fsync(int fd);
or
int fdatasync(int fd);
fsync will flush the file from kernel buffer to the disk. fdatasync will also do except for the meta data.
You have two choices:
Use fileno() to obtain the file descriptor associated with the stdio stream pointer
Don't use <stdio.h> at all, that way you don't need to worry about flush either - all writes will go to the device immediately, and for character devices the write() call won't even return until the lower-level IO has completed (in theory).
For device-level IO I'd say it's pretty unusual to use stdio. I'd strongly recommend using the lower-level open(), read() and write() functions instead (based on your later reply):
int fd = open("/dev/i2c", O_RDWR);
ioctl(fd, IOCTL_COMMAND, args);
write(fd, buf, length);
fflush() only flushes the buffering added by the stdio fopen() layer, as managed by the FILE * object. The underlying file itself, as seen by the kernel, is not buffered at this level. This means that writes that bypass the FILE * layer, using fileno() and a raw write(), are also not buffered in a way that fflush() would flush.
As others have pointed out, try not mixing the two. If you need to use "raw" I/O functions such as ioctl(), then open() the file yourself directly, without using fopen<() and friends from stdio.
Have you tried disabling buffering?
setvbuf(fd, NULL, _IONBF, 0);
It sounds like what you are looking for is the fsync() function (or fdatasync()?), or you could use the O_SYNC flag in your open() call.
If you want to go the other way round (associate FILE* with existing file descriptor), use fdopen() :
FDOPEN(P)
NAME
fdopen - associate a stream with a file descriptor
SYNOPSIS
#include <stdio.h>
FILE *fdopen(int fildes, const char *mode);