I would like to write a simple API which
allows the user to open a file.
let the user write data to the file
track the write calls and sanity check the written data after each write call.
prevents the data from beeing written to disk if it is not valid -> discard(file)
As a starting point i wrote the test program below, which opens a file in fully buffered "rb+" mode using fopen and setvbuf.
The stream is opened in fully buffered mode for the following reason:
http://www.cplusplus.com/reference/cstdio/setvbuf/
mode
Specifies a mode for file buffering.
Three special macro constants [...]:
_IOFBF Full buffering: On output, data is written once the buffer is full (or flushed). On Input, the buffer is filled when an input
operation is requested and the buffer is empty.
My testprogram contains comments where a validity check could be placed and where the buffer contents should be discarded.
My question is how do i accomplish the discard(file) operation which means the step of getting rid of invalid buffer contents ?
The idea behind this is to assemble some data in the buffer, do a regular validity check after each or several write operations and write the data to disk only, if the data is valid.
Therefore i would need to discard the buffer, if the validity check fails.
When the validity check passes, the whole buffer contents should be written to the file.
My code draft looks like in the following. This is a simplified example:
#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
int main(void)
{
static uint8_t buffer[10000];
/* The following would be part of mylib_init */
FILE *file = fopen("test", "wb+");
if (file == NULL){
print ("open error!");
exit(-1);
}
if ( 0 != setvbuf(file , buffer, _IOFBF , sizeof(buffer) ) ){
print("Could not set buffer!");
fclose(file);
exit (-2);
}
/* The following would be part of mylib_write_data.
Each write and check resembles one func call */
// Pretend the user writes some data into the file
// ...
// fwrite(x)
if (data_in_buffer_not_valid(buffer)){
discard(file);
}
// ...
// fwrite(y)
//
if (data_in_buffer_not_valid(buffer)){
discard(file);
}
// ...
// fwrite(z)
// ...
// The following would be part of mylib_exit
// Cleanup stuff
fclose(file)
return 0;
}
If you want to have some like "scratch" temporary file that you want to write your data into and then retrieve them later, then the portable interface would be tmpfile() - it's an interface created just for that. Write to that file, rewind if you want, and when you're ready, rewind it and read from it block by block to another file.
On linux you may use fmemopen and fopencookie to write to a buffer via FILE* - these functions are not available on windows.
I would also strongly consider just creating your own interface that would store the result in memory. Writing an interface like struct mystream; mystream_init(struct mystream *); mystream_printf(struct mystream *, const char *fmt, ...); etc. is some of the tasks you sometimes do in C when fopencookie is not available. And consider writing the interface for storing data, so that instead of calling fwrite you would actually call the function that would check the data and write them and process them along the way.
As for setvbuf, note the standard. From C11 7.21.3p3:
When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block. When a stream is fully buffered, [...]. When a stream is line buffered, [...] Support for these characteristics is implementation-defined, and may be affected via the setbuf and setvbuf functions.
And these buffering modes may just be not supported at all. And from C11 7.21.5.6:
The setvbuf function may be used only after the stream pointed to by stream has been associated with an open file and before any other operation (other than an unsuccessful call to setvbuf) is performed on the stream. [...] The contents of the array at any time are indeterminate.
You can't count on anything what will be the content of the buffer. Do not expecting any data there.
In the book Advanced Programming in the UNIX Environments (2nd edition), the author wrote in Section 5.5 (stream operations of the standard I/O library) that:
When a file is opened for reading and writing (the plus sign in the type), the following restrictions apply.
Output cannot be directly followed by input without an intervening fflush, fseek, fsetpos, or rewind.
Input cannot be directly followed by output without an intervening fseek, fsetpos, or rewind, or an input operation that encounters an end of file.
I got confused about this. Could anyone explain a little about this? For example, in what situation the input and output function calls violating the above restrictions will cause unexpected behavior of the program? I guess the reason for the restrictions may be related to the buffering in the library, but I'm not so clear.
You aren't allowed to intersperse input and output operations. For example, you can't use formatted input to seek to a particular point in the file, then start writing bytes starting at that point. This allows the implementation to assume that at any time, the sole I/O buffer will only contain either data to be read (to you) or written (to the OS), without doing any safety checks.
f = fopen( "myfile", "rw" ); /* open for read and write */
fscanf( f, "hello, world\n" ); /* scan past file header */
fprintf( f, "daturghhhf\n" ); /* write some data - illegal */
This is OK, though, if you do an fseek( f, 0, SEEK_CUR ); between the fscanf and the fprintf because that changes the mode of the I/O buffer without repositioning it.
Why is it done this way? As far as I can tell, because OS vendors often want to support automatic mode switching, but fail. The stdio spec allows a buggy implementation to be compliant, and a working implementation of automatic mode switching simply implements a compatible extension.
It's not clear what you're asking.
Your basic question is "Why does the book say I can't do this?" Well, the book says you can't do it because the POSIX/SUS/etc. standard says it's undefined behavior in the fopen specification, which it does to align with the ISO C standard (N1124 working draft, because the final version is not free), 7.19.5.3.
Then you ask, "in what situation the input and output function calls violating the above restrictions will cause unexpected behavior of the program?"
Undefined behavior will always cause unexpected behavior, because the whole point is that you're not allowed to expect anything. (See 3.4.3 and 4 in the C standard linked above.)
But on top of that, it's not even clear what they could have specified that would make any sense. Look at this:
int main(int argc, char *argv[]) {
FILE *fp = fopen("foo", "r+");
fseek(fp, 0, SEEK_SET);
fwrite("foo", 1, 3, fp);
fseek(fp, 0, SEEK_SET);
fwrite("bar", 1, 3, fp);
char buf[4] = { 0 };
size_t ret = fread(buf, 1, 3, fp);
printf("%d %s\n", (int)ret, buf);
}
So, should this print out 3 foo because that's what's on disk, or 3 bar because that's what's in the "conceptual file", or 0 because there's nothing after what's been written so you're reading at EOF? And if you think there's an obvious answer, consider the fact that it's possible that bar has been flushed already—or even that it's been partially flushed, so the disk file now contains boo.
If you're asking the more practical question "Can I get away with it in some circumstances?", well, I believe on most Unix platforms, the above code will give you an occasional segfault, but 3 xyz (either 3 uninitialized characters, or in more complicated cases 3 characters that happened to be in the buffer before it got overwritten) the rest of the time. So, no, you can't get away with it.
Finally, you say, "I guess the reason for the restrictions may be related to the buffering in the library, but I'm not so clear." This sounds like you're asking about the rationale.
You're right that it's about buffering. As I pointed out above, there really is no intuitive right thing to do here—but also, think about the implementation. Remember that the Unix way has always been "if the simplest and most efficient code is good enough, do that".
There are three ways you could implement something like stdio:
Use a shared buffer for read and write, and write code to switch contexts as needed. This is going to be a bit complicated, and will flush buffers more often than you'd ideally like.
Use two separate buffers, and cache-style code to determine when one operation needs to copy from and/or invalidate the other buffer. This is even more complicated, and makes a FILE object take twice as much memory.
Use a shared buffer, and just don't allow interleaving reads and writes without explicit flushes in between. This is dead-simple, and as efficient as possible.
Use a shared buffer, and implicitly flush between interleaved reads and writes. This is almost as simple, and almost as efficient, and a lot safer, but not really any better in any way other than safety.
So, Unix went with #3, and documented it, and SUS, POSIX, C89, etc. standardized that behavior.
You might say, "Come on, it can't be that inefficient." Well, you have to remember that Unix was designed for low-end 1970s systems, and the basic philosophy that it's not worth trading off even a little efficiency unless there's some actual benefit. But, most importantly, consider that stdio has to handle trivial functions like getc and putc, not just fancy stuff like fscanf and fprintf, and adding anything to those functions (or macros) that makes them 5x as slow would make a huge difference in a lot of real-world code.
If you look at modern implementations from, e.g., *BSD, glibc, Darwin, MSVCRT, etc. (most of which are open source, or at least commercial-but-shared-source), most of them do things the same way. A few add safety checks, but they generally give you an error for interleaving rather than implicitly flushing—after all, if your code is wrong, it's better to tell you that your code is wrong than to try to DWIM.
For example, look at early Darwin (OS X) fopen, fread, and fwrite (chosen because it's nice and simple, and has easily-linkable code that's syntax-colored but also copy-pastable). All that fread has to do is copy bytes out of the buffer, and refill the buffer if it runs out. You can't get any simpler than that.
reason 1
find the real file position to start.
due to the buffer implementation of the stdio, the stdio stream position may differ from the OS file position. when you read 1 byte, stdio mark the file position to 1. Due to the buffering, stdio may read 4096 bytes from the underlying file, where OS would record its file position at 4096. When you switch to output, you really need to choose which position you want to use.
reason 2
find the right buffer cursor to start.
tl;dr,
if an underlying implementation only uses a single shared buffer for both read and write, you have to flush the buffer when changing IO direction.
Take this glibc used in chromium os to demo how fwrite, fseek, and fflush handle the single shared buffer.
fwrite fill buffer impl:
fill_buffer:
while (to_write > 0)
{
register size_t n = to_write;
if (n > buffer_space)
n = buffer_space;
buffer_space -= n;
written += n;
to_write -= n;
if (n < 20)
while (n-- > 0)
*stream->__bufp++ = *p++;
else
{
memcpy ((void *) stream->__bufp, (void *) p, n);
stream->__bufp += n;
p += n;
}
if (to_write == 0)
/* Done writing. */
break;
else if (buffer_space == 0)
{
/* We have filled the buffer, so flush it. */
if (fflush (stream) == EOF)
break;
from this code snippet, we can see, if buffer is full, it will flush it.
Let's take a look at fflush
int
fflush (stream)
register FILE *stream;
{
if (stream == NULL) {...}
if (!__validfp (stream) || !stream->__mode.__write)
{
__set_errno (EINVAL);
return EOF;
}
return __flshfp (stream, EOF);
}
it uses __flshfp
/* Flush the buffer for FP and also write C if FLUSH_ONLY is nonzero.
This is the function used by putc and fflush. */
int
__flshfp (fp, c)
register FILE *fp;
int c;
{
/* Make room in the buffer. */
(*fp->__room_funcs.__output) (fp, flush_only ? EOF : (unsigned char) c);
}
the __room_funcs.__output by default is using flushbuf
/* Write out the buffered data. */
wrote = (*fp->__io_funcs.__write) (fp->__cookie, fp->__buffer,
to_write);
Now we are close. What's __write? Trace the default settings aforementioned, it's __stdio_write
int
__stdio_write (cookie, buf, n)
void *cookie;
register const char *buf;
register size_t n;
{
const int fd = (int) cookie;
register size_t written = 0;
while (n > 0)
{
int count = __write (fd, buf, (int) n);
if (count > 0)
{
buf += count;
written += count;
n -= count;
}
else if (count < 0
#if defined (EINTR) && defined (EINTR_REPEAT)
&& errno != EINTR
#endif
)
/* Write error. */
return -1;
}
return (int) written;
}
__write is the system call to write(3).
As we can see, the fwrite is only using only one single buffer. If you change direction, it can still store the previous write contents. From the above example, you can call fflush to empty the buffer.
The same applies to fseek
/* Move the file position of STREAM to OFFSET
bytes from the beginning of the file if WHENCE
is SEEK_SET, the end of the file is it is SEEK_END,
or the current position if it is SEEK_CUR. */
int
fseek (stream, offset, whence)
register FILE *stream;
long int offset;
int whence;
{
...
if (stream->__mode.__write && __flshfp (stream, EOF) == EOF)
return EOF;
...
/* O is now an absolute position, the new target. */
stream->__target = o;
/* Set bufp and both end pointers to the beginning of the buffer.
The next i/o will force a call to the input/output room function. */
stream->__bufp
= stream->__get_limit = stream->__put_limit = stream->__buffer;
...
}
it will soft flush (reset) the buffer at the end, which means read buffer will be emptied after this call.
This obeys the C99 rationale:
A change of input/output direction on an update file is only allowed following a successful fsetpos, fseek, rewind, or fflush operation, since these are precisely the functions which assure that the I/O buffer has been flushed.
Is it alright for multiple processes to access (write) to the same file at the same time? Using the following code, it seems to work, but I have my doubts.
Use case in the instance is an executable that gets called every time an email is received and logs it's output to a central file.
if (freopen(console_logfile, "a+", stdout) == NULL || freopen(error_logfile, "a+", stderr) == NULL) {
perror("freopen");
}
printf("Hello World!");
This is running on CentOS and compiled as C.
Using the C standard IO facility introduces a new layer of complexity; the file is modified solely via write(2)-family of system calls (or memory mappings, but that's not used in this case) -- the C standard IO wrappers may postpone writing to the file for a while and may not submit complete requests in one system call.
The write(2) call itself should behave well:
[...] If the file was
open(2)ed with O_APPEND, the file offset is first set to the
end of the file before writing. The adjustment of the file
offset and the write operation are performed as an atomic
step.
POSIX requires that a read(2) which can be proved to occur
after a write() has returned returns the new data. Note that
not all file systems are POSIX conforming.
Thus your underlying write(2) calls will behave properly.
For the higher-level C standard IO streams, you'll also need to take care of the buffering. The setvbuf(3) function can be used to request unbuffered output, line-buffered output, or block-buffered output. The default behavior changes from stream to stream -- if standard output and standard error are writing to the terminal, then they are line-buffered and unbuffered by default. Otherwise, block-buffering is the default.
You might wish to manually select line-buffered if your data is naturally line-oriented, to prevent interleaved data. If your data is not line-oriented, you might wish to use un-buffered or leave it block-buffered but manually flush the data whenever you've accumulated a single "unit" of output.
If you are writing more than BUFSIZ bytes at a time, your writes might become interleaved. The setvbuf(3) function can help prevent the interleaving.
It might be premature to talk about performance, but line-buffering is going to be slower than block buffering. If you're logging near the speed of the disk, you might wish to take another approach entirely to ensure your writes aren't interleaved.
This answer was incorrect. It does work:
So the race condition would be:
process 1 opens it for append, then
later process 2 opens it for append, then
later still 1 writes and closes, then
finally 2 writes and closes.
I'd be impressed if that 'worked' because it isn't clear to me what
working should mean. I assume 'working' means all of the bytes written
by the two processes are inthe log file? I'd expect that they both
write starting at the same byte offset, so one will replace the others
bytes. It will all be okay upto and including step 3. and only show up
as a problem at step 4, Seems like an easy test to write: open getchar
... write close.
Is it critical that they can have the file open simultaneously? A
more obvious solution if the write is quick, is to open exclusive.
For a quick check on your system, try:
/* write the first command line argument to a file called foo
* stackoverflow topic 9880935
*/
#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
int main (int argc, const char * argv[]) {
if (argc <2) {
fprintf(stderr, "Error: need some text to write to the file Foo\n");
exit(1);
}
FILE* fp = freopen("foo", "a+", stdout);
if (fp == NULL) {
perror("Error failed to open file\n");
exit(1);
}
fprintf(stderr, "Press a key to continue\n");
(void) getchar(); /* Yes, I really mean to ignore the character */
if (printf("%s\n", argv[1]) < 0) {
perror("Error failed to write to file: ");
exit(1);
}
fclose(fp);
return 0;
}
I've searched high and low, but can not find the answer to what I would've thought to be a rather simple question. I'm rather new to C, and due to the restrictions placed on me during this project I'm having a bit of trouble figuring out how to do this.
I am trying to read in text from a text file, and store that data in an array. Simple enough. HOWEVER, I'm forced to do so by using the command line operator '<' and to redirect the stdin to the text file.
The only way I can seem to figure out how to properly open a file and perform the subsequent operations is the following:
#include <stdio.h>
FILE *fr;
main()
{
fr = fopen ("mytext.txt", "r"); /* open the file for reading */
The problem with that is that I can't seem to get the first parameter of fopen() to be the filename provided by the stdin '<'. It only works if I explicitly type a string in for the parameter. For example, if I were to run
myprog.o < mytxt.txt
how could I pass the name of the text file provided by that stdin redirection to the fopen function? Or is there a different/better way to do what I'm trying to do?
Thanks in advance.
You need to read from stdin instead of trying to open the file directly.
This is because of how redirection works - think of it a bit like this:
The file is opened (for purposes of demonstration, let's say fopen is used for this).
The existing stdin is closed. It no longer refers to a terminal or similar construct.
stdin is replaced with the open file in step 1.
Any reads from stdin now work directly from the input file.
By using input redirection you can permit your user to either redirect a file or directly type content into your program.
As for your actual problem, you might be better off passing the filename as an argument to your program. You would use argv and call your program like so:
myprog.o mytxt.txt
In this case, argv[1] will be mytxt.txt.
A C program never sees the redirection because it is handled by the shell. The C program should look at argc and read from stdin if no args are given or from the given files otherwise.
There is a standard FILE* handle declared within <stdio.h> that points to the standard input for the executing process. Such file handle is called stdin.
If all you ever want this program to do is read from standard input, then you don't need to open any files. The OS and C libraries will handle opening and closing the file for you.
So to read a file in from standard input, and write it back out, try something as simple as
#include <stdio.h>
int main( int argc, char ** argv ) {
int ch = getchar();
while ( ch != EOF ) {
putchar( ch );
ch = getchar();
}
}
As you can see, no opening or closing of files. putchar and getchar write to stdin and stdout relatively.
If you want to be more explicit, you can use the predefined file handles.
int ch = fgetc( stdin );
while ( ch != EOF ) {
fputc( ch, stdout );
ch = fgetc( stdin );
}
You should look up printf() and fprintf(), scanf() and fscanf(), and all the other wonderful stdio functions.
I'm trying to parse data from stdin in binary mode under Win32.
The first thing my code does is to check for a 4byte header at the beginning:
int riff_header;
fread(&riff_header, sizeof(riff_header), 1, ifp);
// 'RIFF' = little-endian
if (riff_header != 0x46464952) {
fprintf(stderr, "wav2msu: Incorrect header: Invalid format or endianness\n");
fprintf(stderr, " Value was: 0x%x\n", riff_header);
return -1;
}
stdin has been switched to binary mode before reading from it:
if (*argv[argc-1] == '-') {
fprintf(stderr, "Reading from stdin.\n");
infile = stdin;
// We need to switch stdin to binary mode, or else we run
// into problems under Windows
freopen(NULL, "rb", stdin);
}
This code works fine under Linux, however on Win32 (specifically Windows XP), the fread only seems to read a single byte and thus cause the evaluation to fail.
Example:
> ffmeg.exe -i ..\test.mp3 -f wav pipe:1 2> nul |..\foo.exe -o test.bin -
Reading from stdin.
foo: Incorrect header: Invalid format or endianness
Value was: 0x4
What am I doing wrong?
According to the MSDN documentation, it's not permitted to pass NULL for the path parameter of freopen, so the call to freopen is almost certainly failing; have you checked the return value and the value of errno? C89 does not specify the behavior of freopen when path is NULL; C99 does, but the Microsoft C runtime is not (and does not claim to be) C99-compliant.
If you really need to read binary info from stdin, you might have to use platform-specific code and read the raw binary data directly with ReadFile on the file GetStdHandle(STD_INPUT_HANDLE).
At http://pubs.opengroup.org/onlinepubs/009695399/functions/freopen.html I have found the following:
If filename is a null pointer, the freopen() function shall attempt to
change the mode of the stream to that specified by mode, as if the
name of the file currently associated with the stream had been used.
In this case, the file descriptor associated with the stream need not
be closed if the call to freopen() succeeds. It is
implementation-defined which changes of mode are permitted (if any),
and under what circumstances.
Maybe you should check if the change of mode (from text to binary) is allowed by the compiler and libraries you are using. Which compiler are you using?
Update / summary
Using MinGW you can call setmode() to switch the mode of the stdin stream.
You should set the mode to _O_BINARY, which is defined in fcntl.h.
For more information see e.g. http://gnuwin32.sourceforge.net/compile.html