Each time I start the program, the message "This message 2" is displayed in a random order relative to other messages.
#include <stdio.h>
int main()
{
fprintf(stdout,"This is message 1\n");
fprintf(stderr,"This is message 2\n");
fprintf(stdout,"This is message 3\n");
return 0;
}
Examples:
Example 1
Example 2
Why is this happening and how to fix it?
No randomness is displayed in the examples; both show message 2 first, and this is the result of deterministic behavior.
Per C 2018 7.21.3 7, the standard error stream is not fully buffered. This means it is either unbuffered (printed characters are sent as soon as possible) or line buffered (printed characters are sent when a new-line character is printed or sooner). Since "This is message 2\n" ends with a new-line character, both the unbuffered and line buffered modes send it as soon as possible.
Also per 7.21.3 7, the standard output stream is fully buffered if and only if it can be determined not to refer to an interactive device. Thus, we have two possibilities:
The stream can be determined to refer to an interactive device. Then it is not fully buffered, so it is unbuffered or line buffered, and the strings ending with new-line characters are sent as soon as possible.
The stream can not be determined to refer to an interactive device. Then it is fully buffered, so the strings are held in a buffer until the buffer is full or a flush is requested, which occurs at normal program termination.
Thus the total behavior of the program when the output is going directly to an interactive device is that all strings are sent immediately, so the output will be:
This is message 1
This is message 2
This is message 3
And the total behavior of the program when the output is not going to an interactive device is that message 2 is sent immediately and messages 1 and 3 are held in a buffer until the end of the program, so the output is:
This is message 2
This is message 1
This is message 3
Both cases are deterministic, not random. The output shown in the question is evidence that the IDE is running the program with the standard output captured in some way and presented to the user indirectly (as through a pipe or a redirection to a temporary file) rather than sent directly to the display device.
This can be changed by manually inserting fflush(stdout) calls where you want the standard output to be sent immediately to the associated device or file, or it can be changed by making the standard output stream line-buffered:
setvbuf(stdout, 0, _IOLBF, 0); // Must be before stdout is used.
(You could also use setvbuf to make the standard error stream fully buffered, but that is inadvisable.)
If the IDE is capturing both the standard output stream and the standard error stream in the same way, this will result in the messages appearing in the 1, 2, 3 order. However, it is possible the IDE is capturing the streams separately and displaying the standard error results first, in which case changing the buffering will not change the display.
Related
what will be the output of the following snippet and why?
#include<stdio.h>
void main(){
printf("Before while");
while(1);
printf("After while");
}
If you had just taken the effort to compile and run, you probably would have seen nothing.
That's because:
Even though you're printing Before while, you do not send a newline character at the end. Since standard output is generally line buffered for an interactive device(a), it should cache this until you send a newline, flush the stream, or exit the program.
You do none of those things because while(1); (note the semicolon) is an infinite loop that does nothing in its body, meaning it will never reach the second printf.
(a) If you're interested in this behaviour, there are basically three buffering strategies for output.
Unbuffered means output is passed straight through;
Line-buffered means output is cached until a newline is output (although it may also be output without a newline if, for example, the buffer fills up);
Fully-buffered means output is cached until the buffer is full at which point it's output and the buffer starts afresh.
The rules in the ISO C standard (C11 7.21.3 Files /7) for the three standard streams are:
At program startup, three text streams are predefined and need not be opened explicitly - standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output).
As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.
I'm just curious which conditions should be satisfied to flush stdout buffer automatically.
First of all I was confused that this pseudo code doesn't print output every iteration:
while (1) {
printf("Any text");
sleep(1);
}
But if I add newline character it will.
After few experiments i found that on my machine stdout buffer is flushed:
When I put to stdout 1025 characters or more;
When I read stdin;
When I put newline character to stdout;
The first condition is totally clear - when the buffer is full it should be flushed. The second one is also reasonable. But why newline character causes flushing? What are the others implicit conditions for this?
Rules of automatic flushing stdout buffer is implementation-defined (ID). It is ID when the stream is unbuffered, fully buffered, or line buffered.
When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block.
When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled.
When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered. Furthermore, characters are intended to be transmitted as a block to the host environment when a buffer is filled, when input is requested on an unbuffered stream, or when input is requested on a line buffered stream that requires the transmission of characters from the host environment.
Support for these characteristics is implementation-defined, ... C11dr §7.21.3 3
I'm just curious which conditions should be satisfied to flush stdout buffer automatically.
If code wants to insure output is certainly flushed, use fflush(). Other conditions that may automatically flush the stream are implementation defined.
A output stream which is line-buffered shall be flushed whenever a newline is output.
An implementation may (but is not required to) flush all line-buffered output streams whenever a read is attempted from any line-buffered input stream.
Implementations are not allowed to make streams fully-buffered by default unless it can be determined that they are not associated with an "interactive device". So when stdin/stdout are terminals they can't be fully-buffered, only line-buffered (or unbuffered).
If you only need flushing when the output is to a terminal, it suffices to assume that writing a newline results in flushing. Otherwise you should explicitly call fflush wherever you need flushing.
See the man page for setbuf(3). By default, stdout is set to line buffering mode.
printf() and its variants work with buffered output, and delegate to write(). So this buffering is controlled by the C library implementation of printf, with the buffer and buffer settings located in the FILE structure.
It's also worth noting the difference between section 3 and section 2 of the unix man pages. Section 2 is made up of function calls that directly talk to the operating system and do things that it would otherwise be impossible to do from a pure user program. Section 3 is made up of function calls that a user could reproduce on their own, which often delegate to section 2 calls. Section 2 functions contain the low-level "magic" that allow C programs to interact with the outside world and perform I/O. Section 3 functions can provide a more convenient interface to the section 2 functions.
printf, scanf, getchar, fputs, and other FILE * functions all are section 3 functions that delegate to write() and read(), which are section 2 functions. read() and write() do not buffer. printf() interacts with the buffer in the FILE structure, and occasionally decides to send the contents of that buffer out through write().
There are many circumstances when buffered output on a stream is flushed automatically:
When you try to do output and the output buffer is full.
When the stream is closed.
When the program terminates by calling exit.
When a newline is written, if the stream is line buffered.
Whenever an input operation on any stream actually reads data from its file.
stdout is line buffered by default.
If you want to flush the buffered output at another time,You can call fflush.
Online C2011 standard
7.21.3 Files
...
3 When a stream is unbuffered, characters are intended to appear from the source or at the
destination as soon as possible. Otherwise characters may be accumulated and
transmitted to or from the host environment as a block. When a stream is fully buffered,
characters are intended to be transmitted to or from the host environment as a block when
a buffer is filled. When a stream is line buffered, characters are intended to be
transmitted to or from the host environment as a block when a new-line character is
encountered. Furthermore, characters are intended to be transmitted as a block to the host
environment when a buffer is filled, when input is requested on an unbuffered stream, or
when input is requested on a line buffered stream that requires the transmission of
characters from the host environment. Support for these characteristics is
implementation-defined, and may be affected via the setbuf and setvbuf functions.
...
7 At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device.
So, a line buffered stream will flush on a newline. On most systems I have experience with, stdout is line buffered in an interactive session.
In my program in C, running on Linux, which creates sub-processes using system() I noticed that when I redirected stdout to a pipe or to a file then the output of the sub-processes was sent before the output of buffered I/O functions like printf(). When stdout was left to go to the terminal then the output was in the expected order. I simplified the program to the following example:
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
printf("1. output from printf()\n");
system("echo '2. output from a command called using system()'");
return EXIT_SUCCESS;
}
The expected output when stdout goes to the terminal:
$ ./iobuffer
1. output from printf()
2. output from a command called using system()
The output out of order when stdout is redirected to a pipe or a file:
$ ./iobuffer | cat
2. output from a command called using system()
1. output from printf()
A terminal generally uses line-buffering, while a pipe would use block buffering.
This means that your printf call, which includes a newline, will fill the line buffer, triggering a flush. When redirecting, no flush takes place until the program completes.
echo on the other hand, always flushes the buffer it is writing to when it completes.
With line buffering (terminal output), the order is:
printf() prints a line with newline, buffer is flushed, you see 1. output from printf() being printed.
echo writes output, exits, flushes the buffer, you see 2. output from a command called using system() printed.
With block buffering, the order is:
printf() prints a line with newline, not fully filling the buffer.
echo writes output, exits, flushes the buffer, you see 2. output from a command called using system() printed.
Your program exits, flushes its block buffer, you see 1. output from printf() being printed.
Your options are to use to flush explicitly using fflush() or to set the buffering on stdout explicitly with setvbuf().
This reply is complementing the reply of Martijn Pieters. Citations of sources describing the default buffering modes of streams are at the end.
Solutions
Here are three basic solutions with explanation:
a) Flush the buffers before calling a sub-process. This option could be significantly more effective when you do a lot of output from the main program and just few sub-process calls.
printf("1. output from printf()\n");
fflush(stdout);
system("echo '2. output from a command called using system()'");
b) Change the buffering of stdout to line-buffering (or unbuffered) for the whole program. This option is a small change to the program as you call sevbuf() only at the beginning and the rest of the program stays the same.
if(setvbuf(stdin, NULL, _IOLBF, BUFSIZ))
err(EXIT_FAILURE, NULL);
printf("1. output from printf()\n");
system("echo '2. output from a command called using system()'");
Edit:
c) Change the buffering of stdout to line-buffering (or unbuffered) for the whole program by an external utility. This option does not change the program at all so you do not need to recompile or even have sources of the program. You just call the program using the stdbuf utility.
$ stdbuf -oL ./iobuffer | cat
1. output from printf()
2. output from a command called using system()
References - describing why the buffering mode changes
The initial buffering setting is described for example in the documents below. Streams to interactive devices like terminal are by default line-buffered so that newline-ended messages appear immediately on the terminal. Pipes, files etc. use block-buffering (or full-buffering) for better performance.
The GNU C Library Reference Manual
http://www.gnu.org/software/libc/manual/html_node/Buffering-Concepts.html#Buffering-Concepts
Newly opened streams are normally fully buffered, with one exception:
a stream connected to an interactive device such as a terminal is
initially line buffered.
Linux man-pages: stdin (3)
http://linux.die.net/man/3/stdin
The stream stderr is unbuffered. The stream stdout is line-buffered
when it points to a terminal. Partial lines will not appear until
fflush(3) or exit(3) is called, or a newline is printed. This can
produce unexpected results, especially with debugging output. The
buffering mode of the standard streams (or any other stream) can be
changed using the setbuf(3) or setvbuf(3) call.
There are also a mention of the buffering of a terminal driver.
ISO/IEC 9899:201x C11 Committee Draft — April 12, 2011;
7.21.3 Files, page 301
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf
As initially opened, the standard error stream is not fully buffered;
the standard input and standard output streams are fully buffered if
and only if the stream can be determined not to refer to an
interactive device.
The Open Group: System Interfaces and Headers Issue 4, Version 2;
2.4 Standard I/O Streams, page 32
https://www2.opengroup.org/ogsys/catalog/C435 (free registration needed for download)
When opened, the standard
error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be
determined not to refer to an interactive device.
There is also a very interesting chapter 2.4.1 "Interaction of File Descriptors and Standard I/O Streams" about combining of buffered and unbuffered I/O which somewhat relates to sub-process calls.
I know this is because of buffering of stdout but when can i expect output of stdout in the following proggram. If i run, I am always getting "stderr" as output. If i add '\n' or fflush(stdout) then only i am getting both statements. If i don't add '\n' or fflush(stdout),i am not getting "stdout" as output. when will i get all buffered "stdout"s as output if i don't add '\n' or fflush(stdout).
#include <stdio.h>
#include <unistd.h>
int main()
{
for(;;)
{
fprintf(stdout,"stdout");
fprintf(stderr,"stderr");
sleep(1);
}
return 0;
}
Indeed, a new line or a flush because by default stdout is line buffered when it refers a terminal device.
To be more precise: standard input and standard output are fully buffered, if and only if they do not refer to an interactive device. Standard error is never fully buffered.
About line buffering, quoting from APUE:
Line buffering comes with two caveats. First, the size of the buffer that the standard I/O library is using to collect each line is fixed, so I/O might take place if we fill this buffer before writing a newline. Second, whenever input is requested through the standard I/O library from either (a) an unbuffered stream or (b) a line-buffered stream (that requiresdata to be requested from the kernel), all line-buffered output streams are flushed. The reason for the qualifier on (b) is that the requested data may alreadybe
in the buffer, which doesn't require data to be read from the kernel. Obviously, any input from an unbuffered stream, item (a), requires data to be obtained from the kernel.
To change it to unbuffered, use setvbuf:
setvbuf(stdout, NULL, _IONBF, 0);
output sent to stderr usually prints immediately. It is the standard error output. Errors, of course, should be displayed immediately, and so any buffer that exists is flushed immediately.
stdout, on the other hand, is buffered. Getting that buffer to print out varies by language, but usually at least one of two things are required: /n, or some other type of newline character, or .flush();. If you do not provide these options, then usually you have to wait for the buffer to fill. This can take any amount of time, as it pretty much entirely depends on how large the buffer is.
By convention, the way streams are handled is that you shove as much information into them as you want, and then call .flush(). If the buffer happens to fill up in the meantime, then the data is sent to wherever it's going and the buffer starts filling up again.
Here, though, you're filling up a buffer that takes in text. Text does not really take up a whole lot of space compared to binary information (this is a VERY subjective statement and should be interpreted loosely), and so filling up the buffer could take quite a while.
The best practice is for you to manually flush the stream (stdout) once you have "fed" it all of the text you want printed.
fprintf is by default line buffered. You can alter the behavior by calling setvbuf. It allows you to set it to 'unbuffered', 'line buffered', or 'fully buffered'.
So I wrote a test program and here is the code
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
int i;
printf("%s", "entering\n");
fflush(stdout);
for (i = 0 ; i < 3 ; i++)
{
fork();
fflush(stdout);
}
printf("%s", "exiting\n");
}
When I compile and run it in the terminal it displays what I expect it to: "entering" once, and "exiting" some times. When I run it and redirect the output to a file it displays entering for every exiting.
1) Why does it not output the same thing to terminal and to a file every time?
2) Why does it display entering in the file 8 times but not entering in in the terminal only once (once is what I would expect it to do).
3) Do the fflush() statements make a difference when my output goes to a file?
It has to do with the buffering on the standard file handles. From ISO C99 7.19.3/7 Files:
As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.
When you're writing to the terminal, the output is most likely (a) line-buffered, meaning it will be flushed whenever a newline is sent.
With redirection, it's fully buffered meaning it will only flush when the buffer is full.
You can use setvbuf before operating on stdout, to set it to unbuffered or line buffered, and you won't have to worry about flushing.
setvbuf (stdout, NULL, _IONBF, BUFSIZ); // _IONBF for no buffering.
// _IOFBF for full buffering.
// _IOLBF for line buffering.
Just keep in mind that unbuffered output may affect performance quite a bit if you're sending the output to a file. Also keep in mind that support for this is implementation-defined, not guaranteed by the standard.
C99 section 7.19.3/3 is the relevant bit:
When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block.
When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled.
When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered.
Furthermore, characters are intended to be transmitted as a block to the host environment when a buffer is filled, when input is requested on an unbuffered stream, or when input is requested on a line buffered stream that requires the transmission of characters from the host environment.
Support for these characteristics is implementation-defined, and may be affected via the setbuf and setvbuf functions.
(a) Whether standard input and output are unbuffered or line buffered where the underlying file is possibly interactive is not specified by the standard (see here). Line buffering is the most common (by far) from what I've encountered.
As to why you're getting multiple entering messages even though you flush it before forking, that's a trickier one. You haven't listed your environment so this is supposition at best but I see a few possibilities (although there may well be more).
Firstly, the fflush may be failing for some reason. This is actually possible but easy to check since it behaves similarly to write (because it usually calls write under the covers).
In other words, check errno after the flush to see if there's been a problem. If so, the C runtime buffers will still be unflushed in all children so they'll all write entering.
Secondly, even if the buffers are flushed, there may be more buffering going on below (at the write level or within the terminal drivers themselves) which is duplicated by the fork, resulting in multiple outputs to the terminal. I consider this unlikely.
Thirdly, this may just be a weird platform-specific issue. When I run your code on my Ubuntu 11.04 box, I see no difference between the terminal output and the file output variants. They both output one entering and eight exiting messages.
If that third one is the case then you really have no recourse: ISO C doesn't mandate what happens in this case because ISO C knows nothing of fork. I can't find anything in POSIX.1 that seems to indicate one way or another but it may be there.
For what it's worth, if I comment out only the first fflush, I get entering twice followed by eight exiting messages. If I comment out only the second, it asts the same as if they're both there, one entering followed by eight exiting.
If I comment out both of them, I get eight entering/exiting pairs.
Regarding the multiple "entering" printed, it may be a bug(?) due to your environment.
I have already noticed the following strange behavior when running a program on zsh/Cygwin/WinXP compiled with MSVC98. If the program crashes, the stdout buffer is discarded and the program is rerun a small number of time.
For instance:
#include <stdio.h>
int main(int argc, char *argv[]) {
printf("hello\n");
fflush(stdout);
*(char*) 0 = 1;
}
will give on my PC:
> cl.exe crash.c
[etc...]
> ./crash.exe
hello
hello
hello
hello
hello
hello
Without the flush(), nothing is printed.
The same program compiled with gcc has a more "expected" behavior, even without the flush():
hello
zsh: segmentation fault (core dumped) ./a.exe
No idea why. I tried to investigate a bit, but without success.