So a FILE stream can have both input and output buffers. You can adjust the output stream using setvbuf (I am unaware of any method to play with the input buffer size and behavior).
Also, by default the buffer is BUFSIZ (not sure if this is a POSIX or C thing). It is very clear what this means for stdin/stdout/stderr, but what are the defaults for newly opened files? Are they buffered for both input and output? Or perhaps just one?
If it is buffered, does output default to block or line mode?
EDIT: I've done some tests to see how Jonathan Leffler's answer affected real world programs. It seems that if you do a read then a write. The write will cause the unused portion of the input buffer to dropped entirely. In fact, the there will be some seeks that are done to keep things at the right file offsets. I used this simple test program:
/* input file contains "ABCDEFGHIJKLMNOPQRSTUVWXYZ" */
#include <stdio.h>
#include <stdlib.h>
int main() {
FILE *f = fopen("test.txt", "r+b");
char ch;
fread(&ch, 1, 1, f);
fwrite("test", 4, 1, f);
fclose(f);
return 0;
}
resulted in the following system calls:
read(3, "ABCDEFGHIJKLMNOPQRSTUVWXYZ\n", 4096) = 27 // attempt to read 4096 chars, got 27
lseek(3, -26, SEEK_CUR) = 1 // at this point, I've done my write already, so forget the 26 chars I never asked for and seek to where I should be if we really just read one character...
write(3, "test", 4) = 4 // and write my test
close(3) = 0
While these are clearly implementation details I found them to be very interesting as far as how the standard library could be implemented. Thanks Jonathan for your insightful answer.
A single file stream has a single buffer. If the file is used for both input and output, then you have to ensure that you do appropriate operations (fseek() or equivalents) between the read and write (or write and read) operations.
The buffering behaviour of the standard channels is platform dependent.
Typically, stdout is line buffered when the output goes to the terminal. However, if stdout is going to a file or pipe rather than to a terminal, it most usually switches to full buffering.
Typically, stderr is either line buffered or unbuffered, to ensure that error messages get seen (for example, even if the program is about to crash).
Typically, stdin is line buffered; this means you get a chance to edit your input (backspacing over errors, etc). You would seldom adjust this. Again, if the input is coming from a file (or pipe), the behaviour might be different.
Newly opened files will generally be fully buffered. A particular implementation might change that to line buffering if the device is a terminal.
Your premise - that there are two buffers - is incorrect.
Section 7.19.3 of C99, it says:
At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device.
So, as originally stated, stderr is either unbuffered or line buffered (it is not fully buffered).
Related
I am going through "C PRIMER PLUS" and there is this topic about "OUTPUT FLUSHING".
Now it says:
printf() statements sends output to an intermediate storage called buffer.
Every now and then, the material in the buffer is sent to the screen. The
standard C rules for when output is sent from the buffer to the screen are
clear:
It is sent when the buffer gets full.
When a newline character is encountered.
When there is impending input.
(Sending the output from the buffer to the screen or file is called flushing
the buffer.)
Now, To verify the above statements. I wrote this simple program :
#include<stdio.h>
int main(int argc, char** argv) {
printf("Hello World");
return 0;
}
so, neither the printf() contains a new line, nor it has some impending input(for e.g. a scanf() statement or any other input statement). Then why does it print the contents on the output screen.
Let's suppose first condition validated to true. The buffer got full(Which can't happen at all).
Keeping that in mind, I truncated the statement inside printf() to
printf("Hi");
Still it prints the statement on the console.
So whats the deal here, All of the above conditions are false but still I'm getting the output on screen.
Can you elaborate please. It appears I'm making a mistake in understanding the concept. Any help is highly appreciated.
EDIT: As suggested by a very useful comment, that maybe the execution of exit() function after the end of program is causing all the buffers to flush, resulting in the output on the console. But then if we hold the screen before the execution of exit(). Like this,
#include<stdio.h>
int main(int argc, char** argv) {
printf("Hello World!");
getchar();
return 0;
}
It still outputs on the console.
Output buffering is an optimization technique. Writing data to some devices (hard disks f.e.) is an expensive operation; that's why the buffering appeared. In essence, it avoids writing data byte-by-byte (or char-by-char) and collects it in a buffer in order to write several KiB of data at once.
Being an optimization, output buffering must be transparent to the user (it is transparent even to the program). It must not affect the behaviour of the program; with or without buffering (or with different sizes of the buffer), the program must behave the same. This is what the rules you mentioned are for.
A buffer is just an area in memory where the data to be written is temporarily stored until enough data accumulates to make the actual writing process to the device efficient. Some devices (hard disk etc.) do not even allow writing (or reading) data in small pieces but only in blocks of some fixed size.
The rules of buffer flushing:
It is sent when the buffer gets full.
This is obvious. The buffer is full, its purpose was fulfilled, let's push the data forward to the device. Also, probably there is more data to come from the program, we need to make room for it.
When a newline character is encountered.
There are two types of devices: line-mode and block-mode. This rule applies only to the line-mode devices (the terminal, for example). It doesn't make much sense to flush the buffer on newlines when writing to disk. But it makes a lot of sense to do it when the program is writing to the terminal. In front of the terminal there is the user waiting impatiently for output. Don't let them wait too much.
But why output to terminal needs buffering? Writing on the terminal is not expensive. That's correct, when the terminal is physically located near the processor. Not also when the terminal and the processor are half the globe apart and the user runs the program through a remote connection.
When there is impending input.
It should read "when there is impeding input on the same device" to make it clear.
Reading is also buffered for the same reason as writing: efficiency. The reading code uses its own buffer. It fills the buffer when needed then scanf() and the other input-reading functions get their data from the input buffer.
When an input is about to happen on the same device, the buffer must be flushed (the data actually written to the device) in order to ensure consistency. The program has send some data to the output and now it expects to read back the same data; that's why the data must be flushed to the device in order for the reading code find it there and load it.
But why the buffers are flushed when the application exits?
Err... buffering is transparent, it must not affect the application behaviour. Your application has sent some data to the output. The data must be there (on the output device) when the application quits.
The buffers are also flushed when the associated files are closed, for the same reason. And this is what happens when the application exits: the cleanup code close all the open files (standard input and output are just files from the application point of view), closing forces flushing the buffers.
Part of the specification for exit() in the C standard (POSIX link given) is:
Next, all open streams with unwritten buffered data are flushed, all open streams are closed, …
So, when the program exits, pending output is flushed, regardless of newlines, etc. Similarly, when the file is closed (fclose()), pending output is written:
Any unwritten buffered data for the stream are delivered to the host environment to be written to the file; any unread buffered data are discarded.
And, of course, the fflush() function flushes the output.
The rules quoted in the question are not wholly accurate.
When the buffer is full — this is correct.
When a newline is encountered — this is not correct, though it often applies. If the output device is an 'interactive device', then line buffering is the default. However, if the output device is 'non-interactive' (disk file, a pipe, etc), then the output is not necessarily (or usually) line-buffered.
When there is impending input — this too is not correct, though it is commonly the way it works. Again, it depends on whether the input and output devices are 'interactive'.
The output buffering mode can be modified by calling setvbuf()
to set no buffering, line buffering or full buffering.
The standard says (§7.21.3):
¶3 When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block. When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled. When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered. Furthermore, characters are intended to be transmitted as a block to the host environment when a buffer is filled, when input is requested on an unbuffered stream, or when input is requested on a line buffered stream that requires the transmission of characters from the host environment. Support for these characteristics is implementation-defined, and may be affected via the setbuf and setvbuf functions.
…
¶7 At program startup, three text streams are predefined and need not be opened explicitly — standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.
Also, §5.1.2.3 Program execution says:
The input and output dynamics of interactive devices shall take place as specified in 7.21.3. The intent of these requirements is that unbuffered or line-buffered output appear as soon as possible, to ensure that prompting messages actually appear prior to a program waiting for input.
The strange behavior of printf, buffering can be explained with below simple C code. please read through entire thing execute and understand as the below is not obvious (bit tricky)
#include <stdio.h>
int main()
{
int a=0,b=0,c=0;
printf ("Enter two numbers");
while (1)
{
sleep (1000);
}
scanf("%d%d",&b,&c);
a=b+c;
printf("The sum is %d",a);
return 1;
}
EXPERIMENT #1:
Action: Compile and Run above code
Observations:
The expected output is
Enter two numbers
But this output is not seen
EXPERIMENT #2:
Action: Move Scanf statement above while loop.
#include <stdio.h>
int main()
{
int a=0,b=0,c=0;
printf ("Enter two numbers");
scanf("%d%d",&b,&c);
while (1)
{
sleep (1000);
}
a=b+c;
printf("The sum is %d",a);
return 1;
}
Observations: Now the output is printed (reason below in the end)(just by scanf position change)
EXPERIMENT #3:
Action: Now add \n to print statement as below
#include <stdio.h>
int main()
{
int a=0,b=0,c=0;
printf ("Enter two numbers\n");
while (1)
{
sleep (1000);
}
scanf("%d%d",&b,&c);
a=b+c;
printf("The sum is %d",a);
return 1;
}
Observation: The output Enter two numbers is seen (after adding \n)
EXPERIMENT #4:
Action: Now remove \n from the printf line, comment out while loop, scanf line, addition line, printf line for printing result
#include <stdio.h>
int main()
{
int a=0,b=0,c=0;
printf ("Enter two numbers");
// while (1)
// {
// sleep (1000);
// }
// scanf("%d%d",&b,&c);
// a=b+c;
// printf("The sum is %d",a);
return 1;
}
Observations: The line "Enter two numbers" is printed to screen.
ANSWER:
The reason behind the strange behavior is described in Richard Stevens book.
PRINTF PRINTS TO SCREEN WHEN
The job of printf is to write output to stdout buffer. kernel flushes output buffers when
kernel need to read something in from input buffer. (EXPERIMENT #2)
when it encounters newline (since stdout is by default set to
linebuffered)(EXPERIMENT #3)
after program exits (all output buffers are flushed) (EXPERIMENT #4)
By default stdout set to line buffering
so printf will not print as the line did not end.
if it is no buffered, all lines are output as is.
Full buffered then, only when buffer is full it is flushed.
I had started learning C programming, so I'm a beginner, while learning about standard streams of text, I came up with the lines "stdout" stream is buffered while "stderr" stream is not buffered, but I am not able to make sense with this lines.
I already have read about "buffer" on this forum and I like candy analogy, but I am not able to figure out what is meant when one says: "This stream is buffered and the other one is not." What is the effect?
What is the difference?
Update: Does it affect the speed of processing?
Buffer is a block of memory which belongs to a stream and is used to hold stream data temporarily. When the first I/O operation occurs on a file, malloc is called and a buffer is obtained. Characters that are written to a stream are normally accumulated in the buffer (before being transmitted to the file in chunks), instead of appearing as soon as they are output by the application program. Similarly, streams retrieve input from the host environment in blocks rather than on a character-by-character basis. This is done to increase efficiency, as file and console I/O is slow in comparison to memory operations.
GCC provides three types of buffering - unbuffered, block buffered, and line buffered. Unbuffered means that characters appear on the destination file as soon as written (for an output stream), or input is read from a file on a character-by-character basis instead of reading in blocks (for input streams). Block buffered means that characters are saved up in the buffer and written or read as a block. Line buffered means that characters are saved up only till a newline is written into or read from the buffer.
stdin and stdout are block buffered if and only if they can be determined not to refer to an interactive device else they are line buffered (this is true of any stream). stderr is always unbuffered by default.
The standard library provides functions to alter the default behaviour of streams. You can use fflush to force the data out of the output stream buffer (fflush is undefined for input streams). You can make the stream unbuffered using the setbuf function.
Buffering is collecting up many elements before writing them, or reading many elements at once before processing them. Lots of information out there on the Internet, for example, this
and other SO questions like this
EDIT in response to the question update: And yes, it's done for performance reasons. Writing and reading from disks etc will in any case write or read a 'block' of some sort for most devices, and there's a fair overhead in doing so. So batching these operations up can make for a dramatic performance difference
A program writing to buffered output can perform the output in the time it takes to write to the buffer which is typically very fast, independent of the speed of the output device which may be slow.
With buffered output the information is queues and a separate process deals with the output rendering.
With unbuffered output, the data is written directly to the output device, so runs at the speed on the device. This is important for error output because if the output were buffered it would be possible for the process to fail before the buffered output has made it to the display - so the program might terminate with no diagnostic output.
int main()
{
printf("Hello"); // doesn't display anything on the screen
printf("\n"); // hello is display on the screen
return 0;
}
All characters(candidate of printing) are buffered until a new line is received? Correct?
Q1 - Why does it wait before printing on terminal until a newline char?
Q2 - Where are the characters of first printf (i.e. "Hello") buffered?
Q3 - What is the flow of printing printf()->puts()->putchar() -> now where? driver? Does the driver has a control to wait until \n?
Q4 - What is the role stdout that is attached to a process?
Looking for a in-depth picture. Feel free to edit the question, if something doesn't makes sense.
printf is not writing directly to the screen, instead it writes to the output stream, which is by default buffered. The reason for this is, that there may not even be a screen attached and the output can go to a file as well. For performance reasons, it is better for a system if access to disc is buffered and then executed in one step with appropriately sized chunks, rather than writing every time.
You can even change the size of the buffer and set it to 0, which means that all output goes directly to the target, which may be usefull for logging purposes.
setbuf(stdout, NULL);
The buffer is flushed either when it is full, or if certain criterions are fullfilled, like printing a newline. So when you would execute the printf in a loop, you would notice that it will write out in chunks unless you have a newline inbetween.
I'll start with some definitions and then go on to answer your questions.
File: It is an ordered sequence of bytes. It can be a disk file, a stream of bytes generated by a program (such as a pipeline), a TCP/IP socket, a stream of bytes received from or sent to a peripheral device (such as the keyboard or the display) etc. The latter two are interactive files. Files are typically the principal means by which a program communicates with its environment.
Stream: It is a representation of flow of data from one place to another, e.g., from disk to memory, memory to disk, one program to another etc. A stream is a source of data where data can be put into (write) or taken data out of (read). Thus, it's an interface for writing data into or reading data from a file which can be any type as stated above. Before you can perform any operation on a file, the file must be opened. Opening a file associates it with a stream. Streams are represented by FILE data type defined in stdio.h header. A FILE object (it's a structure) holds all of the internal state information about the connection to the associated file, including such things as the file position indicator and buffering information. FILE objects are allocated and managed internally by the input/output library functions and you should not try to create your own objects of FILE type, the library does it for us. The programs should deal only with pointers to these objects (FILE *) rather than the objects themselves.
Buffer: Buffer is a block of memory which belongs to a stream and is used to hold stream data temporarily. When the first I/O operation occurs on a file, malloc is called and a buffer is obtained. Characters that are written to a stream are normally accumulated in the buffer (before being transmitted to the file in chunks), instead of appearing as soon as they are output by the application program. Similarly, streams retrieve input from the host environment in blocks rather than on a character-by-character basis. This is done to increase efficiency, as file and console I/O is slow in comparison to memory operations.
The C library provides three predefined text streams (FILE *) open and available for use at program start-up. These are stdin (the standard input stream, which is the normal source of input for the program), stdout (the standard output stream, which is used for normal output from the program), and stderr (the standard error stream, which is used for error messages and diagnostics issued by the program). Whether these streams are buffered or unbuffered is implementation-defined and not required by the standard.
GCC provides three types of buffering - unbuffered, block buffered, and line buffered. Unbuffered means that characters appear on the destination file as soon as written (for an output stream), or input is read from a file on a character-by-character basis instead of reading in blocks (for input streams). Block buffered means that characters are saved up in the buffer and written or read as a block. Line buffered means that characters are saved up only till a newline is written into or read from the buffer.
stdin and stdout are block buffered if and only if they can be determined not to refer to an interactive device else they are line buffered (this is true of any stream). stderr is always unbuffered by default.
The standard library provides functions to alter the default behaviour of streams. You can use fflush to force the data out of the output stream buffer (fflush is undefined for input streams). You can make the stream unbuffered using the setbuf function.
Now, let's come to your questions.
Unmarked question: Yes, becausestdout normally refers to a display terminal unless you have output redirection using > operator.
Q1: It waits because stdout is newline buffered when it refers to a terminal.
Q2: The characters are buffered, well, in the buffer allocated to the stdout stream.
Q3: Flow of the printing is: memory --> stdout buffer --> display terminal. There are kernel buffers as well controlled by the OS which the data pass through before appearing on the terminal.
Q4: stdout refers to the standard output stream which is usually a terminal.
Finally, here's a sample code to experiment things before I finish my answer.
#include <stdio.h>
#include <limits.h>
int main(void) {
// setbuf(stdout, NULL); // make stdout unbuffered
printf("Hello, World!"); // no newline
// printf("Hello, World!"); // with a newline
// only for demonstrating that stdout is line buffered
for(size_t i = 0; i < UINT_MAX; i++)
; // null statement
printf("\n"); // flush the buffer
return 0;
}
Yes, by default, standard output is line buffered when it's connected to a terminal. The buffer is managed by the operating system, normally you don't have to worry about it.
You can change this behavior using setbuf() or setvbuf(), for example, to change it to no buffer:
setbuf(stdout, NULL);
All the functions of printf, puts, putchar outputs to the standard output, so they use the same buffer.
If you wish, you can flush out the characters before the new line by calling
fflush(stdout);
This can be handy if you're slowly printing something like a progress bar where each character gets printed without a newline.
int main()
{
printf("Hello"); // Doesn't display anything on the screen
fflush(stdout); // Now, hello appears on the screen
printf("\n"); // The new line gets printed
return 0;
}
#include <stdio.h>
int main()
{
int age;
printf("Enter Your age: ");
scanf("%lf", &age);
printf("Your age is %d\n", age);
return 0;
}
If we run the program the prompt will be displayed on the terminal screen, even though there is no newline character at the end of the prompt. How can this happen?
That most likely happens because your C standard library chooses to flush the output buffers in its implementation of scanf. That's a good idea because
this is a common problem and you usually want the prompt at the same line as the cursor and
it's not a performance problem because waiting for user input is way slower than flushing the buffer anyways
You can run an experiment:
printf("Enter a number: "); /* the prompt */
sleep(10); /* Add this. */
scanf("%lf", &number);
You'll see the prompt popping up after 10 seconds. It's not documented - that I know of - but it appears in this case scanf causes stdout to be flushed.
You shouldn't count on this behavior however and you should fflush(stdout) when printing a prompt string.
As noted in a comment, the C standard (ISO/IEC 9899:2011) says:
§5.1.2.3 Program execution
¶6 The least requirements on a conforming implementation are:
Accesses to volatile objects are evaluated strictly according to the rules of the abstract
machine.
At program termination, all data written into files shall be identical to the result that
execution of the program according to the abstract semantics would have produced.
The input and output dynamics of interactive devices shall take place as specified in
7.21.3. The intent of these requirements is that unbuffered or line-buffered output
appear as soon as possible, to ensure that prompting messages actually appear prior to
a program waiting for input.
This is the observable behavior of the program.
¶7 What constitutes an interactive device is implementation-defined.
¶8 More stringent correspondences between abstract and actual semantics may be defined by
each implementation.
And the library section describing the functions in <stdio.h> says:
§7.21.3 Files
¶3 When a stream is unbuffered, characters are intended to appear from the source or at the
destination as soon as possible. Otherwise characters may be accumulated and
transmitted to or from the host environment as a block. When a stream is fully buffered,
characters are intended to be transmitted to or from the host environment as a block when
a buffer is filled. When a stream is line buffered, characters are intended to be
transmitted to or from the host environment as a block when a new-line character is
encountered. Furthermore, characters are intended to be transmitted as a block to the host
environment when a buffer is filled, when input is requested on an unbuffered stream, or
when input is requested on a line buffered stream that requires the transmission of
characters from the host environment. Support for these characteristics is
implementation-defined, and may be affected via the setbuf and setvbuf functions.
¶7 At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device.
Note the 'intent' comment in §5.1.2.3 ¶6.
The specification in §7.21.3 ¶7 means that standard error is either unbuffered or line buffered at program startup, and standard input and standard output are either unbuffered or line buffered (usually line buffered) unless the output is not an interactive device. On Unix, disk files, pipes, FIFOs, sockets (amongst others) are not interactive. Usually tty and pty devices are deemed interactive.
So, if you run your program with output going to a pipe (e.g. to cat or less or more), you probably* won't see a prompt, but if you type the number, you'll probably get the prompt followed by the first line of output with a newline all on a single line (and the number you typed won't appear). If you redirect the input from a file, you won't see the entered value at all. If you redirect to cat, you'll likely see your input on one line, then the prompt and the output. If you redirect to more, you may not see your input at all (separately from the response printing). If you redirect to less, you may get all sorts of interesting screen effects, and still not see the input you type.
You also need to fix the revised code. The %lf format is not appropriate for reading an integer (it is a hangover from a previous version of the code).
* Since the behaviour is implementation defined, what actually happens depends on your implementation. An implementation might do the equivalent of fflush(0) before any input, for example; this would send the prompt to the device. But no implementation is required to do that, and most aren't that thorough (partly because such thoroughness has a runtime cost associated with it).
Could anyone clarify on the types of buffers used by a program?
For eg:
I have a C program that reads from a stdin to stdout.
What are the buffers involved here? I'm aware that there are 2.
One provided by the kernel on which a user don't have any control.
One provided with standard streams namely stdout, stdin and stderr. Each having a separate buffer.
Is my understanding correct?
Thanks,
John
If you are working on linux/unix then you could more easily understand that there are three streams namely
1.STDIN: FILE DESCRIPTOR VALUE 0 (IN unix)
2.STDOUT :FILE DESCRIPTOR VALUE 1
3.STDERR :FILE DESCRIPTOR VALUE 2
By default these streams correspond to keyboard and monitor.In unix we can change these streams to read input from file instead of keyboard.To display output on a file rather than monitor using close(),dup() system calls.Yes there are 3 buffers involved.To clear the contents of input buffer in c we use fflush() function.
If you want to know more about handling these streams in UNIX then let me Know.
The kernel (or other underlying system) could have any number of layers of buffering, depending on what device is being read from and the details of the kernel implementation; in some systems there is no buffering at that level, with the data being read directly into the userspace buffer.
The stdio library allocates a buffer for stdin; the size is implementation-dependent but you can control the size and even use your own buffer with setvbuf. It also allows you to control whether I/O is fully buffered (as much data is read into the buffer as is available), line buffered (data is is only read until a newline is encountered), or unbuffered. The default is line buffering if the system can determine that the input is a terminal, else fully buffered.
The story is similar for stdout. stderr is by default unbuffered.