Can a program detect if it's stdin is
coming from keyboard input vs. being
redirected from a file? in c ?
Yes, on most systems. On POSIX systems (Linux, Mac OS X, BSD, plus many more), you can use the isatty(3) function on file descriptor 0 (standard input). On Windows, you can use _isatty. For example:
if(isatty(0))
{
// standard input is a terminal device
}
Note that many programs use isatty to alter their output buffering behavior: if standard output is a terminal, the output is line buffered (e.g. it's flushed after every newline gets printed), whereas if standard output is not a terminal, it's fully-buffered (e.g. it's flushed only when the buffer becomes full, the process exits normally, or the stream is explicitly flushed by the application).
Related
I have a simple C file, that writes to the standard output using printf and write(1,..) functions.
File: main.c
#include<stdio.h>
#include<unistd.h>
int main()
{
printf("Hello\n");
printf("Mars\n");
fsync(1);
write(1,"Ola\n",4);
write(1,"Earth\n",6);
}
> gcc main.c -o test
When I redirect the output to a file, I see different order than the output in the terminal. It seems, that when I am using redirection write(..) writes before than printf(..), even tough, printf() is earlier in code and there is a fsync(..) in between.
Is there some variant of the redirection operator that will ensure order, or am I doing something wrong.
Output in terminal:
> ./test
Hello
Mars
Ola
Earth
Output in out file:
> ./test >out
# or ./test|tee out
> cat out
Ola
Earth
Hello
Mars
printf writes to the C stream stdout, which is buffered by the C standard library routines and ultimately written to Unix file descriptor 1.
fsync and write operate on Unix file descriptor 1. fsync flushes data from file descriptor 1, but it has no effect on the buffers of the streams of the C standard library.
The C standard library I/O routines operate differently depending on whether standard output is going to a terminal or to a file. This is detected during program start-up. When output is going to a terminal, it is deemed important to get it to the user, and the buffer is flushed when \n is written to a stream. This is called line buffered, and output to a terminal is set to line buffered by default.
When output is going to a file, it is not deemed important to flush it promptly, since a user is not expected to see it right away and performance is better if output is fully buffered. So the buffer is flushed only when it is fully, the stream is closed, or an explicit flush request is made. This is called fully buffered, and output to a file is set to fully buffered by default.
This is generally addressed in C 2018 7.21.3, in which paragraph 7 says:
… the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.
(When the output is a terminal, the C implementation may also use unbuffered mode instead of line buffered, in which case all characters are sent to the terminal as soon as they are written. This is uncommon.)
You can explicitly requesting flushing of stdout with fflush(stdout).
After stdout is opened but before any other operation is performed on it, you can change its buffer method with setvbuf(stdout, NULL, mode, 0), where mode is _IOFBF, _IOLBF, or _IONBF for full, line, or no buffering, respectively. (You can also pass a char array and its size in place of NULL and 0, and the library will use that array for the buffer, after which you should not use the array for any other purpose. If you pass NULL and a size, the library may use that size to perform its own buffer allocation, but that is not guaranteed.) setvbuf returns an int that may be non-zero if it cannot honor the request.
have a look at the dup2 system call: https://www.geeksforgeeks.org/dup-dup2-linux-system-call/
you can redirect stdoutput / input to a file
I'm just curious which conditions should be satisfied to flush stdout buffer automatically.
First of all I was confused that this pseudo code doesn't print output every iteration:
while (1) {
printf("Any text");
sleep(1);
}
But if I add newline character it will.
After few experiments i found that on my machine stdout buffer is flushed:
When I put to stdout 1025 characters or more;
When I read stdin;
When I put newline character to stdout;
The first condition is totally clear - when the buffer is full it should be flushed. The second one is also reasonable. But why newline character causes flushing? What are the others implicit conditions for this?
Rules of automatic flushing stdout buffer is implementation-defined (ID). It is ID when the stream is unbuffered, fully buffered, or line buffered.
When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block.
When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled.
When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered. Furthermore, characters are intended to be transmitted as a block to the host environment when a buffer is filled, when input is requested on an unbuffered stream, or when input is requested on a line buffered stream that requires the transmission of characters from the host environment.
Support for these characteristics is implementation-defined, ... C11dr §7.21.3 3
I'm just curious which conditions should be satisfied to flush stdout buffer automatically.
If code wants to insure output is certainly flushed, use fflush(). Other conditions that may automatically flush the stream are implementation defined.
A output stream which is line-buffered shall be flushed whenever a newline is output.
An implementation may (but is not required to) flush all line-buffered output streams whenever a read is attempted from any line-buffered input stream.
Implementations are not allowed to make streams fully-buffered by default unless it can be determined that they are not associated with an "interactive device". So when stdin/stdout are terminals they can't be fully-buffered, only line-buffered (or unbuffered).
If you only need flushing when the output is to a terminal, it suffices to assume that writing a newline results in flushing. Otherwise you should explicitly call fflush wherever you need flushing.
See the man page for setbuf(3). By default, stdout is set to line buffering mode.
printf() and its variants work with buffered output, and delegate to write(). So this buffering is controlled by the C library implementation of printf, with the buffer and buffer settings located in the FILE structure.
It's also worth noting the difference between section 3 and section 2 of the unix man pages. Section 2 is made up of function calls that directly talk to the operating system and do things that it would otherwise be impossible to do from a pure user program. Section 3 is made up of function calls that a user could reproduce on their own, which often delegate to section 2 calls. Section 2 functions contain the low-level "magic" that allow C programs to interact with the outside world and perform I/O. Section 3 functions can provide a more convenient interface to the section 2 functions.
printf, scanf, getchar, fputs, and other FILE * functions all are section 3 functions that delegate to write() and read(), which are section 2 functions. read() and write() do not buffer. printf() interacts with the buffer in the FILE structure, and occasionally decides to send the contents of that buffer out through write().
There are many circumstances when buffered output on a stream is flushed automatically:
When you try to do output and the output buffer is full.
When the stream is closed.
When the program terminates by calling exit.
When a newline is written, if the stream is line buffered.
Whenever an input operation on any stream actually reads data from its file.
stdout is line buffered by default.
If you want to flush the buffered output at another time,You can call fflush.
Online C2011 standard
7.21.3 Files
...
3 When a stream is unbuffered, characters are intended to appear from the source or at the
destination as soon as possible. Otherwise characters may be accumulated and
transmitted to or from the host environment as a block. When a stream is fully buffered,
characters are intended to be transmitted to or from the host environment as a block when
a buffer is filled. When a stream is line buffered, characters are intended to be
transmitted to or from the host environment as a block when a new-line character is
encountered. Furthermore, characters are intended to be transmitted as a block to the host
environment when a buffer is filled, when input is requested on an unbuffered stream, or
when input is requested on a line buffered stream that requires the transmission of
characters from the host environment. Support for these characteristics is
implementation-defined, and may be affected via the setbuf and setvbuf functions.
...
7 At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device.
So, a line buffered stream will flush on a newline. On most systems I have experience with, stdout is line buffered in an interactive session.
In my program in C, running on Linux, which creates sub-processes using system() I noticed that when I redirected stdout to a pipe or to a file then the output of the sub-processes was sent before the output of buffered I/O functions like printf(). When stdout was left to go to the terminal then the output was in the expected order. I simplified the program to the following example:
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
printf("1. output from printf()\n");
system("echo '2. output from a command called using system()'");
return EXIT_SUCCESS;
}
The expected output when stdout goes to the terminal:
$ ./iobuffer
1. output from printf()
2. output from a command called using system()
The output out of order when stdout is redirected to a pipe or a file:
$ ./iobuffer | cat
2. output from a command called using system()
1. output from printf()
A terminal generally uses line-buffering, while a pipe would use block buffering.
This means that your printf call, which includes a newline, will fill the line buffer, triggering a flush. When redirecting, no flush takes place until the program completes.
echo on the other hand, always flushes the buffer it is writing to when it completes.
With line buffering (terminal output), the order is:
printf() prints a line with newline, buffer is flushed, you see 1. output from printf() being printed.
echo writes output, exits, flushes the buffer, you see 2. output from a command called using system() printed.
With block buffering, the order is:
printf() prints a line with newline, not fully filling the buffer.
echo writes output, exits, flushes the buffer, you see 2. output from a command called using system() printed.
Your program exits, flushes its block buffer, you see 1. output from printf() being printed.
Your options are to use to flush explicitly using fflush() or to set the buffering on stdout explicitly with setvbuf().
This reply is complementing the reply of Martijn Pieters. Citations of sources describing the default buffering modes of streams are at the end.
Solutions
Here are three basic solutions with explanation:
a) Flush the buffers before calling a sub-process. This option could be significantly more effective when you do a lot of output from the main program and just few sub-process calls.
printf("1. output from printf()\n");
fflush(stdout);
system("echo '2. output from a command called using system()'");
b) Change the buffering of stdout to line-buffering (or unbuffered) for the whole program. This option is a small change to the program as you call sevbuf() only at the beginning and the rest of the program stays the same.
if(setvbuf(stdin, NULL, _IOLBF, BUFSIZ))
err(EXIT_FAILURE, NULL);
printf("1. output from printf()\n");
system("echo '2. output from a command called using system()'");
Edit:
c) Change the buffering of stdout to line-buffering (or unbuffered) for the whole program by an external utility. This option does not change the program at all so you do not need to recompile or even have sources of the program. You just call the program using the stdbuf utility.
$ stdbuf -oL ./iobuffer | cat
1. output from printf()
2. output from a command called using system()
References - describing why the buffering mode changes
The initial buffering setting is described for example in the documents below. Streams to interactive devices like terminal are by default line-buffered so that newline-ended messages appear immediately on the terminal. Pipes, files etc. use block-buffering (or full-buffering) for better performance.
The GNU C Library Reference Manual
http://www.gnu.org/software/libc/manual/html_node/Buffering-Concepts.html#Buffering-Concepts
Newly opened streams are normally fully buffered, with one exception:
a stream connected to an interactive device such as a terminal is
initially line buffered.
Linux man-pages: stdin (3)
http://linux.die.net/man/3/stdin
The stream stderr is unbuffered. The stream stdout is line-buffered
when it points to a terminal. Partial lines will not appear until
fflush(3) or exit(3) is called, or a newline is printed. This can
produce unexpected results, especially with debugging output. The
buffering mode of the standard streams (or any other stream) can be
changed using the setbuf(3) or setvbuf(3) call.
There are also a mention of the buffering of a terminal driver.
ISO/IEC 9899:201x C11 Committee Draft — April 12, 2011;
7.21.3 Files, page 301
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf
As initially opened, the standard error stream is not fully buffered;
the standard input and standard output streams are fully buffered if
and only if the stream can be determined not to refer to an
interactive device.
The Open Group: System Interfaces and Headers Issue 4, Version 2;
2.4 Standard I/O Streams, page 32
https://www2.opengroup.org/ogsys/catalog/C435 (free registration needed for download)
When opened, the standard
error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be
determined not to refer to an interactive device.
There is also a very interesting chapter 2.4.1 "Interaction of File Descriptors and Standard I/O Streams" about combining of buffered and unbuffered I/O which somewhat relates to sub-process calls.
When does the Read System call terminates when taking input from STDIN ??
There are quite a few parts to this.
First, let's clarify the distinction between OS-level IO and stdio-level IO. read(2) and write(2) (POSIX IO) are specified by POSIX, and operate using file descriptors (numbers starting from 0); fread(3) and fwrite(3) (stdio IO) are specified by ISO C and operate on file handles, such as STDIN, which on POSIX systems encapsulate file descriptors and add some things (such as output buffering) on top of them.
So, read(2) and write(2) don't do any buffering on their own. The buffering you see on standard input (file descriptor 0, not STDIN, which is one abstraction above that) is done by the terminal (or terminal emulation). Search for canonical mode to disable it.
At the stdio-level, fwrite(3) (and printf(3), fprintf(3), et al.) does output buffering depending on what the output is connected to.
See also:
How to check if a key was pressed in Linux?
Single characters are not printed on the terminal
Does printing to the screen cause a switch to kernel mode and running OS code in Unix?
I have 3 different processes that all print out single characters using printf. But I can't see them in the terminal. When I add a newline, printf("\n H") so each character is on a new line, I can see them. Why doesn't it work without the newline character?
Its a matter of flushing. If you flush the buffers after each printf, you should get output closer to what you want. To flush the standard output simply do fflush( stdout ).
The C standard defines 3 types of buffering for output streams:
Unbuffered → no buffering done
Line-buffered → buffer until newline seen
Fully-bufferd → buffer up to the buffer size
An output stream's buffering type can be changed via the setvbuf(3) and setbuf(3) functions.
The C standard requires stderr to not be fully-buffered at startup (it is usually unbuffered on many implementations, so as to see errors as soon as posible); and stdout to be fully-buffered only if it can be determined to not refer to a terminal (when it refers to a terminal, many implementations initialize it as line-buffered, which is what you are seeing).
use'write(1,&c,1)' system call, or
fprintf(stderr,'%c', c);