I am trying to learn the libuv api and wrote the following test:
#include <stdio.h>
#include <stdlib.h>
#include <uv.h>
void timer_cb(uv_timer_t* timer) {
int* i = timer->data;
--*i;
if(*i == 0) {
uv_timer_stop(timer);
}
printf("timer %d\n", *i);
//fflush(stdout);
}
int main() {
uv_loop_t* loop = uv_default_loop();
uv_timer_t* timer = malloc(sizeof(uv_timer_t));
uv_timer_init(loop, timer);
int i = 5;
timer->data = &i;
uv_timer_start(timer, timer_cb, 1000, 2000);
uv_run(loop, UV_RUN_DEFAULT);
printf("Now quitting.\n");
uv_close(timer, 0);
uv_loop_close(loop);
return 0;
}
When run it, no output is displayed until the program finishes running, and then all the output is displayed at once. If I uncomment the fflush line it works as expected, writing every 2 seconds.
Can someone please explain this to me? Why is stdout not flushed after the newline, as is explained here and in other places? Why do I need tomanually flush it?
Stream buffering is implementation-defined.
Per 7.21.3 Files, paragraph 3 of the C Standard:
When a stream is
unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters
may be accumulated and transmitted to or from the host
environment as a block. When a stream is
fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled. When a
stream is
line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line
character is encountered. Furthermore, characters are intended to be
transmitted as a block to the host environment when a buffer is
filled, when input is requested on an unbuffered stream, or when
input is requested on a line buffered stream that requires
the transmission of characters from the host
environment. Support for these characteristics is
implementation-defined, and may be affected via the setbuf and
setvbuf functions.
The type of buffering is dependent on your implementation, and your implementation apparently isn't line-buffering in your example.
There is no strict requirement, that stdout is line buffered. It may be fully buffered as well (or not buffered at all), in which case \n does not trigger to flush the stream.
C11 (N1570) 7.21.3/7 Files:
As initially opened, the standard error stream is not fully buffered;
the standard input and standard output streams are fully buffered if
and only if the stream can be determined not to refer to an
interactive device.
C11 (N1570) 5.1.2.3/7 Program execution:
What constitutes an interactive device is implementation-defined.
You could try to force specific type of buffering by setvbuf standard function. For instance, to set line buffering for stdout, you may try with:
setvbuf(stdout, buff, _IOLBF, size);
where buff is declared as character array of size elements (e.g. 1024).
Note that setvbuf has to be called before any other I/O operation, that is performed to the stream.
For some reason, your system is deciding that your stdout is not interactive. Are you doing some strange redirect of stdout or doing something weird with your terminal? You should be able to override using setbuf or you can use stderr instead of stdout.
Related
The behaviour of printf() seems to depend on the location of stdout.
If stdout is sent to the console, then printf() is line-buffered and is flushed after a newline is printed.
If stdout is redirected to a file, the buffer is not flushed unless fflush() is called.
Moreover, if printf() is used before stdout is redirected to file, subsequent writes (to the file) are line-buffered and are flushed after newline.
When is stdout line-buffered, and when does fflush() need to be called?
Minimal example of each:
void RedirectStdout2File(const char* log_path) {
int fd = open(log_path, O_RDWR|O_APPEND|O_CREAT,S_IRWXU|S_IRWXG|S_IRWXO);
dup2(fd,STDOUT_FILENO);
if (fd != STDOUT_FILENO) close(fd);
}
int main_1(int argc, char* argv[]) {
/* Case 1: stdout is line-buffered when run from console */
printf("No redirect; printed immediately\n");
sleep(10);
}
int main_2a(int argc, char* argv[]) {
/* Case 2a: stdout is not line-buffered when redirected to file */
RedirectStdout2File(argv[0]);
printf("Will not go to file!\n");
RedirectStdout2File("/dev/null");
}
int main_2b(int argc, char* argv[]) {
/* Case 2b: flushing stdout does send output to file */
RedirectStdout2File(argv[0]);
printf("Will go to file if flushed\n");
fflush(stdout);
RedirectStdout2File("/dev/null");
}
int main_3(int argc, char* argv[]) {
/* Case 3: printf before redirect; printf is line-buffered after */
printf("Before redirect\n");
RedirectStdout2File(argv[0]);
printf("Does go to file!\n");
RedirectStdout2File("/dev/null");
}
Flushing for stdout is determined by its buffering behaviour. The buffering can be set to three modes: _IOFBF (full buffering: waits until fflush() if possible), _IOLBF (line buffering: newline triggers automatic flush), and _IONBF (direct write always used). "Support for these characteristics is implementation-defined, and may be affected via the setbuf() and setvbuf() functions." [C99:7.19.3.3]
"At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device." [C99:7.19.3.7]
Explanation of observed behaviour
So, what happens is that the implementation does something platform-specific to decide whether stdout is going to be line-buffered. In most libc implementations, this test is done when the stream is first used.
Behaviour #1 is easily explained: when the stream is for an interactive device, it is line-buffered, and the printf() is flushed automatically.
Case #2 is also now expected: when we redirect to a file, the stream is fully buffered and will not be flushed except with fflush(), unless you write gobloads of data to it.
Finally, we understand case #3 too for implementations that only perform the check on the underlying fd once. Because we forced stdout's buffer to be initialised in the first printf(), stdout acquired the line-buffered mode. When we swap out the fd to go to file, it's still line-buffered, so the data is flushed automatically.
Some actual implementations
Each libc has latitude in how it interprets these requirements, since C99 doesn't specify what an "interactive device" is, nor does POSIX's stdio entry extend this (beyond requiring stderr to be open for reading).
Glibc. See filedoalloc.c:L111. Here we use stat() to test if the fd is a tty, and set the buffering mode accordingly. (This is called from fileops.c.) stdout initially has a null buffer, and it's allocated on the first use of the stream based on the characteristics of fd 1.
BSD libc. Very similar, but much cleaner code to follow! See this line in makebuf.c
Your are wrongly combining buffered and unbuffered IO functions. Such a combination must be done very carefully especially when the code has to be portable. (and it is bad to write unportable code...)
It is certainly best to avoid combining buffered and unbuffered IO on the same file descriptor.
Buffered IO: fprintf(), fopen(), fclose(), freopen()...
Unbuffered IO: write(), open(), close(), dup()...
When you use dup2() to redirect stdout. The function is not aware of the buffer which was filled by fprintf(). So when dup2() closes the old descriptor 1 it does not flush the buffer and the content could be flushed to a different output. In your case 2a it was sent to /dev/null.
The solution
In your case it is best to use freopen() instead of dup2(). This solves all your problems:
It flushes the buffers of the original FILE stream. (case 2a)
It sets the buffering mode according to the newly opened file. (case 3)
Here is the correct implementation of your function:
void RedirectStdout2File(const char* log_path) {
if(freopen(log_path, "a+", stdout) == NULL) err(EXIT_FAILURE, NULL);
}
Unfortunately with buffered IO you cannot directly set permissions of a newly created file. You have to use other calls to change the permissions or you can use unportable glibc extensions. See the fopen() man page.
You should not close the file descriptor, so remove close(fd) and close
stdout_bak_fd if you want the message to be print only in the file.
I am writing a multi-threaded program. The structure of the program is as given below
int main()
{
printf("To Call threads\n");
thread1=pthread_create(&trd1,NULL,process1,(void *)sleepTimeForP1);
thread2=pthread_create(&trd2,NULL,process2,(void *)sleepTimeForP2);
pthread_join(trd1, NULL);
pthread_join(trd2, NULL);
return 0;
}
The code for thread is
void *process1 (void *sleepTimeForP1)
{
int *tsleepTimeForP1 = (int *)sleepTimeForP1;
while(1)
{
pthread_mutex_lock(&lock);
printf("inside thread\n");
pthread_mutex_unlock(&lock);
sleep(1);
}
}
The code works fine, and prints the output when no output redirection is done. But when output redirection is given, no ouput is obtained in the file. Now If I give explicit fflush(stdout) after every printf() statement, I am getting the desired output. Why am I required to give the explicit fflush() ?
From C99 §7.19.3[7]:
As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.
'Fully buffered' (vs 'line buffered'): C99 §7.19.3[3]
When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled. When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered.
So, when standard-output is interactive (e.g. a shell) it is line-buffered. But when you redirect it to a file it becomes fully buffered and you must use fflush.
These days I was learning the "apue", a result of an typical case confused me. The following are the sample codes of "sample.c":
#include "apue.h"
#include <stdio.h>
#define BUFF_SZ 4096
int main()
{
int n = 0;
char buff[BUFF_SZ] = {'\0'};
while ((n = read(STDIN_FILENO, buff, BUFF_SZ)) > 0) {
printf("read %d bytes\n", n);
if (write(STDOUT_FILENO, buff, n) != n) {
err_sys("write error");
}
}
if (n < 0) {
err_sys("read error");
}
return 0;
}
After compilation gcc sample.c, you can use this command echo Hello | ./a.out and get the following std output on terminal:
read 6 bytesHello
However, if you redirect the output to a file echo Hello | ./a.out > outfile, then use cat outfile to see the content:
Helloread 6 bytes
The ouput changes order after redirection! I wonder if some one could tell me the reason?
For the standard I/O function printf, when you output to a terminal, the standard output is by default line buffered.
printf("read %d bytes\n", n);
\n here cause the output to flush.
However, when you output to a file, it's by default fully buffered. The output won't flush unless the buffer is full, or you explicitly flush it.
The low level system call write, on the other hand, is unbuffered.
In general, intermixing standard I/O calls with system calls is not advised.
printf(), by default, buffers its output, while write() does not, and there is no synchronisation between then.
So, in your code, it is possible that printf() stores its data in a buffer and returns, then write() is called, and - as main() returns, printf()s buffer is flushed so that buffered output appears. From your description, that is happening when output is redirected.
It is also possible that printf() writes data immediately, then write() is called. From your description, that happens when output is not redirected.
Typically, one part of redirection of a stream is changing the buffer - and therefore the behaviour when buffering - for streams like stdout and stdin. The precise change depends on what type of redirection is happening (e.g. to a file, to a pipe, to a different display device, etc).
Imagine that printf() writes data to a buffer and, when flushing that buffer, uses write() to produce output. That means all overt calls of write() will have their output produced immediately, but data that is buffered may be printed out of order.
The problem is that the writes are handled by write(2) call, so you effectively lose control of what happens.
If we look at the documentation for write(2) we can see that the writes are not guaranteed to be actually written until a read() occurs. More specifically:
A successful return from write() does not make any guarantee that data has
been committed to disk. In fact, on some buggy implementations, it does not even
guarantee that space has successfully been reserved for the data. The only way to
be sure is to call fsync(2) after you are done writing all your data.
This means that depending on the implementation and buffering of the write(2) (which may differ even between redirects and printing to screen), you can get different results.
I was writing a small program that had various console output strings depending upon different events. As I was looking up the best way to send these messages I came across something that was a bit confusing.
I have read that stderr is used to shoot messages directly to the console - not buffered. While, in contrast, I read that stdout is buffered and is typically used to redirect messages to various streams?, that may or may not be error messages, to an output file or some other medium.
What is the difference when something is said to be buffered and not buffered? It made sense when I was reading that the message is shot directly to the output and is not buffered .. but at the same time I realized that I was not entirely sure what it meant to be buffered.
Typically, stdout is line buffered, meaning that characters sent to stdout "stack up" until a newline character arrives, at which point that are all outputted.
A buffered stream is one in which you keep writing until a certain threshold. This threshold could be a specific character, as Konrad mentions for line buffering, or another threshold, such as a specific count of characters written.
Buffering is intended to speed up input/output operations. One of the slowest things a computer does is write to a stream (whether a console or a file). When things don't need to be immediately seen, it saves time to store it up for a while.
You are right, stderr is typically an unbuffered stream while stdout typically is buffered. So there can be times when you output things to stdout then output to stderr and stderr appears first on the console. If you want to make stdout behave similarly, you would have to flush it after every write.
When an output stream is buffered, it means that the stream doesn't necessarily output data the moment you tell it to. There can be significant overhead per IO operation, so lots and lots of little IO operations can create a bottleneck. By buffering IO operations and then flushing many at once, this overhead is reduced.
While stdout and stderr may behave differently regarding buffering, that is generally not the deciding factor between them and shouldn't be relied on. If you absolutely need the output immediately, always manually flush the stream.
Assume
int main(void)
{
printf("foo\n");
sleep(10);
printf("bar\n");
}
when executing it on the console
$ ./a.out
you will see foo line and 10 seconds later bar line (--> line buffered). When redirecting the output into a file or a pipe
$ ./a.out > /tmp/file
the file stays empty (--> buffered) until the program terminates (--> implicit fflush() at exit).
When lines above do not contain a \n, you won't see anything on the console either until program terminates.
Internally, printf() adds a character to a buffer. To make things more easy, let me describe fputs(char const *s, FILE *f) instead of. FILE might be defined as
struct FILE {
int fd; /* is 0 for stdin, 1 for stdout, 2 for stderr (usually) */
enum buffer_mode mode;
char buf[4096];
size_t count; /* number of chars in buf[] */
};
typedef struct FILE *FILE;
int fflush(FILE *f)
{
write(f->fd, f->buf, f->count);
f->count = 0;
}
int fputc(int c, FILE *f)
{
if (f->count >= ARRAY_SIZE(f->buf))
fflush(f);
f->buf[f->count++] = c;
}
int fputs(char const *s, FILE *f)
{
while (*s) {
char c = *s++;
fputc(c, f);
if (f->mode == LINE_BUFFERED && c == '\n')
fflush(f);
}
if (f->mode == UNBUFFERED)
fflush(f);
}
have a look at this code:
#include<stdio.h>
#include <unistd.h>
int main()
{
int pipefd[2],n;
char buf[100];
if(pipe(pipefd)<0)
printf("Pipe error");
printf("\nRead fd:%d write fd:%d\n",pipefd[0],pipefd[1]);
if(write(pipefd[1],"Hello Dude!\n",12)!=12)
printf("Write error");
if((n=read(pipefd[0],buf,sizeof(buf)))<=0)
printf("Read error");
write(1,buf,n);
return 0;
}
I expect the printf to print Read fd and write fd before Hello Dude is read from the pipe. But thats not the case... see here. When i tried the same program in our college computer lab my output was
Read fd:3 write fd:4
Hello Dude!
also few of our friends observed that, changing the printf statement to contain more number of \n characters changed the output order... for example..printf("\nRead fd:%d\n write fd:%d\n",pipefd[0],pipefd[1]); meant that Read fd is printed then the message Hello Dude! then the write fd is printed. What is this behaviour??
Note: Out lab uses a linux server on which we run terminals, i don't remember the compiler version though.
It's because printf to the standard output stream is buffered but write to the standard output file descriptor is not.
That means the behaviour can change based on what sort of buffering you have. In C, standard output is line buffered if it can be determined to be connected to an interactive device. Otherwise it's fully buffered (see here for a treatise on why this is so).
Line buffered means it will flush to the file descriptor when it sees a newline. Fully buffered means it will only flush when the buffer fills (for example, 4K worth of data), or when the stream is closed (or when you fflush).
When you run it interactively, the flush happens before the write because printf encounters the \n and flushes automatically.
However, when you run it otherwise (such as by redirecting output to a file or in an online compiler/executor where it would probably do the very same thing to capture data for presentation), the flush happens after the write (because printf is not flushing after every line).
In fact, you don't need all that pipe stuff in there to see this in action, as per the following program:
#include <stdio.h>
#include <unistd.h>
int main (void) {
printf ("Hello\n");
write (1, "Goodbye\n", 8);
return 0;
}
When I execute myprog ; echo === ; myprog >myprog.out ; cat myprog.out, I get:
Hello
Goodbye
===
Goodbye
Hello
and you can see the difference that the different types of buffering makes.
If you want line buffering regardless of redirection, you can try:
setvbuf (stdin, NULL, _IOLBF, BUFSIZ);
early on in your program - it's implementation defined whether an implementation supports this so it may have no effect but I've not seen many where it doesn't work.
You shouldn't mix calls to write and printf on single file descriptor. Change write to fwrite.
Functions which use FILE are buffered. Functions which use file descriptors are not. This is why you may get mixed order.
You can also try calling fflush before write.
When you write onto the same file, or pipe, or whatever by two means at once (direct IO and output stream) you can get this behaviour. The reason is that the output stream is buffered.
With fflush() you can control that behaviour.
What is happening is that printf writes to stdout in a buffered way -- the string is kept in a buffer before being output -- while the 'write' later on writes to stdout unbuffered. This can have the effect that the output from 'write' appears first if the buffer from the printf is only flushed later on.
You can explicitly flush using fflush() -- but even better would be not to mix buffered and non-buffered writes to the same output. Type man printf, man fflush, man fwrite etc. on your terminal to learn more about what these commands do exactly.