I wrote this piece of code that is supposed to redirect something written on the STDOUT by a function to the STDIN so that it can be read by another function. I cannot access these functions, so this is the only way I can use them.
mpz_fput(stdout, c) is one of these function. It just prints on the STDOUT something contained in the c data structure.
Now everything worked fine while debugging as before the following code I had a printf(); followed by a fflush(stdout); (needed to print debugging messages).
Now that I removed these two lines I noticed (using gdb) that this code stays idle on the read() function (last line of this piece of code)
char buffer[BUFSIZ] = "";
int out_pipe[2];
int in_pipe[2];
int saved_stdout;
int saved_stdin;
int errno;
// REDIRECT STDIN
saved_stdin = dup(STDIN_FILENO); /* save stdin for later */
if(errno= pipe(in_pipe) != 0 ) { /* make a pipe */
printf("\n%s",strerror(errno));
exit(1);
}
close(STDIN_FILENO);
dup2(in_pipe[0], STDIN_FILENO); /* redirect pipe to stdin */
// REDIRECT STDOUT
saved_stdout = dup(STDOUT_FILENO); /* save stdout for display later */
if(errno= pipe(out_pipe) != 0 ) { /* make a pipe */
printf("\n%s",strerror(errno));
exit(1);
}
dup2(out_pipe[1], STDOUT_FILENO); /* redirect stdout to the pipe */
close(out_pipe[1]);
mpz_fput(stdout,c); // put c on stdout
read(out_pipe[0], buffer, BUFSIZ); // read c from stdout pipe into buffer
any idea why is that?
Seems you used the blocking type. In this case, out_pipe[0] is a blocking handle. So read() blocked, and waiting for anything out from out_pipe[0].
Besides, I think there's something to do with the fflush():
For output streams, fflush() forces a write of all user-space buffered data
for the given output or update stream via the stream's underlying write
function. For input streams, fflush() discards any buffered data that has
been fetched from the underlying file, but has not been by the application.
The open status of the stream is unaffected.
In your case, you redirected pipe to STDOUT, then called fflush() to make everything in STDOUT flushed and move them to read() buffer. Then you called read() to read them out. If you didn't call fflush(), and read() buffer would be empty. Since it's a blocking handle used by read(), you can't read anything from the buffer, so it will be blocked.
This is the brief theory, I suggest you to read Linux manpage for more details.
Related
int main()
{
int fd=open("/dev/pts/0",O_RDWR);
if(fd==-1){
printf("Error");
exit(1);
}
dup2(fd,0);
char c[20];
printf("reading from file\n");
scanf("%s",c);
}
In above code "/dev/pts/0" is set as stdin.scanf behaves normally.
But when I set to filename like "inp.txt" it doesn't wait directly reads whatever it finds.
Why is that like that?What to do if I want to make it wait?
When you read from a file, the data is already there, so why would scanf() wait? Or any other way of reading the file?
/dev/pts/<i>N</i> are Unix 98 pseudoterminals, which by their very nature are interactive. A blocking read from one waits for interactive input. A nonblocking read would just tell you that there is no data to read right now.
If you create a pipe between processes, associate the read end with an stdio FILE handle via fdopen(), you can use scanf() to scan data from the pipe. That, too, will wait for input, unless all write ends of the pipe are closed (then the scanf will fail with end-of-input). So, there is nothing special about the pseudoterminals in this respect.
the code as follows:
int main(int argc, char **argv)
{
const char *file = "/tmp/out";
int fd = open(file, O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU);
int stdout_tmp = dup(1);
close(1);
dup(fd);
puts("hello world!");
// fflush(stdout);
close(1);
close(fd);
dup(stdout_tmp);
puts("redirect completed!");
exit(0);
}
I compile the code succssfully without any warning using gcc10.2.0, Againest my expectation, the two line both are output to the stdout instead "hello world" in the /tmp/out file and "redirect completed!" in stdout. When uncomment the fflush(stdout), it works!
I guess that the puts() doesn't refresh the buffer in user space, after restoring the stdout and exit, the buffer is auto refreshed.
gets() output string with trailing '\n' and the stdout buffer will be auto refreshed when encounter '\n'. Why need to call fflush(stdout) manually?
The stdout file (where puts writes) is line-buffered (i.e. flushes the buffer on newline) only when connected to a terminal (basically when isatty(fileno(stdout)) is true).
When connected to another non-terminal output then it's fully buffered, which means you either need to fill the buffer completely, or call fflush explicitly to flush the buffer.
man 3 setvbuf says:
Normally all files are block buffered.
If a stream refers to a terminal (as stdout normally does), it is line buffered.
Since puts() uses stdout we should expect a flush
(because of the \n).
However, since in your example stdout has not been used before
the redirection, I guess that the buffering behaviour is not chosen
yet.
At your first write attempt to stdout, the underlying file descriptor
is not a terminal anymore but a regular file.
I guess the buffering behaviour is chosen at this moment.
If you add another call to puts() in your example before the
redirection, then the buffering behaviour is chosen for the terminal
and then does not change afterwards when the redirection is performed.
In this case, your example works as you expect (without the explicit
fflush()).
edit
Still in man 3 setvbuf:
When the first I/O operation occurs on a file, malloc(3) is called, and a buffer is obtained.
and further:
The setvbuf() function may only be used after opening a stream and before any other operations have been performed on it.
On linux, this is consistent with your example.
In the MSDN page for setvbuf:
stream must refer to an open file that has not undergone an I/O operation since it was opened.
The behaviour of printf() seems to depend on the location of stdout.
If stdout is sent to the console, then printf() is line-buffered and is flushed after a newline is printed.
If stdout is redirected to a file, the buffer is not flushed unless fflush() is called.
Moreover, if printf() is used before stdout is redirected to file, subsequent writes (to the file) are line-buffered and are flushed after newline.
When is stdout line-buffered, and when does fflush() need to be called?
Minimal example of each:
void RedirectStdout2File(const char* log_path) {
int fd = open(log_path, O_RDWR|O_APPEND|O_CREAT,S_IRWXU|S_IRWXG|S_IRWXO);
dup2(fd,STDOUT_FILENO);
if (fd != STDOUT_FILENO) close(fd);
}
int main_1(int argc, char* argv[]) {
/* Case 1: stdout is line-buffered when run from console */
printf("No redirect; printed immediately\n");
sleep(10);
}
int main_2a(int argc, char* argv[]) {
/* Case 2a: stdout is not line-buffered when redirected to file */
RedirectStdout2File(argv[0]);
printf("Will not go to file!\n");
RedirectStdout2File("/dev/null");
}
int main_2b(int argc, char* argv[]) {
/* Case 2b: flushing stdout does send output to file */
RedirectStdout2File(argv[0]);
printf("Will go to file if flushed\n");
fflush(stdout);
RedirectStdout2File("/dev/null");
}
int main_3(int argc, char* argv[]) {
/* Case 3: printf before redirect; printf is line-buffered after */
printf("Before redirect\n");
RedirectStdout2File(argv[0]);
printf("Does go to file!\n");
RedirectStdout2File("/dev/null");
}
Flushing for stdout is determined by its buffering behaviour. The buffering can be set to three modes: _IOFBF (full buffering: waits until fflush() if possible), _IOLBF (line buffering: newline triggers automatic flush), and _IONBF (direct write always used). "Support for these characteristics is implementation-defined, and may be affected via the setbuf() and setvbuf() functions." [C99:7.19.3.3]
"At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device." [C99:7.19.3.7]
Explanation of observed behaviour
So, what happens is that the implementation does something platform-specific to decide whether stdout is going to be line-buffered. In most libc implementations, this test is done when the stream is first used.
Behaviour #1 is easily explained: when the stream is for an interactive device, it is line-buffered, and the printf() is flushed automatically.
Case #2 is also now expected: when we redirect to a file, the stream is fully buffered and will not be flushed except with fflush(), unless you write gobloads of data to it.
Finally, we understand case #3 too for implementations that only perform the check on the underlying fd once. Because we forced stdout's buffer to be initialised in the first printf(), stdout acquired the line-buffered mode. When we swap out the fd to go to file, it's still line-buffered, so the data is flushed automatically.
Some actual implementations
Each libc has latitude in how it interprets these requirements, since C99 doesn't specify what an "interactive device" is, nor does POSIX's stdio entry extend this (beyond requiring stderr to be open for reading).
Glibc. See filedoalloc.c:L111. Here we use stat() to test if the fd is a tty, and set the buffering mode accordingly. (This is called from fileops.c.) stdout initially has a null buffer, and it's allocated on the first use of the stream based on the characteristics of fd 1.
BSD libc. Very similar, but much cleaner code to follow! See this line in makebuf.c
Your are wrongly combining buffered and unbuffered IO functions. Such a combination must be done very carefully especially when the code has to be portable. (and it is bad to write unportable code...)
It is certainly best to avoid combining buffered and unbuffered IO on the same file descriptor.
Buffered IO: fprintf(), fopen(), fclose(), freopen()...
Unbuffered IO: write(), open(), close(), dup()...
When you use dup2() to redirect stdout. The function is not aware of the buffer which was filled by fprintf(). So when dup2() closes the old descriptor 1 it does not flush the buffer and the content could be flushed to a different output. In your case 2a it was sent to /dev/null.
The solution
In your case it is best to use freopen() instead of dup2(). This solves all your problems:
It flushes the buffers of the original FILE stream. (case 2a)
It sets the buffering mode according to the newly opened file. (case 3)
Here is the correct implementation of your function:
void RedirectStdout2File(const char* log_path) {
if(freopen(log_path, "a+", stdout) == NULL) err(EXIT_FAILURE, NULL);
}
Unfortunately with buffered IO you cannot directly set permissions of a newly created file. You have to use other calls to change the permissions or you can use unportable glibc extensions. See the fopen() man page.
You should not close the file descriptor, so remove close(fd) and close
stdout_bak_fd if you want the message to be print only in the file.
The behaviour of printf() seems to depend on the location of stdout.
If stdout is sent to the console, then printf() is line-buffered and is flushed after a newline is printed.
If stdout is redirected to a file, the buffer is not flushed unless fflush() is called.
Moreover, if printf() is used before stdout is redirected to file, subsequent writes (to the file) are line-buffered and are flushed after newline.
When is stdout line-buffered, and when does fflush() need to be called?
Minimal example of each:
void RedirectStdout2File(const char* log_path) {
int fd = open(log_path, O_RDWR|O_APPEND|O_CREAT,S_IRWXU|S_IRWXG|S_IRWXO);
dup2(fd,STDOUT_FILENO);
if (fd != STDOUT_FILENO) close(fd);
}
int main_1(int argc, char* argv[]) {
/* Case 1: stdout is line-buffered when run from console */
printf("No redirect; printed immediately\n");
sleep(10);
}
int main_2a(int argc, char* argv[]) {
/* Case 2a: stdout is not line-buffered when redirected to file */
RedirectStdout2File(argv[0]);
printf("Will not go to file!\n");
RedirectStdout2File("/dev/null");
}
int main_2b(int argc, char* argv[]) {
/* Case 2b: flushing stdout does send output to file */
RedirectStdout2File(argv[0]);
printf("Will go to file if flushed\n");
fflush(stdout);
RedirectStdout2File("/dev/null");
}
int main_3(int argc, char* argv[]) {
/* Case 3: printf before redirect; printf is line-buffered after */
printf("Before redirect\n");
RedirectStdout2File(argv[0]);
printf("Does go to file!\n");
RedirectStdout2File("/dev/null");
}
Flushing for stdout is determined by its buffering behaviour. The buffering can be set to three modes: _IOFBF (full buffering: waits until fflush() if possible), _IOLBF (line buffering: newline triggers automatic flush), and _IONBF (direct write always used). "Support for these characteristics is implementation-defined, and may be affected via the setbuf() and setvbuf() functions." [C99:7.19.3.3]
"At program startup, three text streams are predefined and need not be opened explicitly
— standard input (for reading conventional input), standard output (for writing
conventional output), and standard error (for writing diagnostic output). As initially
opened, the standard error stream is not fully buffered; the standard input and standard
output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device." [C99:7.19.3.7]
Explanation of observed behaviour
So, what happens is that the implementation does something platform-specific to decide whether stdout is going to be line-buffered. In most libc implementations, this test is done when the stream is first used.
Behaviour #1 is easily explained: when the stream is for an interactive device, it is line-buffered, and the printf() is flushed automatically.
Case #2 is also now expected: when we redirect to a file, the stream is fully buffered and will not be flushed except with fflush(), unless you write gobloads of data to it.
Finally, we understand case #3 too for implementations that only perform the check on the underlying fd once. Because we forced stdout's buffer to be initialised in the first printf(), stdout acquired the line-buffered mode. When we swap out the fd to go to file, it's still line-buffered, so the data is flushed automatically.
Some actual implementations
Each libc has latitude in how it interprets these requirements, since C99 doesn't specify what an "interactive device" is, nor does POSIX's stdio entry extend this (beyond requiring stderr to be open for reading).
Glibc. See filedoalloc.c:L111. Here we use stat() to test if the fd is a tty, and set the buffering mode accordingly. (This is called from fileops.c.) stdout initially has a null buffer, and it's allocated on the first use of the stream based on the characteristics of fd 1.
BSD libc. Very similar, but much cleaner code to follow! See this line in makebuf.c
Your are wrongly combining buffered and unbuffered IO functions. Such a combination must be done very carefully especially when the code has to be portable. (and it is bad to write unportable code...)
It is certainly best to avoid combining buffered and unbuffered IO on the same file descriptor.
Buffered IO: fprintf(), fopen(), fclose(), freopen()...
Unbuffered IO: write(), open(), close(), dup()...
When you use dup2() to redirect stdout. The function is not aware of the buffer which was filled by fprintf(). So when dup2() closes the old descriptor 1 it does not flush the buffer and the content could be flushed to a different output. In your case 2a it was sent to /dev/null.
The solution
In your case it is best to use freopen() instead of dup2(). This solves all your problems:
It flushes the buffers of the original FILE stream. (case 2a)
It sets the buffering mode according to the newly opened file. (case 3)
Here is the correct implementation of your function:
void RedirectStdout2File(const char* log_path) {
if(freopen(log_path, "a+", stdout) == NULL) err(EXIT_FAILURE, NULL);
}
Unfortunately with buffered IO you cannot directly set permissions of a newly created file. You have to use other calls to change the permissions or you can use unportable glibc extensions. See the fopen() man page.
You should not close the file descriptor, so remove close(fd) and close
stdout_bak_fd if you want the message to be print only in the file.
I have a C application with many worker threads. It is essential that these do not block so where the worker threads need to write to a file on disk, I have them write to a circular buffer in memory, and then have a dedicated thread for writing that buffer to disk.
The worker threads do not block any more. The dedicated thread can safely block while writing to disk without affecting the worker threads (it does not hold a lock while writing to disk). My memory buffer is tuned to be sufficiently large that the writer thread can keep up.
This all works great. My question is, how do I implement something similar for stdout?
I could macro printf() to write into a memory buffer, but I don't have control over all the code that might write to stdout (some of it is in third-party libraries).
Thoughts?
NickB
I like the idea of using freopen. You might also be able to redirect stdout to a pipe using dup and dup2, and then use read to grab data from the pipe.
Something like so:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define MAX_LEN 40
int main( int argc, char *argv[] ) {
char buffer[MAX_LEN+1] = {0};
int out_pipe[2];
int saved_stdout;
saved_stdout = dup(STDOUT_FILENO); /* save stdout for display later */
if( pipe(out_pipe) != 0 ) { /* make a pipe */
exit(1);
}
dup2(out_pipe[1], STDOUT_FILENO); /* redirect stdout to the pipe */
close(out_pipe[1]);
/* anything sent to printf should now go down the pipe */
printf("ceci n'est pas une pipe");
fflush(stdout);
read(out_pipe[0], buffer, MAX_LEN); /* read from pipe into buffer */
dup2(saved_stdout, STDOUT_FILENO); /* reconnect stdout for testing */
printf("read: %s\n", buffer);
return 0;
}
If you're working with the GNU libc, you might use memory streams string streams.
You can "redirect" stdout into file using freopen().
man freopen says:
The freopen() function opens the file
whose name is the string pointed to
by path and associates the stream
pointed to by stream with it. The
original stream (if it exists) is
closed. The mode argument is used
just as in the fopen() function.
The primary use of the freopen()
function is to change the file
associated with a standard text
stream (stderr, stdin, or stdout).
This file well could be a pipe - worker threads will write to that pipe and writer thread will listen.
Why don't you wrap your entire application in another? Basically, what you want is a smart cat that copies stdin to stdout, buffering as necessary. Then use standard stdin/stdout redirection. This can be done without modifying your current application at all.
~MSalters/# YourCurrentApp | bufcat
You can change how buffering works with setvbuf() or setbuf(). There's a description here: http://publications.gbdirect.co.uk/c_book/chapter9/input_and_output.html.
[Edit]
stdout really is a FILE*. If the existing code works with FILE*s, I don't see what prevents it from working with stdout.
One solution ( for both things your doing ) would be to use a gathering write via writev.
Each thread could for example sprintf into a iovec buffer and then pass the iovec pointers to the writer thread and have it simply call writev with stdout.
Here is an example of using writev from Advanced Unix Programming
Under Windows you would use WSAsend for similar functionality.
The method using the 4096 bigbuf will only sort of work. I've tried this code, and while it does successfully capture stdout into the buffer, it's unusable in a real world case. You have no way of knowing how long the captured output is, so no way of knowing when to terminate the string '\0'. If you try to use the buffer you get 4000 characters of garbage spit out if you had successfully captured 96 characters of stdout output.
In my application, I'm using a perl interpreter in the C program. I have no idea how much output is going to be spit out of what ever document is thrown at the C program, and hence the code above would never allow me to cleanly print that output out anywhere.