I'm trying to understand how standard streams work in Linux, specifically how I can capture say, stdout directly from my terminal with something like tail. Since everything in Linux is a file, isn't it possible to simply do a tail -f /dev/stdout?
So to test this, I wrote a trivial program:
int main(int argc, char * argv[]) {
while (1) {
printf("This takes advantage of stdout\n");
sleep(1);
}
return 0;
}
Then in a separate terminal I did a tail -f stdout, but nothing is printed. Am I doing something wrong?
Each process has its own stdout. So you need to capture the standard output of some running process. The usual way is to make a pipeline, but you may also redirect the stdout to some file.
Learn more about tee(1). Read some bash scripting tutorial
A typical use might be thru batch(1) and a here document; I often do
batch << EOJ
make >& _make.out
EOJ
the >& is a bash-ism or a zsh-ism redirecting both stdout and stderr.
Then in a separate terminal -or even in the same terminal, after the batch sequence above- (in the same current directory)
tail -f _make.out
This enables me to see the progression of e.g. a long-lasting compilation (of course I can interrupt with Ctrl C the tail -f command without harming the compilation). YMMV.
Read also advanced bash scripting guide
Actually, /dev/stdout (notice that stdout alone means nothing particular to the shell) is a symlink to /proc/self/fd/1 . Read proc(5)
BTW, in your C code, take the habit of calling fflush(3) before any important syscall like sleep or fork (your code don't need it because stdout on terminal is line buffered but might be differently buffered when stdout is not a tty). Read stdout(3). Terminals are weird beasts, read tty demystified
addenda
To test in your C or C++ code if stdout is a terminal, use isatty(3), i.e.
if (isatty(STDOUT_FILENO))
stdout_is_a_terminal();
Notice that stdout could be a pipe or a file (then of course the above test would fail).
To write to the terminal, use /dev/tty (see tty(4)). To write on the console use /dev/console (see console(4)...). For system logging, learn about syslog(3) and e.g. rsyslogd(8).
Related
I was tasked with creating a test program in C that reads the contents of the standard input and then prints them.
But I have a little doubt: what is exactly standard input?
Is it what I type in the keyboard? Is it a file I have to read?
Both of them?
And the same goes for standard output: is it the console? a file?
The C standard (e.g. C99 or C11) defines what should be expected from the standard <stdio.h> header (after having suitably #include-d it). See stdio(3) man page.
Then you have the stdin and stdout and stderr file handles (pointers to some FILE which is an abstract data type).
The fact that stdin is related to some device (e.g. a keyboard) is implementation specific.
You could (but that would be unethical and/or inefficient) implement the C standard with e.g. a room of human slaves (that is unethical, if you use paid workers that would be just inefficient), instead of using a computer. Often, computers gives your some implementation of the C standard thru the help of some operating system.
You may want to know, inside your C program, if stdin is a "keyboard" or redirected from some "file". Unfortunately, AFAIK, there is no C99-standard way to know that.
As you mention, stdin, stdout and stderr should be available in your program at startup (i.e. after entering main ....). Hence, unless you fclose the stdin stream, you can read it (with getchar, scanf, getline, fgets, fscanf ... and friends) without any prior care (so you don't need to fopen it yourself).
On Linux or most Posix systems, you might use as an approximation isatty(STDIN_FILENO) - see isatty(3) for more - to test if stdin "is" the "keyboard" (by testing if it is some tty). See also this & that.
Yes, standard input (stdin) is input exepected from the keyboard. So, could be in the form of user input from a basic program or from a command line argument. Standard output (stdout) is the output of the code, usually to the terminal window. You could output your code almost anywhere, i.e. to a file, to a textbox, browser, but the standard is the stdout which is the terminal.
Hope that helps.
Normally, the standard input is the keyboard and the standard output the screen. However, you can redirect this in the command line using the "<" and ">" symbols. A command line like
dir /s > "Tree.txt"
will change the standard output for the dir command to be the specified file. So all output goes to that file. The called application or command itself doesn't normally even notice the difference.
stdin is file descriptor 0, you can get a file to stdin by:
cat file |yourprog
#or
yourprog <file
likewise for stdout (file descriptor 1)
yourprog | someotherprog #pipe your stdout to the stdin of another program
yourprog > somefile #save stdout to a file
yourprog >> somefile #append stdout to a file
and stderr (fd 2)
yourprog 2> errlogfile
if you have a program that takes a file but doesn't handle stdin, you can use the above formats by doing this (assuming -f if the input file argument)
myprog -f /dev/stdin
//and a horrible example of how not to read from stdin and write to stdout
char buf[4096];
while(write(1,buf,read(0,buf,4096)));
standard output (or stdout) refers to the standardized streams of data that are produced by command line programs (i.e., all-text mode programs) in Linux and other Unix-like operating systems.
How do I redirect stderr (or stdout+stderr) to a file if I don't know which shell (bash, csh, dash) is interpreting my command?
My C code running on Linux/FreeBSD/OSX needs to call an external program via the system() function, which will use /bin/sh to interpret the supplied command line. I would like to capture the messages printed by that external program to stderr and save them to a file. The problem is that on different systems /bin/sh points to different shells that have different syntax for redirecting the stderr stream to a file.
The closest thing I found is that bash actually understands the csh-style syntax for redirecting stderr+stdout to a file:
some_program >& output.txt
but dash, which is the default shell on Ubuntu (i.e. very common), does not understand this syntax.
Is there a syntax for stderr redirection that would be correctly interpreted by all common shells? Alternatively, is there a way to tell system() (or some other similar C function?) to use /usr/bin/env bash instead of /bin/sh to interpret the supplied command line?
You have a mistaken assumption, that /bin/sh can be an "alternate" shell like csh that's incompatible with the standard shell syntax. If you had a system setup like that, it would be unusably broken; no shell scripts would work. Pretty much all modern systems attempt to conform, at least superficially, to the POSIX standard, where the sh command processes the Shell Command Language specified in POSIX, which is roughly equivalent to the historical Bourne shell and which bash, dash, ash, etc. (shells which are commonly installed as /bin/sh) are all 99.9% compatible with.
You can completely ignore csh and similar. They're never installed as sh, and only folks who actually want to use them, or who get stuck using them as their interactive shell because some evil sysadmin setup the login shell defaults that way, ever have to care about them.
On any POSIX-like system, you can use
system("some_program > output.txt 2>&1");
This is because POSIX system is equivalent to calling sh, and POSIX sh supports this kind of redirection. This works independently of whether or not a user opening a terminal on the system will see a Csh prompt.
How do I redirect stderr (or stdout+stderr) to a file if I don't know which shell (bash, csh, dash) is interpreting my command?
You don't. Bourne-family shells and csh-family shells have different, incompatible syntax for redirecting stderr. In fact, csh and tcsh do not have a syntax to redirect only stderr at all -- they can redirect it only together with stdout.
If you really could be in any shell at all, then you're pretty much hosed with respect to doing much of anything. One could imagine an obscure, esoteric shell with completely incompatible syntax. For that matter, even an unusual configuration of a standard shell could trip you up -- for example if the IFS variable is set to an unusual value in a Bourne-family shell, then you'll have trouble executing any commands that don't take that into account.
If you can count on executing at least simple commands, then you could execute a known shell within the unknown one to process your command, but that oughtn't to be necessary for the case that seems to interest you.
Alternatively, is there a way to tell system() (or some other similar
C function?) to use /usr/bin/env bash instead of /bin/sh to interpret
the supplied command line?
Not on a POSIX-conforming system. POSIX specifies explicitly that the system() function executes the command by use of /bin/sh -c [the_command]. But this shouldn't be something to worry about, as /bin/sh should be a conforming POSIX shell, or at least pretty close to one. Definitely it should be a Bourne-family shell, which both bash and dash are, but tcsh most definitely is not.
The way to redirect the standard error stream in a POSIX shell is to use the 2> redirection operator (which is a special case of a more general redirection feature applicable to any file descriptor). Whatever shell /bin/sh actually is should recognize that syntax, and in particular bash and dash both do:
some_program 2> output.txt
I think, there is another possibility worth mentioning: You could open the file you want to redirect on stderr in your c-code prior to calling system(). You can dup() the original stderr first, and then restore it again.
fflush(stderr); // Flush pending output
int saved_stderr = dup(fileno(stderr));
int fd = open("output.txt", O_RDWR|O_CREAT|O_TRUNC, 0600);
dup2(fd, fileno(stderr));
close(fd);
system("some_program");
dup2(saved_stderr, fileno(stderr));
close(saved_stderr);
This should perform the output redirection as you need it.
If you don't know the shell.... of course you don't know how to redirect from it, despite of the fact that you can see what value the $SHELL has, and act in consequence:
char *shell = getenv("SHELL");
if (*shell) { /* no SHELL variable defined */
/* ... */
} else if (!strcmp(shell, "/bin/sh")) { /* bourne shell */
/* ... */
} /* ... more shells */
Despite of what you say in your question, it is quite unusual to rename /bin/sh to use another shell, as shell scripts use syntax that depends on that. The only case I know is with bash(1), and I have seen this only in Linux (and remarkably, last versions of solaris), but the syntax of bash(1) is a superset of the syntax of sh(1), making it possible to run shell scripts made for sh(1) with it. Renaming /bin/sh to perl for example, would make your system probably completely unusable, as many system tools depend of /bin/sh to be a bourne compatible shell.
By the way, the system(3) library function always calls sh(1) as the command interpreter, so there should be no problem to use it, but there's no solution to capture the output and process it by the parent process (indeed, the parent process is the sh(1) that system(3) fork(2)s)
Another thing you can do is to popen(3) a process. This call gives you a FILE pointer to a pipe of a process. You popen its input in case you popen(3) it for writing, and you popen its output if you want or read its output. Look at the manual for details, as I don't know now if it redirects only its standard output or it also redirects the standard error (I think only redirects standard output, for reasons discussed below, and only if you popen(3) it with a "r" flag).
FILE *f_in = popen("ps aux", "r");
/* read standard output of 'ps aux' command. */
pclose(f_in); /* closes the descriptor and waits for the child to finish */
Another thing you can do is to redirect yourself after fork(2)ing the child, and before the exec(2) call (this way you can decide if you want only stdout or if you want also stderr redirected back to you):
int fd[2];
int res = pipe(fd);
if (res < 0) {
perror("pipe");
exit(EXIT_FAILURE);
}
if ((res = fork()) < 0) {
perror("fork");
exit(EXIT_FAILURE);
} else if (res == 0) { /* child process */
dup2(fd[1], 1); /* redirect pipe to stdout */
dup2(fd[1], 2); /* redirect pipe also to stderr */
close(fd[1]); close(fd[0]); /* we don't need these */
execvp(program, argv);
perror("execvp");
exit(EXIT_FAILURE);
} else { /* parent process */
close(fd[1]); /* we are not going to write in the pipe */
FILE *f_in = fdopen(fd[0]);
/* read standard output and standard error from program from f_in FILE descriptor */
fclose(f_in);
wait(NULL); /* wait for child to finish */
}
You can see a complete example of this (not reading standard error, but it is easy to add --- you have only to add the second dup2() call from above) here. The program executes repeatedly a command you pass to it on the command line. It needs to get access to the output of the subprocess to count the lines, as between invocations, the program goes up as many lines as the program output, to make the next invocation to overlap the output of the last invocation. You can try it and play, making modifications as you like.
NOTE
In your sample redirection, when you use >&, you need to add a number after the ampersand, to indicate which descriptor you are dup()ing. As the number before the > is optional, the one after the & is mandatory. So, if you have not used it, prepare to receive an error (which probably you don't see if you are redirecting stderr) The idea of having two separate output descriptors is to allow you to redirect stdout and at the same time, conserve a channel where to put error messages.
I was tasked with creating a test program in C that reads the contents of the standard input and then prints them.
But I have a little doubt: what is exactly standard input?
Is it what I type in the keyboard? Is it a file I have to read?
Both of them?
And the same goes for standard output: is it the console? a file?
The C standard (e.g. C99 or C11) defines what should be expected from the standard <stdio.h> header (after having suitably #include-d it). See stdio(3) man page.
Then you have the stdin and stdout and stderr file handles (pointers to some FILE which is an abstract data type).
The fact that stdin is related to some device (e.g. a keyboard) is implementation specific.
You could (but that would be unethical and/or inefficient) implement the C standard with e.g. a room of human slaves (that is unethical, if you use paid workers that would be just inefficient), instead of using a computer. Often, computers gives your some implementation of the C standard thru the help of some operating system.
You may want to know, inside your C program, if stdin is a "keyboard" or redirected from some "file". Unfortunately, AFAIK, there is no C99-standard way to know that.
As you mention, stdin, stdout and stderr should be available in your program at startup (i.e. after entering main ....). Hence, unless you fclose the stdin stream, you can read it (with getchar, scanf, getline, fgets, fscanf ... and friends) without any prior care (so you don't need to fopen it yourself).
On Linux or most Posix systems, you might use as an approximation isatty(STDIN_FILENO) - see isatty(3) for more - to test if stdin "is" the "keyboard" (by testing if it is some tty). See also this & that.
Yes, standard input (stdin) is input exepected from the keyboard. So, could be in the form of user input from a basic program or from a command line argument. Standard output (stdout) is the output of the code, usually to the terminal window. You could output your code almost anywhere, i.e. to a file, to a textbox, browser, but the standard is the stdout which is the terminal.
Hope that helps.
Normally, the standard input is the keyboard and the standard output the screen. However, you can redirect this in the command line using the "<" and ">" symbols. A command line like
dir /s > "Tree.txt"
will change the standard output for the dir command to be the specified file. So all output goes to that file. The called application or command itself doesn't normally even notice the difference.
stdin is file descriptor 0, you can get a file to stdin by:
cat file |yourprog
#or
yourprog <file
likewise for stdout (file descriptor 1)
yourprog | someotherprog #pipe your stdout to the stdin of another program
yourprog > somefile #save stdout to a file
yourprog >> somefile #append stdout to a file
and stderr (fd 2)
yourprog 2> errlogfile
if you have a program that takes a file but doesn't handle stdin, you can use the above formats by doing this (assuming -f if the input file argument)
myprog -f /dev/stdin
//and a horrible example of how not to read from stdin and write to stdout
char buf[4096];
while(write(1,buf,read(0,buf,4096)));
standard output (or stdout) refers to the standardized streams of data that are produced by command line programs (i.e., all-text mode programs) in Linux and other Unix-like operating systems.
int main(void)
{
char buf[] = "standard err, output.\n";
printf("standard output.\n");
if (write(STDERR_FILENO,buf, 22) != 22)
printf("write err!\n");
exit(0);
}
Compile using:
gcc -Wall text.c
Then running in the shell:
./a.out > outfile 2 >& 1
Result：outfile´s content are：
standard err, output.
standard output.
./a.out 2 >& 1 >outfile
Result：
This first prints to the terminal: standard err, output.
and the content of outfile are: standard output.
Questions:
I want to ask the difference between 2 >& fd and 2 > file.
Are they all equal to the function dup()?
Another question: why are the contents of outfile:
standard err, output.
standard output.
I expected the content of outfile to be:
standard output.
standard err, output
Actually, in bash, >& is quite similar to dup2. That is, the file descriptor to which it is applied will refer to the same file as the descriptor to the right. So:
$ ./a.out > outfile 2>& 1
It will redirect stdout(1) to the file outfile and, after that, will dup2 stderr(2) to refer to the same file as stdout(1). That is, both stdout and stderr are being redirected to the file.
$ ./a.out 2>& 1 >outfile
It will redirect stderr(2) to refer to the same file as stdout(1), that is, the console, and after that, will redirect stdout(1) to refer to the file outfile. That is, stderr will output to the console and stdout to the file.
And that's exactly what you are getting.
Paradigm Mixing
While there are reasons to do all of these things deliberately, as a learning experience it is probably going to be confusing to mix operations over what I might call "domain boundaries".
Buffered vs non-buffered I/O
The printf() is buffered, the write() is a direct system call. The write happens immediately no matter what, the printf will be (usually) buffered line-by-line when the output is a terminal and block-by-block when the output is a real file. In the file-output case (redirection) your actual printf output will happen only when you return from main() or in some other fashion call exit(3), unless you printf a whole bunch of stuff.
Historic csh redirection vs bash redirection
The now-forgotten (but typically still in a default install) csh that Bill Joy wrote at UCB while a grad student had a few nice features that have been imported into kitchen-sink shells that OR-together every shell feature ever thought of. Yes, I'm talking about bash here. So, in csh, the way to redirect both standard output and standard error was simply to say cmd >& file which was really more civilized that the bag-of-tools approach that the "official" Bourne shell provided. But the Bourne syntax had its good points elsewhere and in any case survived as the dominant paradigm.
But the bash "native" redirection features are somewhat complex and I wouldn't try to summarize them in a SO answer, although others seem to have made a good start. In any case you are using real bash redirection in one test and the legacy-csh syntax that bash also supports in another, and with a program that itself mixes paradigms. The main issue from the shell's point of view is that the order of redirection is quite important in the bash-style syntax while the csh-style syntax simply specifies the end result.
There are several loosely related issues here.
Style comment: I recommend using 2>&1 without spaces. I wasn't even aware that the spaced-out version works (I suspect it didn't in Bourne shell in the mid-80s) and the compressed version is the orthodox way of writing it.
The file-descriptor I/O redirection notations are not all available in the C shell and derivatives; they are avialable in Bourne shell and its derivatives (Korn shell, POSIX shell, Bash, ...).
The difference between >file or 2>file and 2>&1 is what the shell has to do. The first two arrange for output written to a file descriptor (1 in the first case, aka standard output; 2 in the second case, aka standard error) to go to the named file. This means that anything written by the program to standard output goes to file instead. The third notation arranges for 2 (standard error) to go to the same file descriptor as 1 (standard output); anything written to standard error goes to the same file as standard output. It is trivially implemented using dup2(). However, the standard error stream in the program will have its own buffer and the standard output stream in the program will have its own buffer, so the interleaving of the output is not completely determinate if the output goes to a file.
You run the command two different ways, and (not surprisingly) get two different results.
./a.out > outfile 2>&1
I/O redirections are processed left to right. The first one sends standard output to outfile. The second sends standard error to the same place as standard output, so it goes to outfile too.
./a.out 2>&1 >outfile
The first redirection sends standard error to the place where standard output is going, which is currently the terminal. The second redirection then sends standard output to the file (but leaves standard error going to the terminal).
The program uses the printf() function and the write() system call. When the printf() function is used, it buffers its output. If the output is going to a terminal, then it is normally 'line buffered', so output appears when a newline is added to the buffer. However, when the output is going to a file, it is 'fully buffered' and output does not appear until the file stream is flushed or closed or the buffer fills. Note that stderr is not fully buffered, so output written to it appears immediately.
If you run your program without any I/O redirection, you will see:
standard output.
standard err, output
By contrast, the write() system call immediately transfers data to the output file descriptor. In the example, you write to standard error, and what you write will appear immediately. The same would have happened if you had used fprintf(stderr, ...). However, suppose you modified the program to write to STDOUT_FILENO; then when the output is to a file, the output would appear in the order:
standard err, output
standard output.
because the write() is unbuffered while the printf() is buffered.
The 2>&1 part makes the shell do something like that:
dup2(1, 2);
This makes fd 2 a "copy" of fd 1.
The 2> file is interpreted as
fd = open(file, ...);
dup2(fd, 2);
which opens a file and puts the filedescriptor into slot 2.
I am trying to redirect the output of a c program to file, even when it generates some errors because of problems with the input data. I can send the output but the error messages to a file.
Does somebody know how to do it?
From within C source code, you can redirect outputs using freopen():
General outputs:
freopen("myfile.txt", "w", stdout);
Errors:
freopen("myfile_err.txt", "w", stderr);
(This answer applies to bash shell, and similar flavors. You didn't specify your environment and this sort of question needs that detail.)
I assume you know about basic redirection with ">". To also capture STDERR in addition to STDOUT, use the following syntax:
command > file-name 2>&1
For some more background on standard streams and numbers:
http://en.wikipedia.org/wiki/Standard_streams#Standard_input_.28stdin.29
This depends on what you mean and what platform you are using. Very often you can accomplish this from the command line, which has been covered in another answer. If you use this method to accomplish this you should be aware that FILE * stderr is typically written immediately (unbuffered) while FILE * stdout may be buffered (usually line buffered) so you could end up with some of your error messages appearing to have been printed earlier than some other messages, but actually the other messages are just being printed late.
From within a C program you can also do something similar within the stdio system using freopen, which will effect the FILE *, so you could make fprintf(stderr, "fungus"); print to something besides what stderr normally would print to.
But if you want to know how to make a program redirect the actual file descriptors under a unix like system you need to learn about the dup and dup2 system calls. They allow you to duplicate a file descriptor.
int fd = open("some_file", O_WRONLY);
dup2(2,fd);
close(fd);
This code will make "some_file" the new stderr at the OS level. The dup2 call will close and replace file descriptor 2 (stderr, which is usually used by FILE * stderr but not necessarily if you call freopen(x,y,stderr) since that may make FILE *stderr use a different file descriptor).
This is how shell programs redirect input and output of programs. The open all of the files that the new program will need, fork, then the child uses dup2 to set up the files descriptors for the new program, then it closes any files that the new program won't need (usually just leaving 0, 1, and 2 open), and then uses one of the exec functions to become the program that the shell was told to run. (some of this isn't entirely accurate because some shells may rely on close on exe flags)
Using a simple linux command you can save the output into the file. here is a simple linux terminal command.
ls > file.txt
The output of this command will be stored into the file.
same as you can store the output of the program like this suppose, object file name is a, run the following command to save output in a file:
./a > file.txt