How do I redirect stderr (or stdout+stderr) to a file if I don't know which shell (bash, csh, dash) is interpreting my command?
My C code running on Linux/FreeBSD/OSX needs to call an external program via the system() function, which will use /bin/sh to interpret the supplied command line. I would like to capture the messages printed by that external program to stderr and save them to a file. The problem is that on different systems /bin/sh points to different shells that have different syntax for redirecting the stderr stream to a file.
The closest thing I found is that bash actually understands the csh-style syntax for redirecting stderr+stdout to a file:
some_program >& output.txt
but dash, which is the default shell on Ubuntu (i.e. very common), does not understand this syntax.
Is there a syntax for stderr redirection that would be correctly interpreted by all common shells? Alternatively, is there a way to tell system() (or some other similar C function?) to use /usr/bin/env bash instead of /bin/sh to interpret the supplied command line?
You have a mistaken assumption, that /bin/sh can be an "alternate" shell like csh that's incompatible with the standard shell syntax. If you had a system setup like that, it would be unusably broken; no shell scripts would work. Pretty much all modern systems attempt to conform, at least superficially, to the POSIX standard, where the sh command processes the Shell Command Language specified in POSIX, which is roughly equivalent to the historical Bourne shell and which bash, dash, ash, etc. (shells which are commonly installed as /bin/sh) are all 99.9% compatible with.
You can completely ignore csh and similar. They're never installed as sh, and only folks who actually want to use them, or who get stuck using them as their interactive shell because some evil sysadmin setup the login shell defaults that way, ever have to care about them.
On any POSIX-like system, you can use
system("some_program > output.txt 2>&1");
This is because POSIX system is equivalent to calling sh, and POSIX sh supports this kind of redirection. This works independently of whether or not a user opening a terminal on the system will see a Csh prompt.
How do I redirect stderr (or stdout+stderr) to a file if I don't know which shell (bash, csh, dash) is interpreting my command?
You don't. Bourne-family shells and csh-family shells have different, incompatible syntax for redirecting stderr. In fact, csh and tcsh do not have a syntax to redirect only stderr at all -- they can redirect it only together with stdout.
If you really could be in any shell at all, then you're pretty much hosed with respect to doing much of anything. One could imagine an obscure, esoteric shell with completely incompatible syntax. For that matter, even an unusual configuration of a standard shell could trip you up -- for example if the IFS variable is set to an unusual value in a Bourne-family shell, then you'll have trouble executing any commands that don't take that into account.
If you can count on executing at least simple commands, then you could execute a known shell within the unknown one to process your command, but that oughtn't to be necessary for the case that seems to interest you.
Alternatively, is there a way to tell system() (or some other similar
C function?) to use /usr/bin/env bash instead of /bin/sh to interpret
the supplied command line?
Not on a POSIX-conforming system. POSIX specifies explicitly that the system() function executes the command by use of /bin/sh -c [the_command]. But this shouldn't be something to worry about, as /bin/sh should be a conforming POSIX shell, or at least pretty close to one. Definitely it should be a Bourne-family shell, which both bash and dash are, but tcsh most definitely is not.
The way to redirect the standard error stream in a POSIX shell is to use the 2> redirection operator (which is a special case of a more general redirection feature applicable to any file descriptor). Whatever shell /bin/sh actually is should recognize that syntax, and in particular bash and dash both do:
some_program 2> output.txt
I think, there is another possibility worth mentioning: You could open the file you want to redirect on stderr in your c-code prior to calling system(). You can dup() the original stderr first, and then restore it again.
fflush(stderr); // Flush pending output
int saved_stderr = dup(fileno(stderr));
int fd = open("output.txt", O_RDWR|O_CREAT|O_TRUNC, 0600);
dup2(fd, fileno(stderr));
close(fd);
system("some_program");
dup2(saved_stderr, fileno(stderr));
close(saved_stderr);
This should perform the output redirection as you need it.
If you don't know the shell.... of course you don't know how to redirect from it, despite of the fact that you can see what value the $SHELL has, and act in consequence:
char *shell = getenv("SHELL");
if (*shell) { /* no SHELL variable defined */
/* ... */
} else if (!strcmp(shell, "/bin/sh")) { /* bourne shell */
/* ... */
} /* ... more shells */
Despite of what you say in your question, it is quite unusual to rename /bin/sh to use another shell, as shell scripts use syntax that depends on that. The only case I know is with bash(1), and I have seen this only in Linux (and remarkably, last versions of solaris), but the syntax of bash(1) is a superset of the syntax of sh(1), making it possible to run shell scripts made for sh(1) with it. Renaming /bin/sh to perl for example, would make your system probably completely unusable, as many system tools depend of /bin/sh to be a bourne compatible shell.
By the way, the system(3) library function always calls sh(1) as the command interpreter, so there should be no problem to use it, but there's no solution to capture the output and process it by the parent process (indeed, the parent process is the sh(1) that system(3) fork(2)s)
Another thing you can do is to popen(3) a process. This call gives you a FILE pointer to a pipe of a process. You popen its input in case you popen(3) it for writing, and you popen its output if you want or read its output. Look at the manual for details, as I don't know now if it redirects only its standard output or it also redirects the standard error (I think only redirects standard output, for reasons discussed below, and only if you popen(3) it with a "r" flag).
FILE *f_in = popen("ps aux", "r");
/* read standard output of 'ps aux' command. */
pclose(f_in); /* closes the descriptor and waits for the child to finish */
Another thing you can do is to redirect yourself after fork(2)ing the child, and before the exec(2) call (this way you can decide if you want only stdout or if you want also stderr redirected back to you):
int fd[2];
int res = pipe(fd);
if (res < 0) {
perror("pipe");
exit(EXIT_FAILURE);
}
if ((res = fork()) < 0) {
perror("fork");
exit(EXIT_FAILURE);
} else if (res == 0) { /* child process */
dup2(fd[1], 1); /* redirect pipe to stdout */
dup2(fd[1], 2); /* redirect pipe also to stderr */
close(fd[1]); close(fd[0]); /* we don't need these */
execvp(program, argv);
perror("execvp");
exit(EXIT_FAILURE);
} else { /* parent process */
close(fd[1]); /* we are not going to write in the pipe */
FILE *f_in = fdopen(fd[0]);
/* read standard output and standard error from program from f_in FILE descriptor */
fclose(f_in);
wait(NULL); /* wait for child to finish */
}
You can see a complete example of this (not reading standard error, but it is easy to add --- you have only to add the second dup2() call from above) here. The program executes repeatedly a command you pass to it on the command line. It needs to get access to the output of the subprocess to count the lines, as between invocations, the program goes up as many lines as the program output, to make the next invocation to overlap the output of the last invocation. You can try it and play, making modifications as you like.
NOTE
In your sample redirection, when you use >&, you need to add a number after the ampersand, to indicate which descriptor you are dup()ing. As the number before the > is optional, the one after the & is mandatory. So, if you have not used it, prepare to receive an error (which probably you don't see if you are redirecting stderr) The idea of having two separate output descriptors is to allow you to redirect stdout and at the same time, conserve a channel where to put error messages.
This question already has answers here:
How can I pass the redirection operator '>' as an argument for execv?
(2 answers)
Closed 4 years ago.
I am trying to pass some arguments into an execv call:
char *const paramList[] = {"/bin/grep", "-rn", "comrade",
"/home/sgspectra/Documents/testing_grep/",
">>", "output.txt", NULL};
I believe that ">>" and "output.txt" are being read as arguments for additional directories to search through instead of to send the output to "output.txt". Is it possible to send an argument that would indicate only the previous argument is a directory?
When you type a command into a shell that supports >>, the shell handles the “>> filename’ in your command itself. Then it removes that text from the command line. The rest of the command line (the parts that were not removed, including your command and any other arguments), are then passed to execv or one of its related functions.
To do the same thing, what you need to do is something along the lines of:
Call fork to create a subprocess of your process.
In the parent (where fork returns a process ID for the child it created), go on with other execution—the parent might wait for the child or it might go on to do other things.
In the child (where fork returns 0), close file descriptor 1 (standard output) and use openat to open output.txt in append mode at file descriptor 1. Then call execv.
(Caveat emptor: It has been a few decades since I did things like this, so my memory may have changed, and the precise steps required may have changed.)
A cruder but easier-to-code option is to use execv to execute a shell, which you pass parameters requesting it to interpret and execute your command, including the redirection, as shown in this answer.
I am implementing a minishell that will emulate a real bash shell. I am stacked with the execution of commands in background such as ls &.
My first approach was the following (which does not work)
char *execArgs[] = { "ls", "&", NULL };
execvp("ls", execArgs);
Then, I tried another way by modifying the parent process of the fork() and not waiting for the child in case it should be run in background. The problem here is that then it should print the list of jobs running on background in order to simulate the background bahaviour in a bash shell, but the command jobs is not executed correctly as a parameter of execvp().
My question is, is there any easier way to implement this background calls in C? In case there isn't, what does it fail in either of the options that I have mentioned?
The reasons your example fails is
The "&" as you have it coded is an argument to the "ls" program. When you enter the command at a shell prompt, the "&" is consumed by the shell and the "ls" program never see the backgrounding.
The "exec()" call terminates the current program (i.e., your mini-shell).
What you may want is
system("ls &");
Read a good Unix book. You will need to know fork(), exec(), wait() and more.
In a C program, let's say i wann use Exec functions for executing a given program, for example if i wanna just try ls -l i'll do something like
args[0]="ls";
args[1]="-l";
args[2]=NULL;
...
execvp("ls", args);
and it's all fine. Now what if i wanna also add the redirection to a file (or to stderr)?
I'm stuck, it's obvious that adding >log.txt as a 3rd entry in the array won't work, but I don't know how to proceed.
And also, what if I wanna pass some Input parameters? What if i wanna execute a GCC command like "gcc -o out in redirection>log.txt" ?
[update from comment:]
It's a C program that simulate a shell which can "run strings", string that contains a command, a list o parameters, input and a redirection.
Just set up your file descriptors as the exec-d process shall find them and then do the exec.
For that you need open, dup2 and close.
All functions in the exec-family just replace the current process with whatever one you say.
Run the command in a shell:
char * args[] = {
"sh",
"-c",
"ls -l >out.ls 2>err.ls <in.ls",
NULL
};
...
execvp(args[0], args);
perror("execvp() failed");
I use fork()/exec()/wait() rather than system() when the command has user input as some of its arguments so the user can't put something like...
&rm -rf /home/* && echo HAHA
... as an argument.
I'm assuming popen is as dangerous as system() because it takes a single string and not a list of strings like the exec family of functions do.
I can only get the return value from the exec functions though. Is there a "safe" version of popen that I can run with user input and process stdout / stderr back in the parent process?
The safe way is to set up the necessary pipes yourself, using straight pipe() calls directly.
That's what popen() does under the hood, but it also invokes the shell in order to run the child process. Skipping that step should make it safer.