I'm looking for a way in C to programmatically (ie, not using redirection from the command line) implement 'tee' functionality such that my stdout goes to both stdout and a log file. This needs to work for both my code and all linked libraries that output to stdout. Any way to do this?
You could popen() the tee program.
Or you can fork() and pipe stdout through a child process such as this (adapted from a real live program I wrote, so it works!):
void tee(const char* fname) {
int pipe_fd[2];
check(pipe(pipe_fd));
const pid_t pid = fork();
check(pid);
if(!pid) { // our log child
close(pipe_fd[1]); // Close unused write end
FILE* logFile = fname? fopen(fname,"a"): NULL;
if(fname && !logFile)
fprintf(stderr,"cannot open log file \"%s\": %d (%s)\n",fname,errno,strerror(errno));
char ch;
while(read(pipe_fd[0],&ch,1) > 0) {
//### any timestamp logic or whatever here
putchar(ch);
if(logFile)
fputc(ch,logFile);
if('\n'==ch) {
fflush(stdout);
if(logFile)
fflush(logFile);
}
}
putchar('\n');
close(pipe_fd[0]);
if(logFile)
fclose(logFile);
exit(EXIT_SUCCESS);
} else {
close(pipe_fd[0]); // Close unused read end
// redirect stdout and stderr
dup2(pipe_fd[1],STDOUT_FILENO);
dup2(pipe_fd[1],STDERR_FILENO);
close(pipe_fd[1]);
}
}
The "popen() tee" answers were correct. Here is an example program that does exactly that:
#include "stdio.h"
#include "unistd.h"
int main (int argc, const char * argv[])
{
printf("pre-tee\n");
if(dup2(fileno(popen("tee out.txt", "w")), STDOUT_FILENO) < 0) {
fprintf(stderr, "couldn't redirect output\n");
return 1;
}
printf("post-tee\n");
return 0;
}
Explanation:
popen() returns a FILE*, but dup2() expects a file descriptor (fd), so fileno() converts the FILE* to an fd. Then dup2(..., STDOUT_FILENO) says to replace stdout with the fd from popen().
Meaning, you spawn a child process (popen) that copies all its input to stdout and a file, then you port your stdout to that process.
You could use pipe(2) and dup2(2) to connect your standard out to a file descriptor you can read from. Then you can have a separate thread monitoring that file descriptor, writing everything it gets to a log file and the original stdout (saved avay to another filedescriptor by dup2 before connecting the pipe). But you would need a background thread.
Actually, I think the popen tee method suggested by vatine is probably simpler and safer (as long as you don't need to do anyhing extra with the log file, such as timestamping or encoding or something).
You can use forkpty() with exec() to execute the monitored program with its parameters. forkpty() returns a file descriptor which is redirected to the programs stdin and stdout. Whatever is written to the file descriptor is the input of the program. Whatever is written by the program can be read from the file descriptor.
The second part is to read in a loop the program's output and write it to a file and also print it to stdout.
Example:
pid = forkpty(&fd, NULL, NULL, NULL);
if (pid<0)
return -1;
if (!pid) /* Child */
{
execl("/bin/ping", "/bin/ping", "-c", "1", "-W", "1", "192.168.3.19", NULL);
}
/* Parent */
waitpid(pid, &status, 0);
return WEXITSTATUS(status);
There's no trivial way of doing this in C. I suspect the easiest would be to call popen(3), with tee as the command and the desired log file as an arument, then dup2(2) the file descriptor of the newly-opened FILE* onto fd 1.
But that looks kinda ugly and I must say that I have NOT tried this.
Related
I am trying to do the equivalent of the bash command ls>foo.txt in C.
The code bellow redirects the output to a variable.
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/wait.h>
int main(){
int pfds[2];
char buf[30];
pipe(pfds);
if (!fork()) {
close(pfds[0]);
//close(1);//Close stdout
//dup(pfds[1]);
//execlp("ls", "ls", NULL);
write(pfds[1], "test", 5); //Writing in the pipe
exit(0);
} else {
close(pfds[1]);
read(pfds[0], buf, 5); //Read from pipe
wait(NULL);
}
return 0;
}
The comments lines refer to those operations that I believe that are required for the redirection.
What should I change to redirect the output of ls to foo.txt?
While dealing with redirecting output to a file you may use freopen().
Assuming you are trying to redirect your stdout to a file 'output.txt' then you can write-
freopen("output.txt", "a+", stdout);
Here "a+" for append mode. If the file exists then the file open in append mode. Otherwise a new file is created.
After reopening the stdout with freopen() all output statement (printf, putchar) are redirected to the 'output.txt'. So after that any printf() statement will redirect it's output to the 'output.txt' file.
If you want to resume printf()'s default behavior again (that is printing in terminal/command prompt) then you have to reassign stdout again using the following code-
freopen("/dev/tty", "w", stdout); /*for gcc, ubuntu*/
Or -
freopen("CON", "w", stdout); /*Mingw C++; Windows*/
However similar technique works for 'stdin'.
What your code essentially does is that you open a pipe, then fork the process and in the child process (in commented code) close stdout, duplicate the pipe to stdout and execute and ls command, and then (in non-commented code) write 4 bytes to the pipe. In the parent process, you read data from the pipe and wait for the completion of the child process.
Now you want to redirect stdout to a file. You can do that by opening a file using the open() system call and then duplicating that file descriptor to stdout. Something like (I haven't tested this so beware of bugs in the code):
int filefd = open("foo.txt", O_WRONLY|O_CREAT, 0666);
if (!fork()) {
close(1);//Close stdout
dup(filefd);
execlp("ls", "ls", NULL);
} else {
close(filefd);
wait(NULL);
}
return 0;
However, you can also use the freopen as suggested by the other answer.
However, I have several concerns of your code and of my modified code:
The pipe() and open() system calls can fail. You should always check for system call failure.
The fork() system call can fail. Ditto.
dup2() can be used instead of dup(); otherwise the code will fail if stdin is not open as it duplicates to the first available file descriptor.
The execlp() system call can fail. Ditto.
I think wait() can be interrupted by a signal (EINTR). It's recommended to wrap it around a wrapper that retries the system call if it's aborted by a signal (errno == EINTR).
I am new to using pipes and forking in general. What I want to do is create a program that will execute the "cat" function in bash indirectly such that I can send input to cat through my program and receive the output in a text file.
I am having two problems:
Using the execvp function, is there a way of running "cat" without being forced to interact with the prompts, and instead send input through C?
The other issue is catching the input from the cat and writing it to a text file.
For instance, if I wrote something like
send_cat("hi");
send_cat("hello");
Then in the text file it would read
hi
hello
The solution to the problem you're describing involves the use of the pipe() system call and the dup2 system call.
Basically, you'd set up a pipe() between the parent and child processes, and then your solution should use dup2 to redirect the stdin of the child process that runs cat to come from the stdin of the process that calls execvp. Your solution should do something similar for stdout: use dup2 to redirect the stdout of the execvp child process to the stdout of the program.
Edit: There was a bit of hand-waving done in the above explanation, and you caught me in an extremely generous mood, so such a program structure might look like this:
Edit 2: I first tried writing this example program with cat instead of echo, but then I realized that you'd need to somehow send an EOF signal to the cat process from within the cat process, and sending a '\0' is ineffective.
int pipefd[2];
int result = pipe(pipefd);
if (result < 0) {
// pipe error
perror("pipe failure");
exit(1);
}
// Redirect the program's stdout and stdin to go to and from the pipe, respectively.
// This means that "echo"'s output will go to the pipe, and when "echo" finishes and we return execution to the parent process, we'll be able to read the information that "echo" just output from that pipe
// This is necessary in order to restore stdin and stdout to what they were prior to running this program
int savedStdin = dup(0);
int savedStdout = dup(1);
// Redirect stdin to come from the pipe
if ( dup2(pipefd[0], 0) < 0 ) {
perror("dup2 error");
exit(1);
}
// Close the read end of the pipe because the original descriptor was dupliechoed
close(pipefd[0]);
// Redirect stdout to go to the pipe
if ( dup2(pipefd[1], 1) < 0 ) {
perror("dup2 error");
exit(1);
}
// Close the write end of the pipe because the original descriptor was dupliechoed
close(pipefd[1]);
if ( fork() == 0 ) {
// Child process, will call "echo" and die
execlp("echo", "echo", "Hello_world!", NULL);
// The program should never ever get to this point, ever
// but if it does, we need to handle it
exit(1);
} else {
// Parent process, we need to wait for "echo" to terminate
wait(NULL);
// At this point stdout and stdin are still coming to/from the pipe, so if we do something like cin >> s, that will read from the pipe
// First, let's restore stdout to what it was before we redirected it, so that we can print the output of "echo" to the terminal
if (dup2(savedStdout, 1) < 0 ) {
perror("dup2 error");
exit(1);
}
close(savedStdout);
string s;
// Now we're going to read from stdin (the pipe) and print to stdout (the terminal, if you're running this from the command-line)
while (cin >> s) printf("%s\n", s.c_str() );
// We've read everything from "echo", let's fix stdin now
if (dup2(savedStdin, 0) < 0 ) {
perror("dup2 error");
exit(1);
}
close(savedStdin);
}
I am trying to do the equivalent of the bash command ls>foo.txt in C.
The code bellow redirects the output to a variable.
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/wait.h>
int main(){
int pfds[2];
char buf[30];
pipe(pfds);
if (!fork()) {
close(pfds[0]);
//close(1);//Close stdout
//dup(pfds[1]);
//execlp("ls", "ls", NULL);
write(pfds[1], "test", 5); //Writing in the pipe
exit(0);
} else {
close(pfds[1]);
read(pfds[0], buf, 5); //Read from pipe
wait(NULL);
}
return 0;
}
The comments lines refer to those operations that I believe that are required for the redirection.
What should I change to redirect the output of ls to foo.txt?
While dealing with redirecting output to a file you may use freopen().
Assuming you are trying to redirect your stdout to a file 'output.txt' then you can write-
freopen("output.txt", "a+", stdout);
Here "a+" for append mode. If the file exists then the file open in append mode. Otherwise a new file is created.
After reopening the stdout with freopen() all output statement (printf, putchar) are redirected to the 'output.txt'. So after that any printf() statement will redirect it's output to the 'output.txt' file.
If you want to resume printf()'s default behavior again (that is printing in terminal/command prompt) then you have to reassign stdout again using the following code-
freopen("/dev/tty", "w", stdout); /*for gcc, ubuntu*/
Or -
freopen("CON", "w", stdout); /*Mingw C++; Windows*/
However similar technique works for 'stdin'.
What your code essentially does is that you open a pipe, then fork the process and in the child process (in commented code) close stdout, duplicate the pipe to stdout and execute and ls command, and then (in non-commented code) write 4 bytes to the pipe. In the parent process, you read data from the pipe and wait for the completion of the child process.
Now you want to redirect stdout to a file. You can do that by opening a file using the open() system call and then duplicating that file descriptor to stdout. Something like (I haven't tested this so beware of bugs in the code):
int filefd = open("foo.txt", O_WRONLY|O_CREAT, 0666);
if (!fork()) {
close(1);//Close stdout
dup(filefd);
execlp("ls", "ls", NULL);
} else {
close(filefd);
wait(NULL);
}
return 0;
However, you can also use the freopen as suggested by the other answer.
However, I have several concerns of your code and of my modified code:
The pipe() and open() system calls can fail. You should always check for system call failure.
The fork() system call can fail. Ditto.
dup2() can be used instead of dup(); otherwise the code will fail if stdin is not open as it duplicates to the first available file descriptor.
The execlp() system call can fail. Ditto.
I think wait() can be interrupted by a signal (EINTR). It's recommended to wrap it around a wrapper that retries the system call if it's aborted by a signal (errno == EINTR).
I am trying to do the equivalent of the bash command ls>foo.txt in C.
The code bellow redirects the output to a variable.
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/wait.h>
int main(){
int pfds[2];
char buf[30];
pipe(pfds);
if (!fork()) {
close(pfds[0]);
//close(1);//Close stdout
//dup(pfds[1]);
//execlp("ls", "ls", NULL);
write(pfds[1], "test", 5); //Writing in the pipe
exit(0);
} else {
close(pfds[1]);
read(pfds[0], buf, 5); //Read from pipe
wait(NULL);
}
return 0;
}
The comments lines refer to those operations that I believe that are required for the redirection.
What should I change to redirect the output of ls to foo.txt?
While dealing with redirecting output to a file you may use freopen().
Assuming you are trying to redirect your stdout to a file 'output.txt' then you can write-
freopen("output.txt", "a+", stdout);
Here "a+" for append mode. If the file exists then the file open in append mode. Otherwise a new file is created.
After reopening the stdout with freopen() all output statement (printf, putchar) are redirected to the 'output.txt'. So after that any printf() statement will redirect it's output to the 'output.txt' file.
If you want to resume printf()'s default behavior again (that is printing in terminal/command prompt) then you have to reassign stdout again using the following code-
freopen("/dev/tty", "w", stdout); /*for gcc, ubuntu*/
Or -
freopen("CON", "w", stdout); /*Mingw C++; Windows*/
However similar technique works for 'stdin'.
What your code essentially does is that you open a pipe, then fork the process and in the child process (in commented code) close stdout, duplicate the pipe to stdout and execute and ls command, and then (in non-commented code) write 4 bytes to the pipe. In the parent process, you read data from the pipe and wait for the completion of the child process.
Now you want to redirect stdout to a file. You can do that by opening a file using the open() system call and then duplicating that file descriptor to stdout. Something like (I haven't tested this so beware of bugs in the code):
int filefd = open("foo.txt", O_WRONLY|O_CREAT, 0666);
if (!fork()) {
close(1);//Close stdout
dup(filefd);
execlp("ls", "ls", NULL);
} else {
close(filefd);
wait(NULL);
}
return 0;
However, you can also use the freopen as suggested by the other answer.
However, I have several concerns of your code and of my modified code:
The pipe() and open() system calls can fail. You should always check for system call failure.
The fork() system call can fail. Ditto.
dup2() can be used instead of dup(); otherwise the code will fail if stdin is not open as it duplicates to the first available file descriptor.
The execlp() system call can fail. Ditto.
I think wait() can be interrupted by a signal (EINTR). It's recommended to wrap it around a wrapper that retries the system call if it's aborted by a signal (errno == EINTR).
I'm implementing piping on a simulated file system in C++ (with mostly C). It needs to run commands in the host shell but perform the piping itself on the simulated file system.
I could achieve this with the pipe(), fork(), and system() system calls, but I'd prefer to use popen() (which handles creating a pipe, forking a process, and passing a command to the shell). This may not be possible because (I think) I need to be able to write from the parent process of the pipe, read on the child process end, write the output back from the child, and finally read that output from the parent. The man page for popen() on my system says a bidirectional pipe is possible, but my code needs to run on a system with an older version supporting only unidirectional pipes.
With the separate calls above, I can open/close pipes to achieve this. Is that possible with popen()?
For a trivial example, to run ls -l | grep .txt | grep cmds I need to:
Open a pipe and process to run ls -l on the host; read its output back
Pipe the output of ls -l back to my simulator
Open a pipe and process to run grep .txt on the host on the piped output of ls -l
Pipe the output of this back to the simulator (stuck here)
Open a pipe and process to run grep cmds on the host on the piped output of grep .txt
Pipe the output of this back to the simulator and print it
man popen
From Mac OS X:
The popen() function 'opens' a
process by creating a bidirectional
pipe, forking, and invoking the shell.
Any streams opened by previous popen()
calls in the parent process are closed
in the new child process.
Historically, popen() was implemented
with a unidirectional pipe; hence,
many implementations of popen() only
allow the mode argument to specify
reading or writing, not both. Because
popen() is now implemented using a
bidirectional pipe, the mode argument
may request a bidirectional data flow.
The mode argument is a pointer to a
null-terminated string which must be
'r' for reading, 'w' for writing, or
'r+' for reading and writing.
I'd suggest writing your own function to do the piping/forking/system-ing for you. You could have the function spawn a process and return read/write file descriptors, as in...
typedef void pfunc_t (int rfd, int wfd);
pid_t pcreate(int fds[2], pfunc_t pfunc) {
/* Spawn a process from pfunc, returning it's pid. The fds array passed will
* be filled with two descriptors: fds[0] will read from the child process,
* and fds[1] will write to it.
* Similarly, the child process will receive a reading/writing fd set (in
* that same order) as arguments.
*/
pid_t pid;
int pipes[4];
/* Warning: I'm not handling possible errors in pipe/fork */
pipe(&pipes[0]); /* Parent read/child write pipe */
pipe(&pipes[2]); /* Child read/parent write pipe */
if ((pid = fork()) > 0) {
/* Parent process */
fds[0] = pipes[0];
fds[1] = pipes[3];
close(pipes[1]);
close(pipes[2]);
return pid;
} else {
close(pipes[0]);
close(pipes[3]);
pfunc(pipes[2], pipes[1]);
exit(0);
}
return -1; /* ? */
}
You can add whatever functionality you need in there.
You seem to have answered your own question. If your code needs to work on an older system that doesn't support popen opening bidirectional pipes, then you won't be able to use popen (at least not the one that's supplied).
The real question would be about the exact capabilities of the older systems in question. In particular, does their pipe support creating bidirectional pipes? If they have a pipe that can create a bidirectional pipe, but popen that doesn't, then I'd write the main stream of the code to use popen with a bidirectional pipe, and supply an implementation of popen that can use a bidirectional pipe that gets compiled in an used where needed.
If you need to support systems old enough that pipe only supports unidirectional pipes, then you're pretty much stuck with using pipe, fork, dup2, etc., on your own. I'd probably still wrap this up in a function that works almost like a modern version of popen, but instead of returning one file handle, fills in a small structure with two file handles, one for the child's stdin, the other for the child's stdout.
POSIX stipulates that the popen() call is not designed to provide bi-directional communication:
The mode argument to popen() is a string that specifies I/O mode:
If mode is r, when the child process is started, its file descriptor STDOUT_FILENO shall be the writable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the readable end of the pipe.
If mode is w, when the child process is started its file descriptor STDIN_FILENO shall be the readable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the writable end of the pipe.
If mode is any other value, the result is unspecified.
Any portable code will make no assumptions beyond that. The BSD popen() is similar to what your question describes.
Additionally, pipes are different from sockets and each pipe file descriptor is uni-directional. You would have to create two pipes, one configured for each direction.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in popen simultaneous read and write)
Here's the code (C++, but can be easily converted to C):
#include <unistd.h>
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <utility>
// Like popen(), but returns two FILE*: child's stdin and stdout, respectively.
std::pair<FILE *, FILE *> popen2(const char *__command)
{
// pipes[0]: parent writes, child reads (child's stdin)
// pipes[1]: child writes, parent reads (child's stdout)
int pipes[2][2];
pipe(pipes[0]);
pipe(pipes[1]);
if (fork() > 0)
{
// parent
close(pipes[0][0]);
close(pipes[1][1]);
return {fdopen(pipes[0][1], "w"), fdopen(pipes[1][0], "r")};
}
else
{
// child
close(pipes[0][1]);
close(pipes[1][0]);
dup2(pipes[0][0], STDIN_FILENO);
dup2(pipes[1][1], STDOUT_FILENO);
execl("/bin/sh", "/bin/sh", "-c", __command, NULL);
exit(1);
}
}
Usage:
int main()
{
auto [p_stdin, p_stdout] = popen2("cat -n");
if (p_stdin == NULL || p_stdout == NULL)
{
printf("popen2() failed\n");
return 1;
}
const char msg[] = "Hello there!";
char buf[32];
printf("I say \"%s\"\n", msg);
fwrite(msg, 1, sizeof(msg), p_stdin);
fclose(p_stdin);
fread(buf, 1, sizeof(buf), p_stdout);
fclose(p_stdout);
printf("child says \"%s\"\n", buf);
return 0;
}
Possible Output:
I say "Hello there!"
child says " 1 Hello there!"
No need to create two pipes and waste a filedescriptor in each process. Just use a socket instead. https://stackoverflow.com/a/25177958/894520