I have some code that forks a third-party application and redirects its standard output to the parent process, roughly as follows (no error handling here for brevity):
char* args[] = {"/path/to/3rd/party/binary", "with", "args", NULL};
int fds[2];
pipe2(fds, O_CLOEXEC);
pid_t pid = fork();
if (pid == 0)
{
dup2(fds[1], STDOUT_FILENO);
execvp(args[0], args);
_exit(1);
}
else
{
close(fds[1]);
char buf[1024];
int bytes_read;
while ((bytes_read = read(fds[0], buf, sizeof buf - 1)) > 0)
{
buf[bytes_read] = '\0';
printf("%s", buf);
}
close(fds[0]);
waitpid(pid, NULL, 0);
}
I have no code for the third-party application, it is a proprietary binary. When running the third-party application in a terminal with the same arguments as used in the code above, it eventually finishes. However, when forking the third-party binary using the code above, it does not finish, but becomes a zombie process, and the code above hangs on the read() call.
The third-party binary that is forked itself forks two daemon processes (again, proprietary binaries I do not control), which I think is causing the problem here. The forked daemon processes will have a copy of the duplicated file descriptor, preventing the read() from finishing. Indeed, in case the dup2() call is replaced with:
dup3(fds[1], STDOUT_FILENO, O_CLOEXEC);
the child process finishes, but there is no output redirection to the parent process of course. Also, when the code above is modified to not do any output redirection to the parent over a pipe, the child process finishes correctly.
Is it possible to somehow prevent this hang on the read() call in this situation, or do I need to resort to some form of non-blocking I/O?
Update; using a simple popen() suffers from the same problem.
(Follow-up from: read() hangs on zombie process)
You need to specifically ignore SIGCHLD. It's your responsibility to reap the zombie, but you can't do it when blocked in read. If you call read after the SIGCHLD has been swallowed, you'll stay in read forever.
Related
I am trying to make a program to takes a command including pipes and then executes it. This is a simplified version of it where I'm trying to pipe the ls and wc command:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include<fcntl.h>
int main(){
char* arglist1[] = {"ls", NULL}; // writing process
char* arglist2[] = {"wc", NULL}; // reading process
int pipefd[2];
pid_t p1, p2;
if (pipe(pipefd) < 0) {
printf("\nPipe could not be initialized");
return 0;
}
p1 = fork();
if (p1 < 0) {
printf("\nCould not fork");
return 0;
}
if (p1 == 0) { // Child 1 executing it needs to write at the write end
close(pipefd[0]);
dup2(pipefd[1], STDOUT_FILENO);
close(pipefd[1]);
if (execvp(arglist1[0], arglist1) < 0) {
printf("\nCould not execute command 1..");
exit(0);
}
} else { // Parent executing
p2 = fork();
if (p2 < 0) {
printf("\nCould not fork");
return 0;
}
if (p2 == 0) { // Child 2 executing it needs to read at the read end
close(pipefd[1]);
dup2(pipefd[0], STDIN_FILENO);
close(pipefd[0]);
if (execvp(arglist2[0], arglist2) < 0) {
printf("\nCould not execute command 2..");
exit(0);
}
} else { // parent executing, waiting for two children
wait(NULL);
wait(NULL);
}
}
printf("\n");
return 0;
}
Although there is error handling in the program, it neither shows anything nor ends. Where is it blocking?
Your problem is that the parent doesn't close both the pipe's file descriptors, and the wc process won't die until it gets EOF on the pipe, and that won't happen until every process that has the write end of the pipe open has closed it. You need to close both ends of the pipe in the parent before waiting for the children to die.
Rule of thumb: If you
dup2()
one end of a pipe to standard input or standard output, close both of the
original file descriptors returned by
pipe()
as soon as possible.
In particular, you should close them before using any of the
exec*()
family of functions.
The rule also applies if you duplicate the descriptors with either
dup()
or
fcntl()
with F_DUPFD or F_DUPFD_CLOEXEC.
If the parent process will not communicate with any of its children via
the pipe, it must ensure that it closes both ends of the pipe early
enough (before waiting, for example) so that its children can receive
EOF indications on read (or get SIGPIPE signals or write errors on
write), rather than blocking indefinitely.
Even if the parent uses the pipe without using dup2(), it should
normally close at least one end of the pipe — it is extremely rare for
a program to read and write on both ends of a single pipe.
Note that the O_CLOEXEC option to
open(),
and the FD_CLOEXEC and F_DUPFD_CLOEXEC options to fcntl() can also factor
into this discussion.
If you use
posix_spawn()
and its extensive family of support functions (21 functions in total),
you will need to review how to close file descriptors in the spawned process
(posix_spawn_file_actions_addclose(),
etc.).
Note that using dup2(a, b) is safer than using close(b); dup(a);
for a variety of reasons.
One is that if you want to force the file descriptor to a larger than
usual number, dup2() is the only sensible way to do that.
Another is that if a is the same as b (e.g. both 0), then dup2()
handles it correctly (it doesn't close b before duplicating a)
whereas the separate close() and dup() fails horribly.
This is an unlikely, but not impossible, circumstance.
Side notes:
Error messages should be written to stderr, not stdout, and should end with a newline. They don't normally need to start with a newline.
You don't need to test the return value from the exec*() family of functions. If they succeed, they don't return; if they return, they failed. But it is important to have code after the eec*() call to trap the error.
The program should exit with a non-zero status (e.g. EXIT_FAILURE) if the exec*() function fails. Exiting with status zero reports success.
I'm writing a C program where I fork(), exec(), and wait(). I'd like to take the output of the program I exec'ed to write it to file or buffer.
For example, if I exec ls I want to write file1 file2 etc to buffer/file. I don't think there is a way to read stdout, so does that mean I have to use a pipe? Is there a general procedure here that I haven't been able to find?
For sending the output to another file (I'm leaving out error checking to focus on the important details):
if (fork() == 0)
{
// child
int fd = open(file, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
dup2(fd, 1); // make stdout go to file
dup2(fd, 2); // make stderr go to file - you may choose to not do this
// or perhaps send stderr to another file
close(fd); // fd no longer needed - the dup'ed handles are sufficient
exec(...);
}
For sending the output to a pipe so you can then read the output into a buffer:
int pipefd[2];
pipe(pipefd);
if (fork() == 0)
{
close(pipefd[0]); // close reading end in the child
dup2(pipefd[1], 1); // send stdout to the pipe
dup2(pipefd[1], 2); // send stderr to the pipe
close(pipefd[1]); // this descriptor is no longer needed
exec(...);
}
else
{
// parent
char buffer[1024];
close(pipefd[1]); // close the write end of the pipe in the parent
while (read(pipefd[0], buffer, sizeof(buffer)) != 0)
{
}
}
You need to decide exactly what you want to do - and preferably explain it a bit more clearly.
Option 1: File
If you know which file you want the output of the executed command to go to, then:
Ensure that the parent and child agree on the name (parent decides name before forking).
Parent forks - you have two processes.
Child reorganizes things so that file descriptor 1 (standard output) goes to the file.
Usually, you can leave standard error alone; you might redirect standard input from /dev/null.
Child then execs relevant command; said command runs and any standard output goes to the file (this is the basic shell I/O redirection).
Executed process then terminates.
Meanwhile, the parent process can adopt one of two main strategies:
Open the file for reading, and keep reading until it reaches an EOF. It then needs to double check whether the child died (so there won't be any more data to read), or hang around waiting for more input from the child.
Wait for the child to die and then open the file for reading.
The advantage of the first is that the parent can do some of its work while the child is also running; the advantage of the second is that you don't have to diddle with the I/O system (repeatedly reading past EOF).
Option 2: Pipe
If you want the parent to read the output from the child, arrange for the child to pipe its output back to the parent.
Use popen() to do this the easy way. It will run the process and send the output to your parent process. Note that the parent must be active while the child is generating the output since pipes have a small buffer size (often 4-5 KB) and if the child generates more data than that while the parent is not reading, the child will block until the parent reads. If the parent is waiting for the child to die, you have a deadlock.
Use pipe() etc to do this the hard way. Parent calls pipe(), then forks. The child sorts out the plumbing so that the write end of the pipe is its standard output, and ensures that all other file descriptors relating to the pipe are closed. This might well use the dup2() system call. It then executes the required process, which sends its standard output down the pipe.
Meanwhile, the parent also closes the unwanted ends of the pipe, and then starts reading. When it gets EOF on the pipe, it knows the child has finished and closed the pipe; it can close its end of the pipe too.
Since you look like you're going to be using this in a linux/cygwin environment, you want to use popen. It's like opening a file, only you'll get the executing programs stdout, so you can use your normal fscanf, fread etc.
After forking, use dup2(2) to duplicate the file's FD into stdout's FD, then exec.
You could also use the linux sh command and pass it a command that includes the redirection:
string cmd = "/bin/ls > " + filepath;
execl("/bin/sh", "sh", "-c", cmd.c_str(), 0);
For those such as myself who like a complete example with includes, here's this fantastic answer with a runnable example (still without error handling, left as an exercise):
#include <fcntl.h>
#include <sys/wait.h>
#include <unistd.h>
int main() {
if (fork() == 0) { // child
int fd = open("test.txt", O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
dup2(fd, 1); // make stdout go to file
dup2(fd, 2); // make stderr go to file - you may choose to not do this
// or perhaps send stderr to another file
close(fd); // fd no longer needed - the dup'ed handles are sufficient
execlp("ls", "ls", NULL);
}
else {
while (wait(NULL) > 0) {} // wait for each child process
}
return 0;
}
My mini-shell program accepts pipe command, for example, ls -l | wc -l and uses excevp to execute these commands.
My problem is if there is no fork() for execvp, the pipe command works well but the shell terminates afterward. If there is a fork() for execvp, dead loop happens. And I cannot fix it.
code:
void run_pipe(char **args){
int ps[2];
pipe(ps);
pid_t pid = fork();
pid_t child_pid;
int child_status;
if(pid == 0){ // child process
close(1);
close(ps[0]);
dup2(ps[1], 1);
//e.g. cmd[0] = "ls", cmd[1] = "-l"
char ** cmd = split(args[index], " \t");
//if fork here, program cannot continue with infinite loop somewhere
if(fork()==0){
if (execvp(cmd[0],cmd)==-1){
printf("%s: Command not found.\n", args[0]);
}
}
wait(0);
}
else{ // parent process
close(0);
close(ps[1]);
dup2(ps[0],0);
//e.g. cmd[0] = "wc", cmd[1] = "-l"
char ** cmd = split(args[index+1], " \t");
//if fork here, program cannot continue with infinite loop somewhere
if(fork()==0){
if (execvp(cmd[0],cmd)==-1){
printf("%s: Command not found.\n", args[0]);
}
}
wait(0);
waitpid(pid, &child_status, 0);
}
}
I know fork() is needed for excevp in order to not terminate the shell program, but I still cannot fix it. Any help will be appreciated, thank you!
How should I make two children parallel?
pid = fork();
if( pid == 0){
// child
} else{ // parent
pid1 = fork();
if(pid1 == 0){
// second child
} else // parent
}
is this correct?
Yes, execvp() replaces the program in which it is called with a different one. If you want to spawn another program without ending execution of the one that does the spawning (i.e. a shell) then that program must fork() to create a new process, and have the new process perform the execvp().
Your program source exhibits a false parallelism that probably either confuses you or reflects a deeper confusion. You structure the behavior of the first child forked in just the same way as the behavior of the parent process after the fork, but what should be parallel is the behavior of the first child and the behavior of the second child.
One outcome is that your program has too many forks. The initial process should fork exactly twice -- once for each child it wants to spawn -- and neither child should fork because it's already a process dedicated to one of the commands you want to run. In your actual program, however, the first child does fork. That case is probably rescued by the child also wait()ing for the grandchild, but it's messy and poor form.
Another outcome is that when you set up the second child's file descriptors, you manipulate the parent's, prior to forking, instead of manipulating the child's after forking. Those changes will persist in the parent process, which I'm pretty confident is not what you want. This is probably why the shell seems to hang: when run_pipe() returns (the shell's standard input has been changed to the read end of the pipe).
Additionally, the parent process should close both ends of the pipe after the children have both been forked, for more or less the same reason that the children must each close the end they are not using. In the end, there will be exactly one open copy of the file descriptor for each end of the pipe, one in one child and the other in the other. Failing to do this correctly can also cause a hang under some circumstances, as the processes you fork may not terminate.
Here's a summary of what you want the program to do:
The original process sets up the pipe.
The original process forks twice, once for each command.
Each subprocess manipulates its own file descriptors to use the correct end of the pipe as the appropriate standard FD, and closes the other end of the pipe.
Each subprocess uses execvp() (or one of the other functions in that family) to run the requested program
the parent closes its copies of the file descriptors for both ends of the pipe
the parent uses wait() or waitpid() to collect two children.
Note, too, that you should check the return values of all your function calls and provide appropriate handling for errors.
My question is extension of this one: popen creates an extra sh process
Motives:
1) My program need to create a child which does tail on a file. I need to process the output line by line. That is why I am using popen because it returns FILE *. I can easily fetch single line, do what I need to do and print it.
One problem with popen is that you do not get pid of child (tail command in my case).
2) My program should not exit before its child is done. So I need to do wait; but without pid, I cannot do it.
How can I achieve both the goals?
A possible (kludge) solution: do execvp("tail -f file > tmpfile") and the keep reading that tmpfile. I am not sure how good this solution is, though.
Why aren't you using pipe/fork/exec method?
pid_t pid = 0;
int pipefd[2];
FILE* output;
char line[256];
int status;
pipe(pipefd); //create a pipe
pid = fork(); //span a child process
if (pid == 0)
{
// Child. Let's redirect its standard output to our pipe and replace process with tail
close(pipefd[0]);
dup2(pipefd[1], STDOUT_FILENO);
dup2(pipefd[1], STDERR_FILENO);
execl("/usr/bin/tail", "/usr/bin/tail", "-f", "path/to/your/file", (char*) NULL);
}
//Only parent gets here. Listen to what the tail says
close(pipefd[1]);
output = fdopen(pipefd[0], "r");
while(fgets(line, sizeof(line), output)) //listen to what tail writes to its standard output
{
//if you need to kill the tail application, just kill it:
if(something_goes_wrong)
kill(pid, SIGKILL);
}
//or wait for the child process to terminate
waitpid(pid, &status, 0);
You can use pipe, a function of the exec* family and fdopen. This is non-standard, but so is popen.
You don't need to wait. Just read the pipe up to EOF.
execvp("tail -f file > tmpfile") won't work, redirection is a feature of the shell and you're not running the shell here. Even if it worked it would be an awful solution. Suppose you have read to the end of the file, but the child process has not ended yet. What do you do?
You can use wait as it doesn't want a PID to wait for but simply waits for the any child process to exit. If you have created other child processes you can keep track of them, and if wait returns an unknown PID you can assume it's from your popen process.
I'm not sure why you need the process ID of the child. When the child exits, your pipe read will return an EOF. If you need to terminate the child, just close the pipe.
I'm implementing piping on a simulated file system in C++ (with mostly C). It needs to run commands in the host shell but perform the piping itself on the simulated file system.
I could achieve this with the pipe(), fork(), and system() system calls, but I'd prefer to use popen() (which handles creating a pipe, forking a process, and passing a command to the shell). This may not be possible because (I think) I need to be able to write from the parent process of the pipe, read on the child process end, write the output back from the child, and finally read that output from the parent. The man page for popen() on my system says a bidirectional pipe is possible, but my code needs to run on a system with an older version supporting only unidirectional pipes.
With the separate calls above, I can open/close pipes to achieve this. Is that possible with popen()?
For a trivial example, to run ls -l | grep .txt | grep cmds I need to:
Open a pipe and process to run ls -l on the host; read its output back
Pipe the output of ls -l back to my simulator
Open a pipe and process to run grep .txt on the host on the piped output of ls -l
Pipe the output of this back to the simulator (stuck here)
Open a pipe and process to run grep cmds on the host on the piped output of grep .txt
Pipe the output of this back to the simulator and print it
man popen
From Mac OS X:
The popen() function 'opens' a
process by creating a bidirectional
pipe, forking, and invoking the shell.
Any streams opened by previous popen()
calls in the parent process are closed
in the new child process.
Historically, popen() was implemented
with a unidirectional pipe; hence,
many implementations of popen() only
allow the mode argument to specify
reading or writing, not both. Because
popen() is now implemented using a
bidirectional pipe, the mode argument
may request a bidirectional data flow.
The mode argument is a pointer to a
null-terminated string which must be
'r' for reading, 'w' for writing, or
'r+' for reading and writing.
I'd suggest writing your own function to do the piping/forking/system-ing for you. You could have the function spawn a process and return read/write file descriptors, as in...
typedef void pfunc_t (int rfd, int wfd);
pid_t pcreate(int fds[2], pfunc_t pfunc) {
/* Spawn a process from pfunc, returning it's pid. The fds array passed will
* be filled with two descriptors: fds[0] will read from the child process,
* and fds[1] will write to it.
* Similarly, the child process will receive a reading/writing fd set (in
* that same order) as arguments.
*/
pid_t pid;
int pipes[4];
/* Warning: I'm not handling possible errors in pipe/fork */
pipe(&pipes[0]); /* Parent read/child write pipe */
pipe(&pipes[2]); /* Child read/parent write pipe */
if ((pid = fork()) > 0) {
/* Parent process */
fds[0] = pipes[0];
fds[1] = pipes[3];
close(pipes[1]);
close(pipes[2]);
return pid;
} else {
close(pipes[0]);
close(pipes[3]);
pfunc(pipes[2], pipes[1]);
exit(0);
}
return -1; /* ? */
}
You can add whatever functionality you need in there.
You seem to have answered your own question. If your code needs to work on an older system that doesn't support popen opening bidirectional pipes, then you won't be able to use popen (at least not the one that's supplied).
The real question would be about the exact capabilities of the older systems in question. In particular, does their pipe support creating bidirectional pipes? If they have a pipe that can create a bidirectional pipe, but popen that doesn't, then I'd write the main stream of the code to use popen with a bidirectional pipe, and supply an implementation of popen that can use a bidirectional pipe that gets compiled in an used where needed.
If you need to support systems old enough that pipe only supports unidirectional pipes, then you're pretty much stuck with using pipe, fork, dup2, etc., on your own. I'd probably still wrap this up in a function that works almost like a modern version of popen, but instead of returning one file handle, fills in a small structure with two file handles, one for the child's stdin, the other for the child's stdout.
POSIX stipulates that the popen() call is not designed to provide bi-directional communication:
The mode argument to popen() is a string that specifies I/O mode:
If mode is r, when the child process is started, its file descriptor STDOUT_FILENO shall be the writable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the readable end of the pipe.
If mode is w, when the child process is started its file descriptor STDIN_FILENO shall be the readable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the writable end of the pipe.
If mode is any other value, the result is unspecified.
Any portable code will make no assumptions beyond that. The BSD popen() is similar to what your question describes.
Additionally, pipes are different from sockets and each pipe file descriptor is uni-directional. You would have to create two pipes, one configured for each direction.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in popen simultaneous read and write)
Here's the code (C++, but can be easily converted to C):
#include <unistd.h>
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <utility>
// Like popen(), but returns two FILE*: child's stdin and stdout, respectively.
std::pair<FILE *, FILE *> popen2(const char *__command)
{
// pipes[0]: parent writes, child reads (child's stdin)
// pipes[1]: child writes, parent reads (child's stdout)
int pipes[2][2];
pipe(pipes[0]);
pipe(pipes[1]);
if (fork() > 0)
{
// parent
close(pipes[0][0]);
close(pipes[1][1]);
return {fdopen(pipes[0][1], "w"), fdopen(pipes[1][0], "r")};
}
else
{
// child
close(pipes[0][1]);
close(pipes[1][0]);
dup2(pipes[0][0], STDIN_FILENO);
dup2(pipes[1][1], STDOUT_FILENO);
execl("/bin/sh", "/bin/sh", "-c", __command, NULL);
exit(1);
}
}
Usage:
int main()
{
auto [p_stdin, p_stdout] = popen2("cat -n");
if (p_stdin == NULL || p_stdout == NULL)
{
printf("popen2() failed\n");
return 1;
}
const char msg[] = "Hello there!";
char buf[32];
printf("I say \"%s\"\n", msg);
fwrite(msg, 1, sizeof(msg), p_stdin);
fclose(p_stdin);
fread(buf, 1, sizeof(buf), p_stdout);
fclose(p_stdout);
printf("child says \"%s\"\n", buf);
return 0;
}
Possible Output:
I say "Hello there!"
child says " 1 Hello there!"
No need to create two pipes and waste a filedescriptor in each process. Just use a socket instead. https://stackoverflow.com/a/25177958/894520