I need a little help with my plumbing.
I'm trying to create a program that begins with a single process, spawns child processes based on a user-defined number, and then flow back into another (single) child process. The data flow looks something like this:
/-->-█->-\
█-->--█->--█
\-->-█->-/
I've got the first part of the process creation finished. The fork works well - I run it through a loop limited to the number specified by the user. It's the piping that's throwing me off.
For simplicity, I'm focusing on the first part (from parent to multiple children). So I create the pipe before forking the process - that much is given. Then I close the write-end of the child process, close stdin, dup so the child's 0 redirects to stdin, then close the child's 0. That should take care of the plumbing on the child-side, right?
So then back in the parent process, I need to plumb for the pipes going to the children. For whatever reason, this is the harder part for me. Would someone mind walking me through it a little?
Here's what I've got for this part of the code:
for (i = 0; i < numberOfChildren; ++i) {
(void) pipe(workFDs[i]); /* Creates a pipe before the fork */
if ((workPIDs[i] = fork()) < 0) {
perror("Error: failure when forking workPID #: \n");
exit(-1);
}
if (workPIDs[i] == 0) {
/* ************************* WORKER PROCESS *********************** */
close(workFDs[i][1]); /* Closes the write-end for worker proc */
close(0); /* Closes stdin */
dup(workFDs[i][0]); /* Redirects workFDs 0 to stdin */
close(workFDs[i][0]);
//Fgets to get from the pipe
//Exec sort stuff here
}
} else {
/* *********************** PARENT PROCESS *********************** */
assert(inputPID == getpid()); /* Just to be sure */
close(workFDs[i][0]);
//Fputs to put into the pipe
}
}
Related
I have a multiple processes in progs and I wish to pipe the output from one process to another sequentially. I believe I have already linked the stdin of a process to the read end of a previous process, and linked the stdout to the write end of the pipe. I still am not seeing an output. Am I missing something with the links here?
int pipeFds[numProg][2];
for (int i = 0; i < numProg; i++) {
if (pipe(pipeFds[i]) != 0) { // create pipe for each process
printf("Failed to create pipe!");
exit(1);
}
child_pid = fork();
if (child_pid == -1) {
printf("Error while creating fork!");
exit(1);
}
if (child_pid == 0) {
if (i == 0) {
close(pipeFds[i][READ_END]); // close STDIN first process since it does not read
} else {
// change stdin to read end of pipe for intermediary processes
close(pipeFds[i]);
dup2(pipeFds[i - 1][READ_END], STDIN_FILENO);
}
dup2(pipeFds[i][WRITE_END], STDOUT_FILENO); // change stdout to write end of pipe
execvp(progs[i][0], (char *const * )progs[i]);
} else {
// parent process stuff
}
}
// Close pipes except last pipe for EOF
for (int i = 0; i < numProg - 1; i++) {
close(pipeFds[i][READ_END]);
close(pipeFds[i][WRITE_END]);
}
Remember you need to close all pipes, in each process, for it to work.
Example:
If numProg=2 you create 2 pipes and 2 child processes. Adding the parent, there a 3 processes running, and in each of them you eventually need to close all pipes.
For child_pid==0 you close the [READ_END] but never the [WRITE_END].
For child_pid==1 you do a close(pipeFds[1]). You need to specify [READ_END] and [WRITE_END].
Then each child process exits via execvp which, however, may return control if it fails. From the man page:
The exec() family of functions replaces the current process image with a new process image. .. The exec() functions only return if an error has occurred.
So you may want to add a _exit(0); after execvp to ensure each child process exits properly even if execvp fails.
The parent process closes all pipes but the last. So in the example of NumProg=2, [READ_END] and [WRITE_END] of pipeFd[1] both are never closed.
Lastly, the parent process should wait for all child processes to close (using while(wait(NULL) != -1);) otherwise you may end up with zombie or orphan processes.
your code contains a stray close(pipeFds[i]);
you have to close the pipes within the // parent process stuff. With your code, every child keeps the pipeFds of the previous children open. E.g.
} else {
// parent process stuff
if (i > 0)
close(pipeFds[i - 1][READ_END]);
close(pipeFds[i - 1][READ_END]);
}
it might be more effective to have a single fds[2] pair instead of numProg ones.
Using a fairly standard fork process:
int pipe_to_child[2];
int pipe_from_child[2];
int child_exit_status = -1;
pid_t child_pid = fork();
if (child_pid == 0) {
close(pipe_from_child[0]); // close their read end
close(pipe_to_child[1]); // Close their write end
dup2(pipe_to_child[0], STDIN_FILENO); // Tie the in pipe to stdin
dup2(pipe_from_child[1], STDOUT_FILENO); // Tie stdout to the out pipe
// Run the child process
execve(file_to_run, argv_for_prog, env_for_prog);
}
else {
close(pipe_from_child[1]); // close their write end
close(pipe_to_child[0]); // Close their read end
if (input_to_prog != NULL) write(pipe_to_child[1], input_to_prog, strlen(input_to_prog)); // Send the stdin stuff
close(pipe_to_child[1]); // Done so send EOF
// Wait for the child to end
waitpid(child_pid, &child_exit_status, 0);
// Do post end-of-child stuff
}
This generally works as expected.
However, when the child process, a shell script, sets a further process off in the background. Even though the child process then exits (and is no longer listed by ps), the waitpid doesn't return.
The script is this case is meant to start inadyn-mt (a DDNS updater) running in the background.
#!/bin/sh
inadyn-mt --background
(If I put an & after inadyn-mt it makes no difference)
It turns out that the issue is that the pipes don't get closed. Although the child process exits fine, because it has spawned a further process, this process (even though it doesn't want them) is tied to the pipes to the child's stdin and stdout. The solution I used was to not set up the pipes when I was going to spin off a child from the child.
I'm having trouble with my program waiting for a child process (gzip) to finish and taking a very long time in doing so.
Before it starts waiting it closes the input stream to gzip so this should trigger it to terminate pretty quickly. I've checked the system and gzip isn't consuming any CPU or waiting on IO (to write to disk).
The very odd thing is the timing on when it stops waiting...
The program us using pthreads internally. It's processing 4 pthreads side by side. Each thread processes many units of work and for each unit of work one it kicks off a new gzip process (using fork() and execve()) to write the result. Threads hang when gzip doesn't terminate, but it suddenly does terminate when other threads close their instance.
For clarity, I'm setting up a pipeline that goes: my program(pthread) --> gzip --> file.gz
I guess this could be explained in part by CPU load. But when processes are kicked off minutes apart and the whole system ends up using only 1 core of 4 because of this locking issue, that seems unlikely.
The code to kick off gzip is below. The execPipeProcess is called such that the child writes direct to file, but reads from my program. That is:
execPipeProcess(&process, "gzip", -1, gzFileFd)
Any suggestions?
typedef struct {
int processID;
const char * command;
int stdin;
int stdout;
} ChildProcess;
void closeAndWait(ChildProcess * process) {
if (process->stdin >= 0) {
stdLog("Closing post process stdin");
if (close(process->stdin)) {
exitError(-1,errno, "Failed to close stdin for %s", process->command);
}
}
if (process->stdout >= 0) {
stdLog("Closing post process stdin");
if (close(process->stdout)) {
exitError(-1,errno, "Failed to close stdout for %s", process->command);
}
}
int status;
stdLog("waiting on post process %d", process->processID);
if (waitpid(process->processID, &status, 0) == -1) {
exitError(-1, errno, "Could not wait for %s", process->command);
}
stdLog("post process finished");
if (!WIFEXITED(status)) exitError(-1, 0, "Command did not exit properly %s", process->command);
if (WEXITSTATUS(status)) exitError(-1, 0, "Command %s returned %d not 0", process->command, WEXITSTATUS(status));
process->processID = 0;
}
void execPipeProcess(ChildProcess * process, const char* szCommand, int in, int out) {
// Expand any args
wordexp_t words;
if (wordexp (szCommand, &words, 0)) exitError(-1, 0, "Could not expand command %s\n", szCommand);
// Runs the command
char nChar;
int nResult;
if (in < 0) {
int aStdinPipe[2];
if (pipe(aStdinPipe) < 0) {
exitError(-1, errno, "allocating pipe for child input redirect failed");
}
process->stdin = aStdinPipe[PIPE_WRITE];
in = aStdinPipe[PIPE_READ];
}
else {
process->stdin = -1;
}
if (out < 0) {
int aStdoutPipe[2];
if (pipe(aStdoutPipe) < 0) {
exitError(-1, errno, "allocating pipe for child input redirect failed");
}
process->stdout = aStdoutPipe[PIPE_READ];
out = aStdoutPipe[PIPE_WRITE];
}
else {
process->stdout = -1;
}
process->processID = fork();
if (0 == process->processID) {
// child continues here
// these are for use by parent only
if (process->stdin >= 0) close(process->stdin);
if (process->stdout >= 0) close(process->stdout);
// redirect stdin
if (STDIN_FILENO != in) {
if (dup2(in, STDIN_FILENO) == -1) {
exitError(-1, errno, "redirecting stdin failed");
}
close(in);
}
// redirect stdout
if (STDOUT_FILENO != out) {
if (dup2(out, STDOUT_FILENO) == -1) {
exitError(-1, errno, "redirecting stdout failed");
}
close(out);
}
// we're done with these; they've been duplicated to STDIN and STDOUT
// run child process image
// replace this with any exec* function find easier to use ("man exec")
nResult = execvp(words.we_wordv[0], words.we_wordv);
// if we get here at all, an error occurred, but we are in the child
// process, so just exit
exitError(-1, errno, "could not run %s", szCommand);
} else if (process->processID > 0) {
wordfree(&words);
// parent continues here
// close unused file descriptors, these are for child only
close(in);
close(out);
process->command = szCommand;
} else {
exitError(-1,errno, "Failed to fork");
}
}
Child process inherits open file descriptors.
Every subsequent gzip child process inherits not only pipe file descriptors intended for communication with that particular instance but also file descriptors for pipes connected to previous child process instances.
It means that stdin pipe is still open when the main process performs close since there are some other file descriptors for the same pipe in a few child processes. Once those ones terminate the pipe is finally closed.
A quick fix is to prevent child processes from inheriting pipe file descriptors intended for the master process by setting close-on-exec flag.
Since there are multiple threads involved spawning child processes should be serialized to prevent child process from inheriting pipe fds intended for another child process.
You have not given us enough information to be sure, as the answer depends on how you use the functions presented. However, your closeAndWait() function looks a bit suspicious. It may be reasonable to suppose that that the child process in question will exit when it reaches the end of its stdin, but what is supposed to happen to data it has written or even may still write to its stdout? It is possible that your child processes hang because their standard output is blocked, and it is slow for them to recognize it.
I think this reflects a design problem. If you are capturing the child processes' output, as you seem at least to support doing, then after you close the parent's end of a child's input stream you'll want the parent to continue reading the child's output to its end, and performing whatever processing it intends to do on it. Otherwise you may lose some of it (which for a child performing gzip would mean corrupted data). You cannot do that if you make closing both streams part of the process of terminating the child.
Instead, you should to close the parent's end of the child's stdin first, continue processing its output until you reach its end, and only then try to collect the child. You can make closing the parent's end of the child's output stream part of the process of collecting that child if you like. Alternatively, if you really do want to discard any remaining output from the child, then you should drain its output stream between closing the input and closing the output.
I've been messing around in C trying to figure out how to do this. Let's say I have my main program, the parent process. The parent creates three child processes, each of which will eventually run programs (but that's not important right now). What I'd like to do is make it so that the first child's stdout will be received by the second child's stdin. The second child's stdout will then be received by the third child's stdin.
The parent process's stdin/stdout aren't messed with at all.
So far, what I've got is
pipe(procpipe);
parentPid = getpid();
for(i = 0; i < 3; i++)
{
if(getpid() == parentPid)
{
child[i] = fork();
if(child[i] == 0)
{
mynumber = i+1;
}
}
}
But from there I'm kind of stuck as to how to use dup2 to properly assign my pipes, and in which section of my code to insert it. There are lots of examples on Google and this website of how to pipe from a parent to a child, but I'm yet to see one that will tell me exactly how to connect a child's stdout to another child's stdin.
Edit:
Forgot to mention: assume all my variables are properly initialised. The int 'mynumber' is so a child process knows upon creation which number it is, so I can give it instructions via
if(mynumber == whatever)
So you have a loop that creates several child processes. Each of these child processes will be using two pipes: read from previous and write to the next. To set up a pipe for the reading end you need to close the write end of the pipe, and dup2 the read end into the stdin. Similar for the pipe where the process will be writing.
void set_read(int* lpipe)
{
dup2(lpipe[0], STDIN_FILENO);
close(lpipe[0]); // we have a copy already, so close it
close(lpipe[1]); // not using this end
}
void set_write(int* rpipe)
{
dup2(rpipe[1], STDOUT_FILENO);
close(rpipe[0]); // not using this end
close(rpipe[1]); // we have a copy already, so close it
}
When you fork each children you need to attach the pipes to it.
void fork_and_chain(int* lpipe, int* rpipe)
{
if(!fork())
{
if(lpipe) // there's a pipe from the previous process
set_read(lpipe);
// else you may want to redirect input from somewhere else for the start
if(rpipe) // there's a pipe to the next process
set_write(rpipe);
// else you may want to redirect out to somewhere else for the end
// blah do your stuff
// and make sure the child process terminates in here
// so it won't continue running the chaining code
}
}
With this in hand you can now write a loop that continuously forks, attaches the pipes, and then reuses the output pipe as the input pipe for the next one. Of course, once both ends of a pipe have been connected to child processes, the parent should not leave it open for itself.
// This assumes there are at least two processes to be chained :)
// two pipes: one from the previous in the chain, one to the next in the chain
int lpipe[2], rpipe[2];
// create the first output pipe
pipe(rpipe);
// first child takes input from somewhere else
fork_and_chain(NULL, rpipe);
// output pipe becomes input for the next process.
lpipe[0] = rpipe[0];
lpipe[1] = rpipe[1];
// chain all but the first and last children
for(i = 1; i < N - 1; i++)
{
pipe(rpipe); // make the next output pipe
fork_and_chain(lpipe, rpipe);
close(lpipe[0]); // both ends are attached, close them on parent
close(lpipe[1]);
lpipe[0] = rpipe[0]; // output pipe becomes input pipe
lpipe[1] = rpipe[1];
}
// fork the last one, its output goes somewhere else
fork_and_chain(lpipe, NULL);
close(lpipe[0]);
close(lpipe[1]);
The closing bits are very important! When you fork with an open pipe, there will be four open file descriptors: two on the parent process, and two others on the child process. You have to close all of those you won't be using. That's why the code above always closes the irrelevant ends of the pipes in the child processes, and both ends on the parent.
Also note that I am giving special treatment to the first and the last processes, because I don't know where the input for the chain will come from, and where the output will go to.
I have Implemented a simple program which simulates $ls -l | wc -c command execution using simple pipes and execvp calls.
Now After redirecting stdin and stdout ,when executes the program , shell prompt disappears and it waits for the enter key to be pressed.
Any way of solving this issue . Plz criticize my code also..
Thanks
/* Create pipe */
ret_val=pipe(pipe_fd);
/*Error check */
if(ERROR==ret_val)
{
perror("Pipe creation error \n");
_exit(FAILURE);
}
/*Fork First Child */
pid_one = fork() ;
if( 0 == pid_one ) /*child process block */
{
/* First sub process */
/*printf("First sub process is %d \n",getpid());*/
/* redirect stdout to pipe's write end for sub process one*/
dup2(pipe_fd[1],1);
/*close pipe read end */
close(pipe_fd[0]);
execvp(cmd_one_tokens[0],cmd_one_tokens);
/* if execvp returns then if must have failed */
printf("Unknown Command \n ");
//exit(10);
}
else /*main process block */
{
/*printf(" Main process is %d \n",getpid());*/
/*Wait for first sub process to finish */
//wait(&status);
/*printf("Exit status of first child is %d \n ", WEXITSTATUS(status) );*/
/*Fork second subprocess */
pid_two = fork();
if( 0 == pid_two ) /*second child process block */
{
/* redirect stdin to pipe's read end for sub process two */
dup2(pipe_fd[0],0);
// close(0); /* close normal stdin */
// dup(pipe_fd[0]); /* make stdib same as pfds[0] */
/*close pipe write end */
close(pipe_fd[1]);
/* Second sub process */
/*printf("Second sub process is %d \n",getpid()); */
execvp(cmd_two_tokens[0] , cmd_two_tokens);
/* if execvp returns then if must have failed */
printf("Unknown Command \n ");
}
else /*main process block */
{
/* printf(" Main process is %d \n",getpid()); */
status=-1; /*reset status */
/*Waiting for the second sub process to finish in No hang fashion */
waitpid ( -1 , &status ,WNOHANG);
/*printf("Exit status of second child is %d \n ", WEXITSTATUS(status) ); */
}
}
You have to close the pipe file descriptors in the main process, after the second is forked. Until you have closed them, the child process (wc) will wait for input on the pipe which the main process still has open. You must be very careful to close all the unneeded ends of the pipe.
Your code does not do what you describe you want to do:
You create a pipe, fork a new process, redirect it's stdout to the pipe and make it execute some program (so far so good), then in the parent process you wait for your child to finish and only then fork second process, redirect it's stdin to the pipe other end and make it execute another program.
This is not what "ls | wc" does - in the shell they are running concurrently. Remove the first wait().
pid_one = fork() ;
if( 0 == pid_one ) /*child process block */
You're not checking fork(2) for an error return, which is a very real possibility. (The user could be bumping up against their RLIMIT_NPROC limit, kernel.threads-max, run out of memory for holding task structures, etc.)
More idiomatic use of fork(2) looks like this:
if(-1 == (pid = fork()) {
perror("fork");
exit(1); /* or return -1 or similar */
} else if (0 == pid) {
/* executing as child */
} else {
/* parent, pid is child */
}
execvp(cmd_two_tokens[0] , cmd_two_tokens);
/* if execvp returns then if must have failed */
printf("Unknown Command \n ");
Note that there are many reasons why execvp(3) can fail; simply printing "Unknown Command" may leave your users very confused in the future. It'd be better to call perror("execvp"); and give your users a chance to discover the real reason why their execvp(3) call failed.
waitpid ( -1 , &status ,WNOHANG);
Use of WNOHANG here might be dangerous; if the system is running "just right", your parent might get to this code before the child has even had a chance to begin executing. Because you've asked for it to return immediately if no child has exited, the child will probably be turned into a zombie when it finally does exit -- your code doesn't take the opportunity to wait for the child again.
I'm not sure what the best solution is: if you use SA_NOCLDWAIT to sigaction(2) to avoid creating zombies completely, you won't have an opportunity to ever collect the child's exit status. Installing a SIGCHLD signal handler might interfere with the rest of the process; your clients might have reason to set it themselves. Using a blocking waitpid(2) might stall processing elsewhere. And using a non-blocking waitpid(2) means you still have to collect the child's status sometime, possibly through polling. (But you can't use -1 for the pid in that case, as you might accidentally reap another child process.)