I have a legacy piece of code where I am delegating certain logic to a child process. The requirement is that parent writes to the pipe and the child reads on it. After reading, child writes something to a new pipe and parent reads from it.
In the code below, function send() is called periodically by a dedicated thread in the parent process.
LaunchWorker() ensures that the child process is forked only on the first call to it.
1) I am not being able to figure out how to close the read and write ends of the two descriptors so that the old data written to the pipe is flushed on every write.
2) Also is calling pipe() twice needed for the two descriptors?
Any input to make this code work will be greatly appreciated.
Thanks!
typedef struct
{
WebClient* wc; // user defined class
string url;
string data;
size_t dataLen;
DownloadObserver* downloadCallback; // user defined class
string* token;
}Payload;
PayLoad pl;
static pid_t worker_pid = 0;
int fd1[2];
int fd2[2];
bool done = false;
void LaunchWorker()
{
if (worker_pid != 0)
{
return;
}
pipe(fd1);
pipe(fd2);
worker_pid = fork();
}
void send()
{
//populate pl;
LaunchWorker();
if (worker_pid == 0)
{
while(true)
{
close(fd1[1]);
close(fd2[0]);
int cr = read(fd1[0], &pl, sizeof(Payload));
if (cr > 0)
{
// Upload some data to a remote http endpoint - uses libcurl
//if upload success, done = true
int cw = write(fd2[1], &done, sizeof(bool));
if (cw > 0)
{
// success
}
else
{
// failure
}
}
else
{
// failure
}
}
}
else if (workper_pid > 0)
{
close(fd1[0]);
close(fd2[1]);
int pw = write(fd1[1], &pl, sizeof(Payload));
if (pw > 0)
{
int pr = read(fd2[0], &done, sizeof(bool));
if (pr > 0)
{
// do something with value read
}
else
{
// failure
}
}
else
{
failure
}
}
}
Firstly, you need to close the unused ends of the pipe immediately after the fork, so the parent should close the read end of fd1 and the write end of fd2 and the child should close the write end of fd1 and the read end of fd2. If you don't do this, the pipes will not be closed when you finish.
Then, when you want to stop everything, I would have the writer of each pipe close the write end of the pipe. i.e.
parent closes write end of fd1
child reads eof on fd1 and closes it
child closes the write end of fd2
parent reads eof on fd2 and closes it.
Or the other way around if the child initiates shut down.
Finally, the parent process should issue the wait system call to collect the exit result of the child.
And yes, you do need two pipes. Pipes are one way.
Related
I'm writing a shell in C and am trying to implement multiple pipes. I've done this by creating a two dimensional array with pipes and using a separate pipe everytime. All commands in between pipes are separated by a parsing function and put into a struct. Every command line in-between pipes gets it's own process. And for all commands in the middle I'm trying to read from the previous process and write to the next one. Somewhere here the problem starts. It works fine for one pipe, however when I trying more than one pipe I don't get any output and the program gets stuck. In GDB I get a failed message from the execvp after forking the second process. What can this be due to?
int create_pipe(int* fd)
{
int pipe_id = pipe(fd);
if (pipe_id == -1)
{
return -1;
}
return 0;
}
void write_pipe(int* fd)
{
close(fd[READ]);
if ((dup2(fd[WRITE], STDOUT_FILENO)) < -1)
{
fork_error();
}
close(fd[WRITE]);
}
void read_pipe(int *fd)
{
close(fd[WRITE]);
if (dup2(fd[READ], STDIN_FILENO) < 0)
{
fork_error();
}
close(fd[READ]);
}
void need_to_pipe (int i, int (*fd)[2])
{
if (commands[i].pos == first)
{
write_pipe(fd[i * 2]);
}
else if (commands[i].pos == last)
{
read_pipe(fd[(i-1) *2]);
}
else //if (commands[i].pos == middle)
{
dup2(fd[(i-1)*2][READ], STDIN_FILENO);
close(fd[(i-1)*2][READ]);
close(fd[(i-1)*2][WRITE]);
//close(fd[(i)*2][READ]);
//close(fd[(i)*2][WRITE]);
close(fd[(i)*2][READ]);
dup2(fd[i*2][WRITE], STDOUT_FILENO);
close(fd[(i)*2][WRITE]);
}
}
/**
* Fork a proccess for command with index i in the command pipeline. If needed,
* create a new pipe and update the in and out members for the command..
*/
void fork_cmd(int i, int (*fd)[2]) {
pid_t pid;
switch (pid = fork()) {
case -1:
fork_error();
case 0:
// Child process after a successful fork().
if (!(commands[i].pos == single))
{
need_to_pipe(i, fd);
}
// Execute the command in the contex of the child process.
if (execvp(commands[i].argv[0], commands[i].argv)<0)
{
fprintf(stderr, "command not found: %s\n",
commands[i].argv[0]);
exit(EXIT_FAILURE);
}
default:
// Parent process after a successful fork().
break;
}
}
/**
* Fork one child process for each command in the command pipeline.
*/
void fork_cmds(int n, int (*fd)[2])
{
for (int i = 0; i < n; i++)
{
fork_cmd(i, fd);
}
}
void wait_once ()
{
wait(NULL);
}
/**
* Make the parents wait for all the child processes.
*/
void wait_for_all_cmds(int n)
{
for (int i = 0; i < n; i++)
{
wait_once();
//wait for number of child processes.
}
}
int main() {
int n; // Number of commands in a command pipeline.
size_t size = 128; // Max size of a command line string.
char line[size];
while(true) {
// Buffer for a command line string.
printf(" >>> ");
get_line(line, size);
n = parse_cmds(line, commands);
int fd[(n-1)][2];
for(int i =0;i<n-1;i++)
{
int pipe_id = pipe(fd[i*2]);
if (pipe_id == -1)
{
return -1;
}
}
fork_cmds(n, fd);
for(int i =0;i<n-1;i++)
{
int *fdclose= fd[i*2];
close (fdclose[READ]);
close (fdclose[WRITE]);
}
wait_for_all_cmds(n);
}
exit(EXIT_SUCCESS);
}
You [probably] have too many processes keeping pipe ends open (that do not belong to the given child) because your loop opens all pipes before any forking.
This places an undue burden on each child because it has to close many pipe ends to prevent it from holding open a pipe end, preventing other children from seeing an EOF on their input pipes.
To see this, for debug purposes in your present code, the child could do (just before the exec* call) (e.g.):
fprintf(stderr,"child: %d\n",getpid());
fflush(stderr);
system("ls -l /proc/self/fd 1>&2");
Each child should only have three open streams on stdin, stdout, and stderr (e.g. 0, 1, 2).
I think you'd find that there are many extraneous/detrimental streams open on the various children.
You only need two pipe arrays (e.g.): int pipeinp[2]; int pipeout[2]; Initially, pipeinp is all -1.
Roughly ...
Parent should do a single pipe call at the top of fork_cmd [before the fork] to pipeout.
The child dups (and closes) the read end of pipeinp [if not -1] to stdin.
Child dups/closes the write end of pipeout to stdout.
It closes the read end of pipeout.
After that, the parent should copy pipeout to pipeinp and close the write end of pipeinp
This should be repeated for all pipe stages.
No pipe to pipeout should be done for the last command. And, the [last] child should not change stdout.
For a working example, see my answer: fd leak, custom Shell
static int pipefd[2];
static pid_t cpid = 0;
void sigtimeout(int num) {
kill(cpid, 9);
close(pipefd[1]);
close(pipefd[0]);
pipe(pipefd);
write(pipefd[1], "T/O\n", 5);
}
void settimer(float time) {
struct itimerval timer;
timer.it_value.tv_sec = (int)time;
timer.it_value.tv_usec = (timeout - (int)time) * 1000000;
timer.it_interval.tv_sec = 0;
timer.it_interval.tv_usec = 0;
setitimer(ITIMER_REAL, &timer, NULL);
}
pid_t popen2(char *cmd) {
if (pipe(pipefd) == -1)
return -1;
int pid;
if ((pid = fork()) == -1)
return -1;
if (pid == 0) {
close(STDIN_FILENO);
close(STDERR_FILENO);
dup2(pipefd[1], STDOUT_FILENO);
close(pipefd[0]);
close(pipefd[1]);
execlp("sh", "sh", "-c", cmd, NULL);
_exit(EXIT_SUCCESS);
} else
settimer(timeout);
return pid;
}
void getcmd(const Block *block, char *output)
{
if (block->signal)
{
output[0] = block->signal;
output++;
}
strcpy(output, block->icon);
char *cmd;
if (button)
{
cmd = strcat(exportstring, block->command);
cmd[14] = '0' + button;
cpid = popen2(cmd);
if (cpid == -1) {
close(pipefd[0]);
close(pipefd[1]);
return;
}
cmd[16] = '\0';
}
else
{
cmd = block->command;
cpid = popen2(cmd);
if (cpid == -1) {
close(pipefd[0]);
close(pipefd[1]);
return;
}
}
button = 0;
waitpid(cpid, 0, 0);
settimer(0);
kill(cpid, 9);
close(pipefd[1]);
int i = strlen(block->icon);
read(pipefd[0], output+i, CMDLENGTH-i-delimLen);
close(pipefd[0]);
for (char *c = output; *c; c++)
if (*c == '\n') {
c[1] = '\0';
break;
}
i = strlen(output);
if (delim[0] != '\0') {
//only chop off newline if one is present at the end
i = output[i-1] == '\n' ? i-1 : i;
strncpy(output+i, delim, delimLen);
}
else
output[i++] = '\0';
}
So I'm trying to modify dwmblocks to add a timeout functionality. That way, if a command hangs, the whole status bar doesn't freeze.
Everything seems to work fine, but there is one little catch.
My code makes my whole linux system crash, cause it leaves some file descriptor open every time it runs a command. As you probably know, linux has protections against that, so every other application on my system that tries to open a file descriptor crashes too.
Thing is, I'm literally closing every pipe I open in my code, even the ones that open automatically when you fork. I just can't figure out what the problem is.
I'd really appreciate any help.
BTW: I'm only putting the relevant code here, because the problem is with the file descriptors. This is the only place where I work with file descriptors in my code.
Feel free to ask for more parts of the code if you feel it's relevant in some way :)
Your popen2() function leaves both pipe ends in the parent process open. Parent should always close the write end (that the child process writes to). Perhaps that is the one that leaks.
If you were to print (int)getpid() to show the process ID, you could list the pseudofiles in /proc/PID/fd/ as those describe the file descriptor that process has currently open. (If you use ls -laF /proc/PID/fd/ with the process ID number instead of PID, you can even see what/where the descriptors are open.)
In this answer here, I recently showed how one can implement a safe_pipe(), run(), and wait_all_children() functions for exploring complicated piping schemes between child processes. safe_pipe() sets the close-on-exec flag for each pipe descriptor by default, and run() clears that flag only in the child process if it uses such a descriptor, so the forked child processes do not have the other pipes' descriptors open at all. The included example.c also shows how the parent process must close all pipe ends used by the child processes when the child processes have been started, so that the child processes correctly detect end of inputs. (If the parent process has write ends open, then it could write to the pipe, so any child reading from the read end of that pipe will never see end-of-input.)
If necessary or helpful, I'm willing to explain how safe_pipe() and run() and wait_all_children() do what they do, and also why. In particular, run() uses an extra pipe between the child process prior to exec, and the parent process, to detect problems exec'ing the specified binary (and other errors). While there may be typos in there, the two are very robust implementations, not leaking resources etc.
Well this is embarassing hahaha
I had a typo in my settimer function : timeout was supposed to be time.
So the real problem was due to a racing condition, even though my piping system is far from perfect.
That goes to show how hard bugs can be to debug sometimes.
I am learning Linux and piping that kind stuff for my system programming course right now, I am having a hard time understanding closing file descriptors in an array of pipes now.
// write the code to loop over the command line arguments (remember to skip the executable name)
for (int i = 1; i < argc; i++) {
// call pipe before we fork
if ((pipe(pipe_fd[i-1])) == -1) {
perror("pipe");
exit(1);
}
// call fork
int result = fork();
if (result < 0) { // case: a system call error
// handle the error
perror("fork");
exit(1);
} else if (result == 0) { // case: a child process
// child does their work here
// child only writes to the pipe so close reading end
if (close(pipe_fd[i-1][0]) == -1) {
perror("close reading end from inside child");
exit(1);
}
// before we forked the parent had open the reading ends to
// all previously forked children -- so close those
int child_no;
for (child_no = 0; child_no < i-1; child_no++) {
if (close(pipe_fd[child_no][0]) == -1) {
perror("close reading ends of previously forked children");
exit(1);
}
}
int len = strlen(argv[i]);
// write len to the pipe as an integer
if (write(pipe_fd[i-1][1], &len, sizeof(int)) != sizeof(int)) {
perror("write from child to pipe");
exit(1);
}
// I'm done with the pipe so close it
if (close(pipe_fd[i-1][1]) == -1) {
perror("close pipe after writing");
exit(1);
}
// exit so I don't fork my own children on next loop iteration
exit(0);
} else {
// in the parent but before doing the next loop iteration
// close the end of the pipe that I don't want open
if (close(pipe_fd[i-1][1]) == -1) {
perror("close writing end of pipe in parent");
exit(1);
}
}
}
I will give a list of what I understand right now:
I understand parent and child process need to close those fds that they don't need to use, in this case child is writing to parent, so parent needs to close writing port and child needs to close reading port.
I understand file descriptors are shared among parent process and children process.
The above code is given from my lecture slide, I feel confused by one thing specifically.
In the loop, I observe that each child is closing its reading port once this child is created by fork, and the code that does this action is:
else if (result == 0) { // case: a child process
// child does their work here
// child only writes to the pipe so close reading end
if (close(pipe_fd[i-1][0]) == -1) {
perror("close reading end from inside child");
exit(1);
}
From what I understand at this point is that, each child is going to close its own reading port after being given birth by fork, and I think the latter children created SHOULD NOT worry about closing previous children's reading port.
But my understanding seems not correct after I read this code:
// before we forked the parent had open the reading ends to
// all previously forked children -- so close those
int child_no;
for (child_no = 0; child_no < i-1; child_no++) {
if (close(pipe_fd[child_no][0]) == -1) {
perror("close reading ends of previously forked children");
exit(1);
}
}
I don't understand why the latter children should go to close previous children's reading port, aren't those reading port already be closed once those children are created?
Thanks for helping me out. :)
A descriptor isn't really closed until all processes that have it open close it. Since each child inherits all the pipe descriptors from the previous process, they should close all the ones they're not using.
The main reason to close reading ports is so that the writing process will get an error or signal if it tries to write to the pipe after the reader has exited. If the other children kept all the reading ports opened, this wouldn't happen until all subsequent children exit.
I am trying to run ls|wc using execvp. So I create a pipe and then fork to create a child. I close the appropriate(read./write) end in parent/child and then map the other end to stdout/stdin. Then I run the ls in parent using execvp and wc in child. When I run the program it says
wc:standard input:bad file descriptor.
0 0 0
wc: -:Bad file descriptor
Here is my code:
int main()
{
//int nbBytes = 0; //stream length
int pfd_1[2]; //file descriptor
//char buffer[MAX_FILE_LENGTH];
char* arg[MAX_FILE_LENGTH];
pid_t processPid;
//Create a pipe
if(pipe(pfd_1) == -1)
{
printf("Error in creating pipe");
return 0;
}
//Create a child
processPid = fork();
if(processPid == -1)
{
printf("Erro in fork");
exit(1);
}
else if(processPid == 0) //Child
{
//redirect read end file descriptor to standard input
dup2(pfd_1[0],0);
//Close the write end
if(close(pfd_1[1] == -1))
{
printf("Error in closing the write end file descriptor");
exit(1);
}
arg[0] = "wc";
//arg[1] = "-l";
arg[1] = '\0';
if(execvp(arg[0],arg) == -1)
{
printf("Error in executing ls");
}
}
else //Parent
{
//redirect standard output to the file descriptor
dup2(pfd_1[1],1);
//Close the read end
if(close(pfd_1[0] == -1))
{
printf("Error in closing the read end from parent");
exit(1);
}
//Command
arg[0] = "ls";
arg[1] = "/proc/1/status";
arg[2] = '\0';
if(execvp(arg[0],arg) == -1)
{
printf("Error in executing ls");
}
}
}
Any idea what might be wrong? Why would it consider standard input as bad file descriptor? My understanding was since the stdin and read end file descriptor are aliases so the wc -l would read whatever the output is from the parent process. Do I need to do scanf to read from the stdin?
The problem is in this line:
if(close(pfd_1[1] == -1))
You are closing the result of pfd_1[1] == -1, which is by necessity equal to 0 (as they will never be equal). The correct line would probably be:
if (close(pfd_1[1]) == -1)
Note that you do this again later in attempting to close the read end in the parent process.
If you're going to fork children, you have to call wait() in the parent process in order to avoid "zombie" child processes. So you don't want to overlay the parent process that did the original process forking with another executable via exec.
One quick way to setup a series of pipes in the way you want would be to fork a child for each executable you want to run, and read that data back into a buffer in the parent. Then feed that data from the first child into a new child process that the parent forks off. So each child is fed data from the parent, processes the data, and writes the data back to the parent process, which stores the transformed data in a buffer. That buffer is then fed to the next child, etc., etc. The final results of the data in the buffer are the final output of the pipe.
Here's a little pseudo-code:
//allocate buffer
unsigned char buffer[SIZE];
for (each executable to run in pipeline)
{
pipes[2];
pipe(pipes);
pid_t pid = fork();
if (pid == 0)
{
//setup the pipe in the child process
//call exec
}
else
{
//setup the pipe in the parent process
if (child executable is not the first in the pipeline)
{
//write contents of buffer to child process
}
//read from the pipe until the child exits
//store the results in buffer
//call wait, and maybe also check the return value to make sure the
//child returned successfully
wait(NULL);
//clean up the pipe
}
}
I'm very beginner with linux however I've managed to do my own shell. It's time to add pipelines in there. (That's what, the homework says). Could anyone explain me a little bit more how to do that? I know that in theory, it should work like that.
unsigned char* child_results; //buffer to save results from child processes
for (int i = 0; i < 2; i++) {
pid = fork();
if (pid == 0) {
//if it's the first process to launch, close the read end of the pipe
//otherwise read what the parent writes to the pipe and then close the
//read end of the pipe
//call execvp()
}
else {
//if you've launched the first process, close the write end of the pipe
//otherwise write the contents of child_result to the child process
//and then close the write end of the pipe
//read from the child's pipe what it processed
//save the results in the child_results buffer
wait(NULL); //wait for the child to finish
}
}
However, I can't get it working. I'm doing that whole day and still, nothing. I do understand the idea, but I can't get it work. Could some1 help me? Here's the code of my pipelines part:
for (int i = 0; i <= pipeline_count; i++) {
int pdesc[2];
// creating pipe
pipe(pdesc);
int b = fork();
// child
if (b == 0) {
// 1st pipeline
if (i == 0) {
//<?>
}
// last pipeline
if (i == pipeline_count) {
//<?>
}
// inside pipeline
if (i > 0 && i < pipeline_count) {
//<?>
}
execvp(array[0], array);
}
else {
// parent
//<?>
wait(NULL);
}
}
and here's an example of a shell command
ls -al | tr a-z A-Z
Thanks
You must close the input stream on the child and duplicate with dup the pipe for that channel. The parent does the same with the other side of the pipe. Something like this:
b = fork();
if (b == 0) {
/* Close stdin, and duplicate the input side of the pipe over stdin */
dup2(0, pdesc[0]);
execlp(...);
}
else {
/* Close stdout, and duplicate the output side of the pipe over stdout */
dup2(1, pdesc[1]);
execlp(...);
}
...
I've shown you how to do it on a two processes case, but you can get the general idea and adapt it to other situations.
Hope that helped!