I'm in the process of properly understanding pipes and FDs and I'm trying to program the following thing:
The program basically compresses stuff like gzip does with the options -cf.
The basic idea is:
I create two pipes in the parent process, then I fork it twice so that I'll have two children. In the first child, I redirect the first pipe's read end to stdin, and the second pipe's write end to stdout. Then I exec gzip with the -cf options so that it'll write to stdout (now the writing end of pipe2)
In the second child, I read from pipe2 and either output it directly or save it to a file.
The problem is, however, that no data arrives at the second child and I'm not really sure why. Here's the code:
int main(int argc, char **argv) {
char *file;
int out = 0;
if(argc == 2) {
file = argv[1];
out = 1;
} else if (argc > 2) {
exit(EXIT_FAILURE);
}
int c1pipe[2];
int c2pipe[2];
pipe(c1pipe);
pipe(c2pipe);
int f;
for(int i = 0; i < 2; i++) {
switch(f = fork()) {
case 0: //child
if(i == 0) { //first loop iteration, child 1
close(c1pipe[1]);
dup2(c1pipe[0], fileno(stdin));
close(c1pipe[0]);
close(c2pipe[0]);
dup2(c2pipe[1], fileno(stdout));
close(c2pipe[1]);
execlp("gzip", "gzip", "-cf", (char *) NULL);
} else if (i == 1) { //second loop iteration, child2
close(c1pipe[0]);
close(c1pipe[1]);
close(c2pipe[1]);
FILE *read = fdopen(c2pipe[0], "r");
char buffer[1024];
if(out == 0) { //output to stdout
while(fgets(buffer, 1024, read) != NULL) {
fprintf(stdout, "%s", buffer);
fflush(stdout);
}
} else { //write to specified file
FILE *writeto = fopen(file, "w");
while(fread(buffer, sizeof(char), strlen(buffer)+1, read) > 0) {
fwrite(buffer, sizeof(char), strlen(buffer)+1, writeto);
fflush(writeto);
}
fclose(writeto);
}
close(c2pipe[0]);
fclose(read);
}
break;
case -1: //err
//not implemented
break;
default: //parent
if(i == 0) {
close(c2pipe[0]);
close(c2pipe[1]);
close(c1pipe[0]);
FILE *writer;
writer = fdopen(c1pipe[1], "w");
char buffer[1024];
while(fgets(buffer, sizeof buffer, stdin) != NULL) {
fwrite(buffer, sizeof (char), strlen(buffer)+1, writer);
}
close(c1pipe[1]);
fclose(writer);
}
break;
}
}
return 0;
}
Please excuse the missing error handling as I wanted to create a quick-and-dirty version.
Any help is appreciated.
In the parent process, you are closing both ends of c2pipe before you have forked the second child.
You'd probably have figured this out already if you had put any error handling in on any of the read/write calls. In fact, if you checked for an error on the dup2 calls and then looked at errno, you probably would have found that it was EBADF (bad file descriptor).
Another issue is that your parent process exits before it knows that both child processes have finished. This means that the child processes will receive a signal and will themselves be terminated. The parent needs to call one of the variants of wait() to make sure both children have gone.
Related
I am trying to read specific lines from a file using a parent process to read one line and a child process to read some other line. The simple text file has as contents: "This is line one\n", This is line two\n", etc.
However, I find that when I execute my program multiple times, my child does not always print the line from my file. The printf("Entering child\n") does always get executed, indicating that I am able to enter the child process fine.
Why does this work only sometimes?
int main() {
FILE* fptr = fopen("my_text.txt","r");
int stat;
pid_t pid;
int count = 0;
if(fptr != NULL) {
char line[256];
fflush(stdout);
if((pid = fork()) < 0) {
fprintf(stderr, "fork error: %s\n",strerror(errno));
}
else if(pid == 0) {
printf("entering child\n");
int lineNumber = 2; // read the second line
while (fgets(line, sizeof(line), fptr) != NULL) { // loop until we get to desired line
if (count == lineNumber) {
printf("CHILD read: %s", line);
fflush(stdout);
rewind(fptr);
break;
} else {
count++;
}
}
exit(0); // exit child process
} else { // parent process
int lineNumber = 1;
while (fgets(line, sizeof(line), fptr) != NULL) { // loop until desired line
if (count == lineNumber) {
printf("PARENT read: %s", line);
fflush(stdout);
rewind(fptr);
break;
} else {
count++;
}
}
wait(&stat); // reap the child process
}
fclose(fptr);
} else {
printf("File Does not exist\n");
exit(0);
}
return 0; }
From the above code, I sometimes print "This is line two" (from parent) and "This is line three" (from child), or sometimes only "This is line two". The goal is to get both to print.
The two processes share the open file description, which means that they both share the file offset
and therefore you have a race condition because they read the file concurrently. There are two ways obvious to fix:
use some IPC mechanism for synchronization, or file description locks - see for example flock.
open the file in each process after the fork.
Other advanced methods would e.g. be to read the file with pread, or mmap it before or after the fork...
I am trying to understand why my program hangs. The Parent sends input froma
file it reads to the child program, and the child program will send the result of its computation back to it's parent. However, I have trouble sending the message back through a second pipe. The parent seems to hang when reading from the pipe.
From the other posts, I have read it seems to indicate that the parent should wait for the child to finish by using wait or waitpid (which in my case both of them does not resolve my issue).
I have notice by adding print statement that neither the PARENT or the CHILD finishes.. Could someone please explain to me why this is happening?
Why does this not work?
int main(int argc,char** argv) {
char buffer[1];
int i;
int fd1[2]; int fd2[2];
pipe(fd1); pipe(fd2);
pid_t pid;
// FIRST PROCESS.
// -------------------
pid = fork();
if(pid == 0) {
int cnt;
dup2(fd1[0], STDIN_FILENO);
dup2(fd2[1], STDOUT_FILENO);
for (i = 0; i < 2; i++) {
close(fd1[i]);
close(fd2[i]);
}
while(read(STDIN_FILENO, buffer, sizeof(buffer)) > 0) {
fprintf(stderr, "( %s )", buffer);
cnt = cnt + *buffer - 48;
}
write(STDOUT_FILENO, &cnt, sizeof(cnt));
exit(0);
}
// PARENT.
// ------------------------
int file = open(argv[1], O_RDONLY);
// READ THE FILE.
while(read(file, buffer, 1) > 0) {
if (48 <= *buffer && *buffer <= 57) {
// PIPE TO CHILD.
write(fd1[1], buffer, 1);
}
}
// WAIT FOR CHILD TO FINISH SENDING BACK.
// int status = 0;
// waitpid(pid, &status, 0);
// THIS BLOCK DOESN'T RESOLVE ANYTHING. IT HANGS AT WAIT OR WAITPID.
// **** THIS IS THE PART WHERE IT DOESN'T WORK.
while(read(fd2[0], buffer, 1) > 0) {
fprintf(stderr, "RESULT : %s", buffer);
}
// CLOSING PIPES
for (i = 0; i < 2; i++) {
close(fd1[i]);
close(fd2[i]);
}
close(file);
exit(0);
}
You aren't closing enough file descriptors in the parent soon enough.
Rule of thumb: If you
dup2()
one end of a pipe to standard input or standard output, close both of the
original file descriptors returned by
pipe()
as soon as possible.
In particular, you should close them before using any of the
exec*()
family of functions.
The rule also applies if you duplicate the descriptors with either
dup()
or
fcntl()
with F_DUPFD
Now, your child process is following the RoT perfectly. But the corollary for parent processes is that they need to close the unused ends of the pipe, and they must close the write end of a pipe that they use to signal EOF to the reading end of that pipe. This is where your code fails.
Arguably, before reading the file, the parent process should close the read end of the pipe it uses to write to the child, and it should close the write end of the pipe it uses to read from the child.
Then, after reading the whole of the file, it should close the write end of the pipe to the child, before going into the 'read from child' loop. That loop never terminates because the parent still has the write end of the pipe open, so there's a process that could (but won't) write to the pipe.
Also, since the child writes the bytes of an integer onto a pipe, the parent should read the bytes of an integer. Using char buffer[1]; with a %s format is pointless; you need a null terminator for the string, and a single char buffer can't hold both a null byte and any data.
Along with various other improvements ('0' instead of 48, for example), you might end up with:
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
if (argc != 2) {
fprintf(stderr, "Usage: %s filename\n", argv[0]);
exit(EXIT_FAILURE);
}
int fd1[2];
int fd2[2];
char buffer[1];
pipe(fd1);
pipe(fd2);
pid_t pid = fork();
if (pid == 0) {
int cnt = 0;
dup2(fd1[0], STDIN_FILENO);
dup2(fd2[1], STDOUT_FILENO);
for (int i = 0; i < 2; i++) {
close(fd1[i]);
close(fd2[i]);
}
while (read(STDIN_FILENO, buffer, sizeof(buffer)) > 0) {
fprintf(stderr, "(%c)", buffer[0]); // Changed
cnt = cnt + buffer[0] - '0';
}
putc('\n', stderr); // Aesthetics
write(STDOUT_FILENO, &cnt, sizeof(cnt));
exit(0);
}
int file = open(argv[1], O_RDONLY);
if (file < 0) {
fprintf(stderr, "failed to open file '%s' for reading\n", argv[1]);
exit(EXIT_FAILURE);
}
close(fd1[0]); // Added
close(fd2[1]); // Added
while (read(file, buffer, sizeof(buffer)) > 0) {
if ('0' <= buffer[0] && buffer[0] <= '9') {
write(fd1[1], buffer, sizeof(buffer));
}
}
close(file); // Moved
close(fd1[1]); // Added
// Rewritten
int result;
while (read(fd2[0], &result, sizeof(result)) == sizeof(result)) {
fprintf(stderr, "RESULT : %d\n", result);
}
close(fd2[0]); // Added
// Close loop removed
return 0;
}
If that is stored in file pipe71.c and compiled, I get the following outputs when it is run:
$ ./pipe71 pipe71.c
(2)(0)(1)(2)(2)(2)(1)(1)(2)(0)(0)(2)(1)(0)(2)(2)(1)(0)(2)(1)(2)(0)(0)(0)(0)(0)(1)(0)(1)(1)(0)(2)(1)(0)(0)(0)(0)(9)(1)(1)(1)(1)(2)(0)(2)(0)(0)
RESULT : 49
$ ./pipe71 pipe71
(0)(0)(8)(0)(0)(2)(2)(0)(8)(1)(1)(5)(1)(1)(1)(1)(5)(1)(1)(1)(8)(5)(1)(9)(8)(5)(1)(1)(0)(4)(4)(4)(6)(0)(2)(8)(0)(0)(0)(2)(7)(1)(3)(8)(3)(0)(4)(3)(0)(4)(9)(0)(0)(0)(0)(7)(1)(9)(8)(1)(3)(0)
RESULT : 178
$
Parent has opened a file to read, I fork two children to read from file and write on different files.
child 1 reads the first line, and child 2,reads nothing. When I do an ftell, it reaches the end.
Can anyone please explain this behaviour?
f[0] = fopen("input", "r");
for ( i = 1; i <= 2; i++ ){
if ((pid = fork()) != 0){
waitpid(pid);
}
else
{
snprintf ( buffer, 10, "output%d", i );
printf("opening file %s \n",buffer);
f[i] = fopen( buffer, "w");
fgets(buff2, 10, f[0]);
fprintf(f[i], "%s", buff2);
fclose(f[i]);
_exit(0);
}
}
fclose(f[0]);
Your problem is buffering. stdio reads files on fully buffered mode by default, which means a call to fgets(3) will actually read a huge block of characters from the file, buffer everything, and then return the first line, while leaving the rest in the buffer, in the perspective of being called again in the future (remember that stdio strives for minimizing the number of read(2) and write(2) syscalls). Note that stdio buffering is a user-space thing; all the kernel sees is a single process reading a huge block on that file, and so the cursor is updated accordingly.
Common block sizes are 4096 and 8192; your input file is probably smaller than that and so the first process that calls fgets(3) ends up reading the whole file, leaving the cursor in the end. Buffering is tricky.
What can you do? One solution I can think of is to disable buffering (since this is an input stream we're talking about, we can't use line buffered mode, because line buffering is meaningless for input streams). So if you disable buffering on the input stream before forking, everything will work. This is done with setvbuf(3).
Here's a working example:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <errno.h>
static FILE *f[3];
static char buffer[128];
static char buff2[128];
int main(void) {
pid_t pid;
int i;
if ((f[0] = fopen("input", "r")) == NULL) {
perror("Error opening input file");
exit(EXIT_FAILURE);
}
if (setvbuf(f[0], NULL, _IONBF, 0) < 0) {
perror("setvbuf(3) failed");
exit(EXIT_FAILURE);
}
for (i = 1; i <= 2; i++) {
if ((pid = fork()) < 0) {
perror("fork(2) failed");
exit(EXIT_FAILURE);
}
if (pid != 0) {
if (waitpid(pid, NULL, 0) < 0) {
perror("waitpid(2) failed");
exit(EXIT_FAILURE);
}
} else {
snprintf(buffer, sizeof(buffer), "output%d", i);
printf("opening file %s\n", buffer);
if ((f[i] = fopen(buffer, "w")) == NULL) {
perror("fopen(2) failed");
exit(EXIT_FAILURE);
}
errno = 0;
if (fgets(buff2, sizeof(buff2), f[0]) == NULL) {
if (errno != 0) {
perror("fgets(3) error");
exit(EXIT_FAILURE);
}
}
fprintf(f[i], "%s", buff2);
fclose(f[i]);
exit(EXIT_SUCCESS);
}
}
fclose(f[0]);
return 0;
}
Note that this may incur a significant performance hit. Your code will be making a lot more syscalls, and it might be too expensive for huge files, but it doesn't seem to be a problem since apparently you're dealing with relatively small input files.
Here's an extract of my fork() man page:
The child process has its own copy of the parent's descriptors. These descriptors reference the same underlying objects, so that, for instance, file pointers in file objects are shared between the child and the parent, so that an lseek(2) on a descriptor in the child process can affect a subsequent read or write by the parent. This descriptor copying is also used by the shell to establish standard input and output for newly created processes as well as to set up pipes.
So your behaviour is completely normal. If you want your child to have its own file descriptor, it should open its own file.
For example, you could do the following:
for ( i = 1; i <= 2; i++ )
{
if ((pid = fork()) != 0)
{
waitpid(pid);
}
else
{
f[0] = fopen("input", "r"); // New
snprintf ( buffer, 10, "output%d", i );
printf("opening file %s \n",buffer);
f[i] = fopen( buffer, "w");
fgets(buff2, 10, f[0]);
fprintf(f[i], "%s", buff2);
fclose(f[i]);
fclose(f[0]); //New
_exit(0);
}
}
Also, you should check for errors (almost all the functions in your else could fail with error).
Good morning, I've been "fighting" with this method for a long time and finally I decided to ask for help because I don't get what am I doing wrong. I am trying to create multiple children from the same parent and make the STDOUT of "child 1" the STDIN of "child 2" that way until there are no more children like a pipeline.
My actual code
void filter(void) {
if(Number_cmd != 0) {
int p,i;
int fd[2];
for(i=0;i<Number_cmd;i++)
pipe(fd);
for(p=(Number_cmd-1); p>=0; p--){
switch(fork()) {
case -1:
perror("fork");
exit(1);
case 0:
/* Child */
close(fd[1]);
close(0);
dup(fd[0]);
close(fd[0]);
execlp(filter[p], filter[p], NULL);
perror("exec");
exit(1);
default:
/* Father */
close(fd[0]);
close(1);
dup(fd[1]);
close(fd[1]);
break;
}
}
}
}
void directory(char* directory_name) {
DIR* dir = NULL;
struct dirent* ent;
char fich[1024];
char buff[4096];
int fd, reading;
struct stat sdata;
dir = opendir(directory_name);
while((ent=readdir(dir))!=NULL) {
if(ent->d_name[0]=='.')
continue;
fich[0]='\0';
strcat(fich, directory_name);
strcat(fich, "/");
strcat(fich, ent->d_name);
stat(fich,&sdata);
if(S_ISDIR(sdata.st_mode))
continue;
fd = open(fich, O_RDONLY);
while((reading= read(fd, buff, 4096)) > 0){
(write(1, buff, reading) < reading);
continue;
}
close(fd);
}
closedir(dir);
}
The problem is when im trying to call the method with more than one command, it looks like ii doesnt do anything, but when i run it with one command is working ok.
Thank everyone in advance. (Sorry for my English; it's not my native language)
EDIT
This is the main method:
char** cmd;
int Number_cmd;
int main(int argc, char* argv[]){
cmd = &(argv[2]); /*list of the commands*/
Number_cmd = argc-2; /* number of commands*/
filter();
directory(argv[1]);
return 0;
}
Problem 1:
for(i=0;i<Number_cmd;i++)
pipe(fd);
Unless the variable Number_cmd is 1 (or smaller), you leak pipe descriptors like crazy. You need some sort of array of file descriptors:
int fds[Number_cmd][2];
for (int i = 0; i < Number_cmd; i++)
if (pipe(fd[i]) != 0)
…report error and abandon ship (remembering to close any opened pipes)…
Problem 2:
This is mainly a consequence of Problem 1 — but you don't close enough file descriptors. Essentially, if you have N pipes open, your child will end up closing 2*N file descriptors, after duplicating two to standard input and standard output. The first and last children will be different; they don't override standard input and standard output respectively.
There are probably other issues, but these two spring to mind at once from a fairly quick look at the code.
The goal of this project is to use pipes and forks to execute a line-count utility already written in a multi-process manner (one process per argument). I'm currently working on getting a single process working before expanding to handle multiple args.
Given two executables, lc1 and lc2, I want lc2 to establish a pipe to the stdout file descriptor of lc1, so that when execlp("lc1", argv[1], NULL) is called, the output will be read in by
while ((c= read(pipefd[0], readin, SIZE)) > 0)
According to my Unix book, I should use the open, dup2, close method for redirecting stdout to stdin, and here's my code:
int pid, c, i;
char *readin= (char *)malloc(sizeof(SIZE));
if (pipe(pipefd)== -1)
perror("Can't open a pipe\n");
for (i=1; i< argc; i++){
if ((pid= fork())==-1)
perror("Can't fork\n");
run(argv[i]);
}
//close pipe
close(1);
if (dup2(pipefd[0], 0)==-1)
perror("Can't redirect stdin");
close(pipefd[1]);
for (i=1; i< argc; i++){
if ((wait(NULL))== -1)
perror("Wait error");
while ((c= read(pipefd[0], readin, SIZE)) > 0){;
//print buf count
total += atoi(readin);
}
}
The run function is
void run(char *f){
int fp;
if ((fp= open(f, O_RDONLY)) == -1)
perror("Can't open the file");
close(pipefd[0]);
dup2(pipefd[1], 1);
close(pipefd[1]);
execlp("ls1", f, NULL);
}
When I try to execute this code, I get a stdin redirect error saying bad file descriptor. Why is this happening, and would appreciate any hints to for fixing.
run(argv[i]) is executed by both parent and child because are not assigning the functionality based on the returned PID, so one close after the other may have closed.
See below code, can he handy, I will use the code sample for situations like this. :
int main()
{
int pipe_fd[2] = {0};
int pid = -1;
int status = -1;
int ret_value = INVALID_CMD;
int cmd_output_len = -1;
status = pipe(pipe_fd);
if(status<0)
{
perror("pipe create err");
}
else
{
pid = fork();
if(pid<0)
{
}
else if (pid == 0)
{
/*Child functionality*/
child_func(pipe_fd, cmd);
}
else
{
/*Parent functionality*/
cmd_output_len = parent_fun(pid, pipe_fd);
}
}
return ret_value;
}
int child_func(int pipe_fd[], const char * cmd)
{
int status = 5;
int read_fd = pipe_fd[0]; /*read file descriptor*/
int write_fd = pipe_fd[1]; /*write file descriptor*/
int exit_status = 0;
/*close read fd*/
close(read_fd);
/*dup2 stdout to write fd*/
//status = dup2(1, write_fd);
status = dup2(write_fd, 1);
if(status<0)
{
exit(-1);
}
else
{
system(cmd);
exit(0);
}
}
int parent_fun(int child_id, int pipe_fd[])
{
int status = -1;
int len = 0;
bool_e break_loop = FALSE;
int read_fd = pipe_fd[0]; /*read file descriptor*/
int write_fd = pipe_fd[1]; /*write file descriptor*/
/*close write fd*/
close(write_fd);
while(1)
{
sleep(1);
status = waitpid(child_id, &status, WNOHANG);
switch(status)
{
case 0:
/*Child is still active*/
printf("No process waiting to exit..\n");
len = do_ur_fun(read_fd);
write(1, output, len);
break;
/*case EINTR:
case ECHILD:
case EINVAL:
perror("waitpid error");
break_loop = TRUE;
break;*/
default:
if(status<0)
{
perror("waitpid error");
break_loop = TRUE;
len = -1;
}
else if(child_id == status)
{
/*Valid staus from child*/
len = read_output(read_fd, output);
//write(1, output, len);
break_loop = TRUE;
}
else
{
}
break;
}
if(TRUE == break_loop)
{
break;
}
}
return len;
}
int do_ur_fun (int read_fd)
{
/*Do your exec*/
}
MaheshGupta024 identified a very important problem in your code; I'm assuming you will fix that.
One of the other problem areas is:
close(1);
if (dup2(pipefd[0], 0)==-1)
perror("Can't redirect stdin");
close(pipefd[1]);
for (i=1; i< argc; i++){
if ((wait(NULL))== -1)
perror("Wait error");
while ((c= read(pipefd[0], readin, SIZE)) > 0){;
//print buf count
total += atoi(readin);
}
}
The first close closes the process's standard output; this is seldom a good idea. The next line duplicates the read end of the pipe to standard input - which is fine. As noted in a comment above, perror() does not exit. You then close the write end of the pipe - that's correct; but you should presumably close the read end of the pipe too since you have set it to come from the pipe.
Your loop starts OK; you have redundant parentheses in the wait() line. You read from pipefd[0] instead of standard input - so maybe you didn't want to close pipefd[0] but neither did you need to duplicate it to standard input. You then have a nested loop that reads on the pipe while there's more data to be read from a child - you don't absolutely need the wait() code with its loop since the inner while won't terminate until all the children are dead. On the other hand, there's no great harm in it - after the first child dies, you'll read the data from all the other children, then go into the outer loop and wait for each other child, with the inner loop terminating immediately since there is no data left to read.
So:
Don't close stdout.
Don't dup the pipe read to stdin.
Decide whether you want to clean up the loop - it will work, but could be cleaner.
The run() function is:
void run(char *f){
int fp;
if ((fp= open(f, O_RDONLY)) == -1)
perror("Can't open the file");
close(pipefd[0]);
dup2(pipefd[1], 1);
close(pipefd[1]);
execlp("ls1", f, NULL);
}
The argument should be const char *f (or use name or file instead of f). I would also pass the pipefd array to the function rather than use a global variable
.
Do not call a file descriptor fp; that name conventionally indicates a variable of type FILE *, not int.
However, you don't need to open the file in the first place - unless you want the calling program to do the error reporting instead of the invoked program. However, if you do want the calling program to do the error reporting, you should close the file descriptor before proceeding. (I've already commented on perror() returning).
It would be a good idea to print an error message after execlp(); the only time the function returns is when it fails, so there is no need to test its return value. You might want to exit too - rather than have the failed function go through the rest of the main program after the call to run().
Good points: you did close both the pipe file descriptors.
Hence:
void run(const char *file, int *pipefd)
{
close(pipefd[0]);
dup2(pipefd[1], 1);
close(pipefd[1]);
execlp("ls1", f, NULL);
perror("Failed to exec ls1");
exit(EXIT_FAILURE);
}