I have 2 simple programs that I use to modify text files.
The first one receives a commands file and a list of files to modify. The second program is made to be run by the first one, receives as a parameter a file to be read then reads commands from the stdin, execute them on the file and display to the stdout when it's done. Repeat.
My problem is that each fork seems to be waiting for the others to finish all it's commands rather than starting its own.
Main program
int main(int argc, char const *argv[]){
...
// Multiple forks
for(i=2; i < argc; i++){
...
pipe(pipes[forks]);
forks++;
pid = fork();
if(pid == 0) break;
...
}
if(pid == 0){
dup2(pipes[forks-1][0], STDIN_FILENO);
// Execute the second program in the fork,
// pass a file to be read as a single argument
execl("second_program", "second_program" , argv[i], NULL);
fprintf(stderr, "execl() failed\n");
return EXIT_FAILURE;
} else {
...
// Reads a file containing commands, normalize it and output a string
char *cmds = getcmds(argv[1]);
int f;
for(f=0; f < forks; f++){
// Send the list of commands to the second program
write(pipes[f][1], cmds, strlen(cmds));
}
wait(NULL);
}
return 0;
}
Child program
int main(int argc, char const *argv[]){
...
char buffer[READ_SIZE];
do {
scanf("%s",buffer);
if(strcmp(buffer,"COMMAND_A") == 0){
...
printf("`COMMAND_A on %s\n", filename);
} if(strcmp(buffer,"COMMAND_b") == 0){
...
printf("`COMMAND_b on %s\n", filename);
} else {
break;
}
...
} while(1);
return 0;
}
Example output I'm getting
COMMAND_A on file1
COMMAND_B on file1
...
COMMAND_Z on file1
COMMAND_A on file2
COMMAND_B on file2
...
COMMAND_Z on file2
COMMAND_A on file3
COMMAND_B on file3
...
COMMAND_Z on file3
The forks blocks and complete all theirs tasks rather than running in parallel. Is there something I'm doing wrong?
The forks start running concurrently, but they are all waiting for input through the pipe at the point of calling scanf. Try changing the order in which you write, and the order of the reads will change, too. Also, whenever strlen(cmds) exceeds PIPE_BUF, you may observe different results.
Writing something shorter than PIPE_BUF on a pipe is guaranteed to be delivered in one chunk at the other end. It's not that each command in your cmds wakes up the child processes.
Related
I'm writing a shell in C and am trying to implement multiple pipes. I've done this by creating a two dimensional array with pipes and using a separate pipe everytime. All commands in between pipes are separated by a parsing function and put into a struct. Every command line in-between pipes gets it's own process. And for all commands in the middle I'm trying to read from the previous process and write to the next one. Somewhere here the problem starts. It works fine for one pipe, however when I trying more than one pipe I don't get any output and the program gets stuck. In GDB I get a failed message from the execvp after forking the second process. What can this be due to?
int create_pipe(int* fd)
{
int pipe_id = pipe(fd);
if (pipe_id == -1)
{
return -1;
}
return 0;
}
void write_pipe(int* fd)
{
close(fd[READ]);
if ((dup2(fd[WRITE], STDOUT_FILENO)) < -1)
{
fork_error();
}
close(fd[WRITE]);
}
void read_pipe(int *fd)
{
close(fd[WRITE]);
if (dup2(fd[READ], STDIN_FILENO) < 0)
{
fork_error();
}
close(fd[READ]);
}
void need_to_pipe (int i, int (*fd)[2])
{
if (commands[i].pos == first)
{
write_pipe(fd[i * 2]);
}
else if (commands[i].pos == last)
{
read_pipe(fd[(i-1) *2]);
}
else //if (commands[i].pos == middle)
{
dup2(fd[(i-1)*2][READ], STDIN_FILENO);
close(fd[(i-1)*2][READ]);
close(fd[(i-1)*2][WRITE]);
//close(fd[(i)*2][READ]);
//close(fd[(i)*2][WRITE]);
close(fd[(i)*2][READ]);
dup2(fd[i*2][WRITE], STDOUT_FILENO);
close(fd[(i)*2][WRITE]);
}
}
/**
* Fork a proccess for command with index i in the command pipeline. If needed,
* create a new pipe and update the in and out members for the command..
*/
void fork_cmd(int i, int (*fd)[2]) {
pid_t pid;
switch (pid = fork()) {
case -1:
fork_error();
case 0:
// Child process after a successful fork().
if (!(commands[i].pos == single))
{
need_to_pipe(i, fd);
}
// Execute the command in the contex of the child process.
if (execvp(commands[i].argv[0], commands[i].argv)<0)
{
fprintf(stderr, "command not found: %s\n",
commands[i].argv[0]);
exit(EXIT_FAILURE);
}
default:
// Parent process after a successful fork().
break;
}
}
/**
* Fork one child process for each command in the command pipeline.
*/
void fork_cmds(int n, int (*fd)[2])
{
for (int i = 0; i < n; i++)
{
fork_cmd(i, fd);
}
}
void wait_once ()
{
wait(NULL);
}
/**
* Make the parents wait for all the child processes.
*/
void wait_for_all_cmds(int n)
{
for (int i = 0; i < n; i++)
{
wait_once();
//wait for number of child processes.
}
}
int main() {
int n; // Number of commands in a command pipeline.
size_t size = 128; // Max size of a command line string.
char line[size];
while(true) {
// Buffer for a command line string.
printf(" >>> ");
get_line(line, size);
n = parse_cmds(line, commands);
int fd[(n-1)][2];
for(int i =0;i<n-1;i++)
{
int pipe_id = pipe(fd[i*2]);
if (pipe_id == -1)
{
return -1;
}
}
fork_cmds(n, fd);
for(int i =0;i<n-1;i++)
{
int *fdclose= fd[i*2];
close (fdclose[READ]);
close (fdclose[WRITE]);
}
wait_for_all_cmds(n);
}
exit(EXIT_SUCCESS);
}
You [probably] have too many processes keeping pipe ends open (that do not belong to the given child) because your loop opens all pipes before any forking.
This places an undue burden on each child because it has to close many pipe ends to prevent it from holding open a pipe end, preventing other children from seeing an EOF on their input pipes.
To see this, for debug purposes in your present code, the child could do (just before the exec* call) (e.g.):
fprintf(stderr,"child: %d\n",getpid());
fflush(stderr);
system("ls -l /proc/self/fd 1>&2");
Each child should only have three open streams on stdin, stdout, and stderr (e.g. 0, 1, 2).
I think you'd find that there are many extraneous/detrimental streams open on the various children.
You only need two pipe arrays (e.g.): int pipeinp[2]; int pipeout[2]; Initially, pipeinp is all -1.
Roughly ...
Parent should do a single pipe call at the top of fork_cmd [before the fork] to pipeout.
The child dups (and closes) the read end of pipeinp [if not -1] to stdin.
Child dups/closes the write end of pipeout to stdout.
It closes the read end of pipeout.
After that, the parent should copy pipeout to pipeinp and close the write end of pipeinp
This should be repeated for all pipe stages.
No pipe to pipeout should be done for the last command. And, the [last] child should not change stdout.
For a working example, see my answer: fd leak, custom Shell
I'm fine executing commands like "ls" and stuff like that but I want to do something like "ls | sort" but the execvp system call doesn't support "|". How can I do this using only system calls? when I try something like
char *arg[] = {"ls","|","sort",NULL};
execvp(arg[0],arg);
it doesn't work, how can I do this?
Edit:
char* execString (char string[]){
int link[2];
pipe(link);
if (fork() == 0){
int i = 0;
char *p = strtok(string," ");
char *x[spacecount(string)+2];
while(p){
x[i++] = p;
p = strtok(NULL," ");
}
x[i] = NULL;
dup2(link[1],1);
close(link[0]);
close(link[0]);
execvp(x[0],x);
_exit(0);
} else {
wait(NULL);
close(link[1]);
char buf[512];
int i = 0;
while (read(link[0],&buf[i++],1) == 1);
close(link[0]);
buf[i-2] = '\0';
return strdup(buf);
}
}
This is the function i'm executing to exec a string that contains a command, its return value is a pointer to a string that contains the output from that command, how can I use that output as the input to a new command using execvp or another function from the exec family?
Edit2: So I made a new function that receives two strings as argument and execs the first one then the second one using as input the output from the first exec, I thought it was working fine it worked with ls | head -1 and other variations of ls but when I do something like ls | sort -R it doesn't work, i've tried several things and I can't understand why this is happening, here is the code:
char* execStrings (char previousstring[], char string[]){
int link[2];
pipe(link);
if (fork() == 0){
int i = 0;
char *previouscommand[spacecount(previousstring)+2];
char *temp = strtok(previousstring," ");
while(temp){
previouscommand[i++] = temp;
temp = strtok(NULL," ");
}
previouscommand[i] = NULL;
dup2(link[1],1); /* stdout result redrecting to write end of pipe */
close(link[1]);
close(link[0]);
execvp(previouscommand[0],previouscommand);
} else {
wait(NULL);
int res[2];
pipe(res);
if(fork() == 0){
int i = 0;
char *temp = strtok(string," ");
char *command[spacecount(string)+2];
while(temp){
command[i++] = temp;
temp = strtok(NULL," ");
}
command[i] = NULL;
dup2(link[0],0);
close(link[0]);
close(link[1]);
dup2(res[1],1);
close(res[1]);
close(res[0]);
execvp(command[0],command)
} else {
wait(NULL);
close(res[1]);
char buf[512];
int i = 0;
while (read(res[0],&buf[i++],1) == 1);
close(res[0]);
buf[i-2] = '\0';
return strdup(buf);
}
}
}
you want to do something like ls | sort but the way you are doing like
char *arg[] = {"ls","|","sort",NULL};
execvp(arg[0],arg); /*it won't work */
won't work because here you are calling execvp on ls and sort which are two separate process not single process. Also
ls | sort => output of process-1 make as input to process-2 & execute it
| |
process-1 process-2
To achieve the above create two process by calling fork() and use exec() family function to replace ls and sort in child & parent process.
here is the sample code
int main(void) {
int p[2];
pipe(p);
char *arg[] = {"ls","sort",NULL};
if(fork()==0) {
close(0);/* close the stdin stream so that this
process shoulbn't read from stdin */
dup(p[0]);/* read from read end of pipe */
close(p[1]);
execlp(arg[1],arg[1],(char*)NULL);
}
else{
close(1);/* close the stdout stream, so that o/p shouldn't print on monitor */
dup(p[1]); /* stdout result redrecting to write end of pipe */
close(p[0]);
execlp(arg[0],arg[0],(char*)NULL);
}
return 0;
}
| is a shell feature. You need to do the same thing the shell does, i.e. use pipe, fork, dup2, execvp to create a pipe, spawn a new process, and connect the pipe to the processes' stdin and stdout, respectively.
Take the code from your execString function but replace the else block. In the parent process you should dup2(link[0], 0) (connect the read end of your pipe to stdin), then execvp the other program (e.g. sort).
Remember to close the pipe ends you don't need! I.e. in the producer process, close(link[0]); in the consumer process, close(link[1]). If you forget this part, things may get "stuck" (commands seemingly hanging forever).
Of course, if you want your original program to keep going, you need to wrap that code inside another fork (and waitpid for it).
As an aside, your execString is broken if the command outputs a lot of data. At some point the pipe will get full and the command will pause, waiting for another process to drain the pipe (by reading from it). Your program will be stuck in wait, waiting for the command to terminate. The result is deadlock.
To fix this issue, only call wait after you're done reading from the pipe.
I'll post my code first, then explain the problem I'm having:
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <fcntl.h>
#define MAX_ARGS 20
#define BUFSIZE 1024
int get_args(char* cmdline, char* args[])
{
int i = 0;
/* if no args */
if((args[0] = strtok(cmdline, "\n\t ")) == NULL)
return 0;
while((args[++i] = strtok(NULL, "\n\t ")) != NULL) {
if(i >= MAX_ARGS) {
printf("Too many arguments!\n");
exit(1);
}
}
/* the last one is always NULL */
return i;
}
void execute(char* cmdline)
{
int pid, async, oneapp;
char* args[MAX_ARGS];
char* args2[] = {"-l", NULL};
int nargs = get_args(cmdline, args);
if(nargs <= 0) return;
if(!strcmp(args[0], "quit") || !strcmp(args[0], "exit")) {
exit(0);
}
printf("before the if\n");
printf("%s\n",args[nargs - 2]);
int i = 0;
// EDIT: THIS IS WHAT WAS SUPPOSED TO BE COMMENTED OUT
/*
while (args[i] != ">" && i < nargs - 1) {
printf("%s\n",args[i]);
i++;
}
*/
// Presence of ">" token in args
// causes errors in execvp() because ">" is not
// a built-in Unix command, so remove it from args
args[i - 1] = NULL;
printf("Escaped the while\n");
// File descriptor array for the pipe
int fd[2];
// PID for the forked process
pid_t fpid1;
// Open the pipe
pipe(fd);
// Here we fork
fpid1 = fork();
if (fpid1 < 0)
{
// The case where the fork fails
perror("Fork failed!\n");
exit(-1);
}
else if (fpid1 == 0)
{
//dup2(fd[1], STDOUT_FILENO);
close(fd[1]);
//close(fd[0]);
// File pointer for the file that'll be written to
FILE * file;
// freopen() redirects stdin to args[nargs - 1],
// which contains the name of the file we're writing to
file = freopen(args[nargs - 1], "w+", stdin);
// If we include this line, the functionality works
//execvp(args[0],args);
// We're done writing to the file, so close it
fclose(file);
// We're done using the pipe, so close it (unnecessary?)
//close(fd[1]);
}
else
{
// Wait for the child process to terminate
wait(0);
printf("This is the parent\n");
// Connect write end of pipe (fd[1]) to standard output
dup2(fd[1], STDOUT_FILENO);
// We don't need the read end, so close it
close(fd[0]);
// args[0] contains the command "ls", which is
// what we want to execute
execvp(args[0], args);
// This is just a test line I was using before to check
// whether anything was being written to stdout at all
printf("Exec was here\n");
}
// This is here to make sure program execution
// doesn't continue into the original code, which
// currently causes errors due to incomplete functionality
exit(0);
/* check if async call */
printf("Async call part\n");
if(!strcmp(args[nargs-1], "&")) { async = 1; args[--nargs] = 0; }
else async = 0;
pid = fork();
if(pid == 0) { /* child process */
execvp(args[0], args);
/* return only when exec fails */
perror("exec failed");
exit(-1);
} else if(pid > 0) { /* parent process */
if(!async) waitpid(pid, NULL, 0);
else printf("this is an async call\n");
} else { /* error occurred */
perror("fork failed");
exit(1);
}
}
int main (int argc, char* argv [])
{
char cmdline[BUFSIZE];
for(;;) {
printf("COP4338$ ");
if(fgets(cmdline, BUFSIZE, stdin) == NULL) {
perror("fgets failed");
exit(1);
}
execute(cmdline) ;
}
return 0;
}
So, what's the problem? Simple: the code above creates a file with the expected name, i.e. the name provided in the command line, which gets placed at args[nargs - 1]. For instance, running the program and then typing
ls > test.txt
Creates a file called test.txt... but it doesn't actually write anything to it. I did manage to get the program to print garbage characters to the file more than a few times, but this only happened during bouts of desperate hail mary coding where I was basically just trying to get the program to write SOMETHING to the file.
I do think I've managed to narrow down the cause of the problems to this area of the code:
else if (fpid1 == 0)
{
printf("This is the child.\n");
//dup2(fd[1], STDOUT_FILENO);
close(fd[1]);
//close(fd[0]);
// File pointer for the file that'll be written to
FILE * file;
// freopen() redirects stdin to args[nargs - 1],
// which contains the name of the file we're writing to
file = freopen(args[nargs - 1], "w+", stdout);
// If we include this line, the functionality works
//execvp(args[0],args);
// We're done writing to the file, so close it
fclose(file);
// We're done using the pipe, so close it (unnecessary?)
//close(fd[1]);
}
else
{
// Wait for the child process to terminate
wait(0);
printf("This is the parent\n");
// Connect write end of pipe (fd[1]) to standard output
dup2(fd[1], STDOUT_FILENO);
// We don't need the read end, so close it
close(fd[0]);
// args[0] contains the command "ls", which is
// what we want to execute
execvp(args[0], args);
// This is just a test line I was using before to check
// whether anything was being written to stdout at all
printf("Exec was here\n");
}
More specifically, I believe the problem is with the way I'm using (or trying to use) dup2() and the piping functionality. I basically found this out by process of elimination. I spent a few hours commenting things out, moving code around, adding and removing test code, and I've found the following things:
1.) Removing the calls to dup2() and using execvp(args[0], args) prints the result of the ls command to the console. The parent and child processes begin and end properly. So, the calls to execvp() are working properly.
2.) The line
file = freopen(args[nargs - 1], "w+", stdout)
Successfully creates a file with the correct name, so the call to freopen() isn't failing. While this doesn't immediately prove that this function is working properly as it's written now, consider fact #3:
3.) In the child process block, if we make freopen redirect to the output file from stdin (rather than stdout) and uncomment the call to execvp(args[0], args), like so:
// freopen() redirects stdin to args[nargs - 1],
// which contains the name of the file we're writing to
file = freopen(args[nargs - 1], "w+", stdin);
// If we include this line, the functionality works
execvp(args[0],args);
and run the program, then it works and result of the ls command is successfully written to the output file. Knowing this, it seems pretty safe to say that freopen() isn't the problem either.
In other words, the only thing I haven't been able to successfully do is pipe the output of the execvp() call that's done in the parent process to stdout, and then from stdout to the file using freopen().
Any help is appreciated. I've been at this since 10 AM yesterday and I'm completely out of ideas. I just don't know what I'm doing wrong. Why isn't this working?
Iam trying to create a code with 1 parent and 2 childrens. The method recive 3 parameters:
original_file word1 word2
The parent read a file line by line:
If the line is pair, send the line to method proccess_pair and the word1.
If the line contains the word1, save the lines in the file_1.txt
If the line is odd, send the line to method proccess_odd and the word2.
If the line contains the word1, save the lines in the file_2.txt
Im beginner in c, and i trying with this:
int p_h1[2] // pipe from parent to child1
int p_h2[2];// pipe from parent to child2
int main(int argc, char **argv){
pid_t pdi1, pdi2;
FILE *fd; // for original file
FILE *p_h1f, *p_h2f; //file create for child1 and child2 respectively
char buffer[1024];//buffer
if (pid1<0){
fprintf(stderr,"Error fork \n %s \n",strerror(errno));
exit(EXIT_FAILURE);
}
else if (pid1==0){//Im the child1
//proccess for child 1
proccess_pair(arg[2]);
exit(EXIT_SUCCESS);
}
pid2 = fork();
if (pid2<0){
fprintf(stderr,"Error fork \n %s \n",strerror(errno));
exit(EXIT_FAILURE);
}
else if (pid2==0){//Im the child2
//proccess for child 2
proccess_odd(arg[2]);
exit(EXIT_SUCCESS);
}
//Parent dont read from pipe
close(p_h1[0]);
close(p_h2[0]);
fd = fopen(argv[1],"r"); //I openthe file for read it;
p_h1f = fdopen(p_h1[1],"w")
p_h2f = fdopen(p_h2[1],"w")
int i = 1;
while(fgets(buffer,1024,fd) != NULL){
if (i % 2 ==0){ //check if the lines is pairs
fputs(buffer,p_h1f);
fflush(p_h1f);
}else{
fputs(buffer,p_h2f);
fflush(p_h2f);
}
i++;
}
close(p_h1[1]);
close(p_h2[1]);
fclose(fd);
wait(NULL);
wait(NULL);
}
Both methods(for chil1 and chil2) will be the same(but closing the correct sides of pipes), for this reason i only implement one of them:
void proccess_pair(char *word1){
FILE *fd;
fd = fopen("file_1.txt","w");
//closing the not used
close(p_h1[1]);
close(p_h2[1]);
close(p_h2[0]);
int nsto = dup(1)//duplicate the stdout
dup2(fd,1);//changing stdout->file_1.txt
execlp("grep","grep",word1,NULL);//execution of grep
exit(EXIT_SUCCESS);
}
Im learning and i know that i have many many error, for this reason i need help.
Regards
How i can create many pipes in an array in c?
On a POSIX-conformant system, you can do that by calling pipe() many times on elements of a 2D array of int, as you presented.
¿I can use two different pipes(parent-child1,parent-child2)? I can use an array of pipes?
A pipe itself lives only in the kernel. There is no userspace data
structure representing a pipe, so you cannot have an array of pipes.
The file descriptors for the pipe ends, however, are just ints. The pipe() function takes as its argument a pointer to the first element of an array of at least two int, and (on success) it writes the appropriate file descriptors into the array.
From a C perspective, there is nothing special about the array in which the pipe ends are to be returned. In particular, it can be an element of a multi-dimensional array if you so desire. Or it can be a local variable. Or it can be a member of a struct or a union. Or it can be a large enough block of dynamically-allocated space. It's not special.
Something like this should work:
int new_process(char *word){ // return the writing part of a pipe to a newly created child
int p[2];
pipe( p ); // get a new pipe
if (fork()==0) { // create a new child
dup2(p[0],0); // child's in is pipe's entry
close(p[1]); // close writing part
execlp("grep","grep",word,NULL); // exec
}
close(p[0]); // parent don't need the reading part
return p[1]; // send back the writing part to caller
}
int main(int argc, char **argv) {
int f1 = new_process(argv[1]); // get the pipe to first child
int f2 = new_process(argv[1]); // ...second...
char buffer[1024];//buffer
FILE *fd = fopen(argv[1],"r"); // open the file for reading;
while(fgets(buffer,1024,fd) != NULL) { // read aline
if (i % 2 ==0) { //check if the line no is pair
fputs(buffer,f1); // send it to first child
fflush(f1);
} else{
fputs(buffer,f2); // send it to second child
fflush(f2);
}
i++;
}
close(f1);
close(f2);
fclose(fd);
wait(NULL);
wait(NULL);
}
Don't forget to add necessary controls on failures.
I have a task in Linux and I can't get it work.
I have a program that receives a text file as parameter. It then creates a child process using fork() and sends to the child process, line by line the content of the text file received as parameter. The child process needs to count the lines and return to the parent process the number of lines received.
This is what I have until now, but somewhat the child process does not receive all the lines. For my test I used a text file with 9 lines. The parent sent 9 lines as strings but the child process received only 2 or 3 of them.
What am I doing wrong?
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char *argv[])
{
char string[80];
char readbuffer[80];
int pid, p[2];
FILE *fp;
int i=0;
if(argc != 2)
{
printf("Syntax: %s [file_name]\n", argv[0]);
return 0;
}
fp = fopen(argv[1], "r");
if(!fp)
{
printf("Error: File '%s' does not exist.\n", argv[1]);
return 0;
}
if(pipe(p) == -1)
{
printf("Error: Creating pipe failed.\n");
exit(0);
}
// creates the child process
if((pid=fork()) == -1)
{
printf("Error: Child process could not be created.\n");
exit(0);
}
/* Main process */
if (pid)
{
// close the read
close(p[0]);
while(fgets(string,sizeof(string),fp) != NULL)
{
write(p[1], string, (strlen(string)+1));
printf("%s\n",string);
}
// close the write
close(p[1]);
wait(0);
}
// child process
else
{
// close the write
close(p[1]);
while(read(p[0],readbuffer, sizeof(readbuffer)) != 0)
{
printf("Received string: %s\n", readbuffer);
}
// close the read
close(p[0]);
}
fclose(fp);
}
A pipe is a unidirectional interprocess communication channel. You have to create 2 pipes, one to speak to the child process, the other to read data back.
Remember to close the unused side of the pipe on both processes.
You are sending the null terminator to the other process:
write(p[1], string, (strlen(string)+1));
That makes the result confusing because when you print what you've received, you only see up to the null terminator.
If you do this instead:
write(p[1], string, strlen(string));
you should get what you expect.
You're not counting the number of lines, you're counting the number of times read(2) returns.
When using pipes, read(2) will pull as much data as possible from the pipe: min(pipe_available, space_available). It doesn't care for newlines, 0 bytes etc. Simple tricks to make it work:
Use a loop to walk readbuffer and look for \n
Use fdopen + fgets (I have a feeling this is probably flawed)
look into manpage of pipe ( man 2 pipe ), the program you're trying to write is as an example there, compare it with yours :)