I am trying to implement a shell in C. I can execute simple commands just fine with a simple execvp() but one of the requirements is to manage commands like this: "ls -l | head | tail -4" with a 'for' loop and only one 'pipe()' statement redirecting stdin and stdout. Now after days I'm a bit lost.
N = Number of simple commands (3 in the example: ls, head, tail)
commands = a list of structs with the commands, like this:
commands[0].argv[0]: ls
commands[0].argv[1]: -l
commands[1].argv[0]: head
commands[2].argv[0]: tail
commands[2].argv[1]: -4
So, I made the for loop, and started to redirect stdin and stdout in order to connect all the commands with pipes, but...I'm just clueless why it doesn't work.
for (i=0; i < n; i++){
pipe(pipe);
if(fork()==0){ // CHILD
close(pipe[0]);
close(1);
dup(pipe[1]);
close(pipe[1]);
execvp(commands[i].argv[0], &commands[i].argv[0]);
perror("ERROR: ");
exit(-1);
}else{ // FATHER
close(pipe[1]);
close(0);
dup(pipe[0]);
close(pipe[0]);
}
}
What I want to create is a 'line' of childed processes:
[ls -l] ----pipe----> [head] ----pipe----> [tail -4]
All this processes have a root (the process runing my shell) so, the first father is also a child of the shell process, I'm a bit exhausted already, can anyone help me here please?
I'm not even sure if the childs should be the ones executing the commands.
Thanks guys !!
Nothing complex here, just have in mind that the last command should output to the original process' file descriptor 1 and the first should read from original process file descriptor 0. You just spawn the processes in order, carrying along the input side of the previous pipe call.
So, here's are the types:
#include <unistd.h>
struct command
{
const char **argv;
};
Make a helper function with a simple well defined semantics:
int
spawn_proc (int in, int out, struct command *cmd)
{
pid_t pid;
if ((pid = fork ()) == 0)
{
if (in != 0)
{
dup2 (in, 0);
close (in);
}
if (out != 1)
{
dup2 (out, 1);
close (out);
}
return execvp (cmd->argv [0], (char * const *)cmd->argv);
}
return pid;
}
And here's the main fork routine:
int
fork_pipes (int n, struct command *cmd)
{
int i;
pid_t pid;
int in, fd [2];
/* The first process should get its input from the original file descriptor 0. */
in = 0;
/* Note the loop bound, we spawn here all, but the last stage of the pipeline. */
for (i = 0; i < n - 1; ++i)
{
pipe (fd);
/* f [1] is the write end of the pipe, we carry `in` from the prev iteration. */
spawn_proc (in, fd [1], cmd + i);
/* No need for the write end of the pipe, the child will write here. */
close (fd [1]);
/* Keep the read end of the pipe, the next child will read from there. */
in = fd [0];
}
/* Last stage of the pipeline - set stdin be the read end of the previous pipe
and output to the original file descriptor 1. */
if (in != 0)
dup2 (in, 0);
/* Execute the last stage with the current process. */
return execvp (cmd [i].argv [0], (char * const *)cmd [i].argv);
}
And a small test:
int
main ()
{
const char *ls[] = { "ls", "-l", 0 };
const char *awk[] = { "awk", "{print $1}", 0 };
const char *sort[] = { "sort", 0 };
const char *uniq[] = { "uniq", 0 };
struct command cmd [] = { {ls}, {awk}, {sort}, {uniq} };
return fork_pipes (4, cmd);
}
Appears to work. :)
First, you are prematurely closing the pipes. Close only the end that you don't need in the current process, and remember to close stdin/stdout in the child.
Secondly, you need to remember the fd from the previous command. So, for two processes, this looks like:
int pipe[2];
pipe(pipe);
if ( fork() == 0 ) {
/* Redirect output of process into pipe */
close(stdout);
close(pipe[0]);
dup2( pipe[1], stdout );
execvp(commands[0].argv[0], &commands[0].argv[0]);
}
if ( fork() == 0 ) {
/* Redirect input of process out of pipe */
close(stdin);
close(pipe[1]);
dup2( pipe[0], stdin );
execvp(commands[1].argv[0], &commands[1].argv[0]);
}
/* Main process */
close( pipe[0] );
close( pipe[1] );
waitpid();
Now your job is to add error handling to this and generate n-1 pipes for n processes to start. The code in the first fork() block needs to be run for the appropriate pipe for processes 1..n-1, and the code in the second fork() block for the processes 2..n.
Related
I'm writing a shell in C and am trying to implement multiple pipes. I've done this by creating a two dimensional array with pipes and using a separate pipe everytime. All commands in between pipes are separated by a parsing function and put into a struct. Every command line in-between pipes gets it's own process. And for all commands in the middle I'm trying to read from the previous process and write to the next one. Somewhere here the problem starts. It works fine for one pipe, however when I trying more than one pipe I don't get any output and the program gets stuck. In GDB I get a failed message from the execvp after forking the second process. What can this be due to?
int create_pipe(int* fd)
{
int pipe_id = pipe(fd);
if (pipe_id == -1)
{
return -1;
}
return 0;
}
void write_pipe(int* fd)
{
close(fd[READ]);
if ((dup2(fd[WRITE], STDOUT_FILENO)) < -1)
{
fork_error();
}
close(fd[WRITE]);
}
void read_pipe(int *fd)
{
close(fd[WRITE]);
if (dup2(fd[READ], STDIN_FILENO) < 0)
{
fork_error();
}
close(fd[READ]);
}
void need_to_pipe (int i, int (*fd)[2])
{
if (commands[i].pos == first)
{
write_pipe(fd[i * 2]);
}
else if (commands[i].pos == last)
{
read_pipe(fd[(i-1) *2]);
}
else //if (commands[i].pos == middle)
{
dup2(fd[(i-1)*2][READ], STDIN_FILENO);
close(fd[(i-1)*2][READ]);
close(fd[(i-1)*2][WRITE]);
//close(fd[(i)*2][READ]);
//close(fd[(i)*2][WRITE]);
close(fd[(i)*2][READ]);
dup2(fd[i*2][WRITE], STDOUT_FILENO);
close(fd[(i)*2][WRITE]);
}
}
/**
* Fork a proccess for command with index i in the command pipeline. If needed,
* create a new pipe and update the in and out members for the command..
*/
void fork_cmd(int i, int (*fd)[2]) {
pid_t pid;
switch (pid = fork()) {
case -1:
fork_error();
case 0:
// Child process after a successful fork().
if (!(commands[i].pos == single))
{
need_to_pipe(i, fd);
}
// Execute the command in the contex of the child process.
if (execvp(commands[i].argv[0], commands[i].argv)<0)
{
fprintf(stderr, "command not found: %s\n",
commands[i].argv[0]);
exit(EXIT_FAILURE);
}
default:
// Parent process after a successful fork().
break;
}
}
/**
* Fork one child process for each command in the command pipeline.
*/
void fork_cmds(int n, int (*fd)[2])
{
for (int i = 0; i < n; i++)
{
fork_cmd(i, fd);
}
}
void wait_once ()
{
wait(NULL);
}
/**
* Make the parents wait for all the child processes.
*/
void wait_for_all_cmds(int n)
{
for (int i = 0; i < n; i++)
{
wait_once();
//wait for number of child processes.
}
}
int main() {
int n; // Number of commands in a command pipeline.
size_t size = 128; // Max size of a command line string.
char line[size];
while(true) {
// Buffer for a command line string.
printf(" >>> ");
get_line(line, size);
n = parse_cmds(line, commands);
int fd[(n-1)][2];
for(int i =0;i<n-1;i++)
{
int pipe_id = pipe(fd[i*2]);
if (pipe_id == -1)
{
return -1;
}
}
fork_cmds(n, fd);
for(int i =0;i<n-1;i++)
{
int *fdclose= fd[i*2];
close (fdclose[READ]);
close (fdclose[WRITE]);
}
wait_for_all_cmds(n);
}
exit(EXIT_SUCCESS);
}
You [probably] have too many processes keeping pipe ends open (that do not belong to the given child) because your loop opens all pipes before any forking.
This places an undue burden on each child because it has to close many pipe ends to prevent it from holding open a pipe end, preventing other children from seeing an EOF on their input pipes.
To see this, for debug purposes in your present code, the child could do (just before the exec* call) (e.g.):
fprintf(stderr,"child: %d\n",getpid());
fflush(stderr);
system("ls -l /proc/self/fd 1>&2");
Each child should only have three open streams on stdin, stdout, and stderr (e.g. 0, 1, 2).
I think you'd find that there are many extraneous/detrimental streams open on the various children.
You only need two pipe arrays (e.g.): int pipeinp[2]; int pipeout[2]; Initially, pipeinp is all -1.
Roughly ...
Parent should do a single pipe call at the top of fork_cmd [before the fork] to pipeout.
The child dups (and closes) the read end of pipeinp [if not -1] to stdin.
Child dups/closes the write end of pipeout to stdout.
It closes the read end of pipeout.
After that, the parent should copy pipeout to pipeinp and close the write end of pipeinp
This should be repeated for all pipe stages.
No pipe to pipeout should be done for the last command. And, the [last] child should not change stdout.
For a working example, see my answer: fd leak, custom Shell
I'm trying to execute ls | wc -l through a program in C, instead of using the command line.
This is my current working code:
int main() {
int pfds[2];
pipe(pfds);
pid_t pid = fork();
if ( pid == 0 ) { /* The child process*/
close(1);
dup(pfds[1]);
close(pfds[0]);
execlp("ls", "ls", NULL);
} else { /* The parent process*/
close(0);
dup(pfds[0]);
close(pfds[1]);
wait(0);
execlp("wc", "wc", "-l", NULL);
}
return 0;
}
How would I rewrite this code to work with a for-loop?
For example:
for (i=0; i<2; i++) {
// Rewrite the 2-level pipe here
}
Later, I would like to extend the for loop to execute more processes piped together like a | b | c | ...
In order to pipe multiple commands together, you'll need to keep the parent running to keep fork()ing for each command.
Using a for loop, you will need to do this for the first n - 1 commands (the last one will be executed in the main program):
Create a pipe.
Execute fork().
In the child: overwrite standard input with the read end of the previous pipe, and standard output with the write end of the current pipe.
In the child: execute execve().
In the parent: close unneeded pipes and save read end of current pipe to be used in the next iteration.
Then, after the loop ends, overwrite standard input with the read end of the last pipe and execute execve() of the last command.
Below I've written a simple working example that executes:
ls | wc -l | xargs printf "0x%x\n" | cowsay
It should work for any number of commands (including only 1 single command).
NOTE: I did not add error checks in this code apart for execvp() just to make it short, but you should definitely check for errors after each call to pipe(), dup2(), fork() and any other function.
Code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define MAX_ARGC 3
int main(void) {
char *commands[][MAX_ARGC + 1] = {
{"ls", NULL},
{"wc", "-l", NULL},
{"xargs", "printf", "0x%x\n", NULL},
{"cowsay", NULL}
};
size_t i, n;
int prev_pipe, pfds[2];
n = sizeof(commands) / sizeof(*commands);
prev_pipe = STDIN_FILENO;
for (i = 0; i < n - 1; i++) {
pipe(pfds);
if (fork() == 0) {
// Redirect previous pipe to stdin
if (prev_pipe != STDIN_FILENO) {
dup2(prev_pipe, STDIN_FILENO);
close(prev_pipe);
}
// Redirect stdout to current pipe
dup2(pfds[1], STDOUT_FILENO);
close(pfds[1]);
// Start command
execvp(commands[i][0], commands[i]);
perror("execvp failed");
exit(1);
}
// Close read end of previous pipe (not needed in the parent)
close(prev_pipe);
// Close write end of current pipe (not needed in the parent)
close(pfds[1]);
// Save read end of current pipe to use in next iteration
prev_pipe = pfds[0];
}
// Get stdin from last pipe
if (prev_pipe != STDIN_FILENO) {
dup2(prev_pipe, STDIN_FILENO);
close(prev_pipe);
}
// Start last command
execvp(commands[i][0], commands[i]);
perror("execvp failed");
exit(1);
}
Output on my machine (since ls returned 41 == 0x29 lines):
______
< 0x29 >
------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
I'm trying to execute ls | wc -l through a program in C, instead of using the command line.
This is my current working code:
int main() {
int pfds[2];
pipe(pfds);
pid_t pid = fork();
if ( pid == 0 ) { /* The child process*/
close(1);
dup(pfds[1]);
close(pfds[0]);
execlp("ls", "ls", NULL);
} else { /* The parent process*/
close(0);
dup(pfds[0]);
close(pfds[1]);
wait(0);
execlp("wc", "wc", "-l", NULL);
}
return 0;
}
How would I rewrite this code to work with a for-loop?
For example:
for (i=0; i<2; i++) {
// Rewrite the 2-level pipe here
}
Later, I would like to extend the for loop to execute more processes piped together like a | b | c | ...
In order to pipe multiple commands together, you'll need to keep the parent running to keep fork()ing for each command.
Using a for loop, you will need to do this for the first n - 1 commands (the last one will be executed in the main program):
Create a pipe.
Execute fork().
In the child: overwrite standard input with the read end of the previous pipe, and standard output with the write end of the current pipe.
In the child: execute execve().
In the parent: close unneeded pipes and save read end of current pipe to be used in the next iteration.
Then, after the loop ends, overwrite standard input with the read end of the last pipe and execute execve() of the last command.
Below I've written a simple working example that executes:
ls | wc -l | xargs printf "0x%x\n" | cowsay
It should work for any number of commands (including only 1 single command).
NOTE: I did not add error checks in this code apart for execvp() just to make it short, but you should definitely check for errors after each call to pipe(), dup2(), fork() and any other function.
Code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define MAX_ARGC 3
int main(void) {
char *commands[][MAX_ARGC + 1] = {
{"ls", NULL},
{"wc", "-l", NULL},
{"xargs", "printf", "0x%x\n", NULL},
{"cowsay", NULL}
};
size_t i, n;
int prev_pipe, pfds[2];
n = sizeof(commands) / sizeof(*commands);
prev_pipe = STDIN_FILENO;
for (i = 0; i < n - 1; i++) {
pipe(pfds);
if (fork() == 0) {
// Redirect previous pipe to stdin
if (prev_pipe != STDIN_FILENO) {
dup2(prev_pipe, STDIN_FILENO);
close(prev_pipe);
}
// Redirect stdout to current pipe
dup2(pfds[1], STDOUT_FILENO);
close(pfds[1]);
// Start command
execvp(commands[i][0], commands[i]);
perror("execvp failed");
exit(1);
}
// Close read end of previous pipe (not needed in the parent)
close(prev_pipe);
// Close write end of current pipe (not needed in the parent)
close(pfds[1]);
// Save read end of current pipe to use in next iteration
prev_pipe = pfds[0];
}
// Get stdin from last pipe
if (prev_pipe != STDIN_FILENO) {
dup2(prev_pipe, STDIN_FILENO);
close(prev_pipe);
}
// Start last command
execvp(commands[i][0], commands[i]);
perror("execvp failed");
exit(1);
}
Output on my machine (since ls returned 41 == 0x29 lines):
______
< 0x29 >
------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
I'm trying to make ls | tr a b > text.txt
I have piping done, but I can't add STDOUT to the end of the pipe (STDOUT in my case can be only in the last argument)
I mark the part of the code, in which redirection should be done, I think that file should be opened, and dup2 method used, but I don't know in which way
Methods contains piping -
enum reqType { PIPE, STDOUT };
int spawn_proc (int in, int out, char** cmd) {
pid_t pid;
if ((pid = fork ()) == 0) {
if (in != 0) {
dup2 (in, 0);
close (in);
}
if (out != 1) {
dup2 (out, 1);
close (out);
}
return execvp (cmd[0], cmd);
}
return pid;
}
void fork_pipes (int n, char** cmd[], enum reqType type) {
int i;
pid_t pid;
int in, fd [2];
in = 0;
for (i = 0; i < n - 1; ++i) {
if(type == PIPE || i < n-2) {
pipe (fd);
spawn_proc (in, fd [1], cmd[i]);
close (fd [1]);
in = fd [0];
}
else if(type == STDOUT && i == n-2) {
///HOW TO IMPLEMENT THIS PART?
}
}
if (in != 0)
dup2 (in, 0);
execvp (cmd[i][0], cmd[i]);
}
EDIT
in the marked by /// place I wrote
pipe(fd);
int out = open(cmd[n-1][0],O_WRONLY|O_CREAT|O_TRUNC);
spawn_proc(in, out, cmd[i]);
close(fd[1]);
I think that file should be opened, and dup2 method used, but I don't know in which way
You are right about the mechanisms for implementing the redirection. It should be done on the process intended for tr, and before performing the overlay.
Let's go step by step:
ls | tr a b > text.txt
First create a pipe, then fork().
From now on, there are two processes running in parallel, both of them will be eventually overlaid by means of exec(): one with the ls program, the other with the tr program.
Process for ls:
Close the reading end of the pipe: this process will only write to the pipe.
dup2() the writing end of the pipe to STDOUT: what this process writes to STDOUT is being written to the pipe.
Perform the overlay: exec() with ls.
Process for tr:
Close the writing end of the pipe: this process will only read from the pipe.
dup2() the reading end of the pipe to STDIN: what this process reads from STDIN is coming from the pipe.
In order to perform the redirection to the text.txt file, first open() the file text.txt for writing and with the flags O_CREAT and O_TRUNC, then dup2() the obtained file descriptor to STDOUT.
Perform the overlay: exec() with tr.
Note that, if the command were appending to text.txt instead of truncating it (i.e.: using >> instead of >):
ls | tr a b >> text.txt
You would have to use the flag O_APPEND instead of O_TRUNC when open()ing the text.txt file.
Code Snippet
I've modified your code (also the interface of fork_pipes()). It's a minimal example that runs, I hope it helps.
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int spawn_proc (int in, int out, char** cmd) {
pid_t pid;
if ((pid = fork ()) == 0) {
if (in != 0) {
dup2 (in, 0);
close (in);
}
if (out != 1) {
dup2 (out, 1);
close (out);
}
return execvp (cmd[0], cmd);
}
return pid;
}
void fork_pipes (char** cmd[], const char *redirection) {
int i, n;
int in, out, fd[2];
in = 0;
// obtain n from the NULL terminated cmd array
for (n = 0; cmd[n]; ++n)
;
// process all but the last elemet of the pipe
for (i = 0; i < n-1; ++i) {
pipe(fd);
spawn_proc(in, fd[1], cmd[i]);
close(fd [1]);
in = fd [0];
}
// process the last element of the pipe
if (redirection) {
out = open(redirection, O_WRONLY | O_CREAT | O_TRUNC);
fchmod(out, 0666);
} else
out = STDOUT_FILENO;
if (in != 0)
dup2(in, 0);
spawn_proc(in, out, cmd[i]);
}
int main()
{
char *cmd1[] = {"ls", NULL};
char *cmd2[] = {"tr", "a", "b", NULL};
char **cmd[] = { cmd1, cmd2, NULL};
// redirected to text.txt
fork_pipes(cmd, "text.txt");
// no redirection
fork_pipes(cmd, NULL);
// another example with a longer pipe
{
char *cmd1[] = {"echo", "hello world", NULL};
char *cmd2[] = {"tee", NULL};
char *cmd3[] = {"tee", NULL};
char *cmd4[] = {"tr", "lo", "10", NULL};
char **cmd[] = {cmd1, cmd2, cmd3, cmd4, NULL};
// redirected to redirection.txt
fork_pipes(cmd, "redirection.txt");
// no redirected
fork_pipes(cmd, NULL);
}
return 0;
}
As already pointed out in this comment. You just need to call pipe() once in your example: The pipe() system call only needs to be called once for each pipe operator (i.e.: the | character) found in the compound command. For example, in the following command:
cmd1 | cmd2 | cmd3 | cmd4
pipe() must be called exactly four times, since there are four pipe operators.
I am trying to implement a shell in C. I can execute simple commands just fine with a simple execvp() but one of the requirements is to manage commands like this: "ls -l | head | tail -4" with a 'for' loop and only one 'pipe()' statement redirecting stdin and stdout. Now after days I'm a bit lost.
N = Number of simple commands (3 in the example: ls, head, tail)
commands = a list of structs with the commands, like this:
commands[0].argv[0]: ls
commands[0].argv[1]: -l
commands[1].argv[0]: head
commands[2].argv[0]: tail
commands[2].argv[1]: -4
So, I made the for loop, and started to redirect stdin and stdout in order to connect all the commands with pipes, but...I'm just clueless why it doesn't work.
for (i=0; i < n; i++){
pipe(pipe);
if(fork()==0){ // CHILD
close(pipe[0]);
close(1);
dup(pipe[1]);
close(pipe[1]);
execvp(commands[i].argv[0], &commands[i].argv[0]);
perror("ERROR: ");
exit(-1);
}else{ // FATHER
close(pipe[1]);
close(0);
dup(pipe[0]);
close(pipe[0]);
}
}
What I want to create is a 'line' of childed processes:
[ls -l] ----pipe----> [head] ----pipe----> [tail -4]
All this processes have a root (the process runing my shell) so, the first father is also a child of the shell process, I'm a bit exhausted already, can anyone help me here please?
I'm not even sure if the childs should be the ones executing the commands.
Thanks guys !!
Nothing complex here, just have in mind that the last command should output to the original process' file descriptor 1 and the first should read from original process file descriptor 0. You just spawn the processes in order, carrying along the input side of the previous pipe call.
So, here's are the types:
#include <unistd.h>
struct command
{
const char **argv;
};
Make a helper function with a simple well defined semantics:
int
spawn_proc (int in, int out, struct command *cmd)
{
pid_t pid;
if ((pid = fork ()) == 0)
{
if (in != 0)
{
dup2 (in, 0);
close (in);
}
if (out != 1)
{
dup2 (out, 1);
close (out);
}
return execvp (cmd->argv [0], (char * const *)cmd->argv);
}
return pid;
}
And here's the main fork routine:
int
fork_pipes (int n, struct command *cmd)
{
int i;
pid_t pid;
int in, fd [2];
/* The first process should get its input from the original file descriptor 0. */
in = 0;
/* Note the loop bound, we spawn here all, but the last stage of the pipeline. */
for (i = 0; i < n - 1; ++i)
{
pipe (fd);
/* f [1] is the write end of the pipe, we carry `in` from the prev iteration. */
spawn_proc (in, fd [1], cmd + i);
/* No need for the write end of the pipe, the child will write here. */
close (fd [1]);
/* Keep the read end of the pipe, the next child will read from there. */
in = fd [0];
}
/* Last stage of the pipeline - set stdin be the read end of the previous pipe
and output to the original file descriptor 1. */
if (in != 0)
dup2 (in, 0);
/* Execute the last stage with the current process. */
return execvp (cmd [i].argv [0], (char * const *)cmd [i].argv);
}
And a small test:
int
main ()
{
const char *ls[] = { "ls", "-l", 0 };
const char *awk[] = { "awk", "{print $1}", 0 };
const char *sort[] = { "sort", 0 };
const char *uniq[] = { "uniq", 0 };
struct command cmd [] = { {ls}, {awk}, {sort}, {uniq} };
return fork_pipes (4, cmd);
}
Appears to work. :)
First, you are prematurely closing the pipes. Close only the end that you don't need in the current process, and remember to close stdin/stdout in the child.
Secondly, you need to remember the fd from the previous command. So, for two processes, this looks like:
int pipe[2];
pipe(pipe);
if ( fork() == 0 ) {
/* Redirect output of process into pipe */
close(stdout);
close(pipe[0]);
dup2( pipe[1], stdout );
execvp(commands[0].argv[0], &commands[0].argv[0]);
}
if ( fork() == 0 ) {
/* Redirect input of process out of pipe */
close(stdin);
close(pipe[1]);
dup2( pipe[0], stdin );
execvp(commands[1].argv[0], &commands[1].argv[0]);
}
/* Main process */
close( pipe[0] );
close( pipe[1] );
waitpid();
Now your job is to add error handling to this and generate n-1 pipes for n processes to start. The code in the first fork() block needs to be run for the appropriate pipe for processes 1..n-1, and the code in the second fork() block for the processes 2..n.