I decided to write a simple, hard coded c program to better understand how pipes work.
The program has 3 commands:
find . -name '.' | sed 's/.*\.// | sort
And it works with 1 pipe (if i use only 2 commands) but it fails with 2 pipes(sort just does not get the information).
I think the error is in close or waitpid, but I have tried everything(i could think of) and it still does not work. What am I missing ?
information i used:
Is it possible to have pipe between two child processes created by same parent (LINUX, POSIX)
http://www.quora.com/Unix/How-can-I-write-a-code-for-PIPE-in-C-shell-script-python <--Sams example
Implementation of multiple pipes in C
EDIT:
the commands are written with no mistakes. The problem is definitely not in them (since they work if I only use 1 pipe)
My code:
#include <errno.h>
#include <stdio.h>
#include <unistd.h>
int main(int argc, char* argv[]) {
pid_t test, test2, test3;
int old_pipe[2];
int new_pipe[2];
//test command
//char *argv1[] = {"ls", "-1", "-h", NULL};
char *argv1[] = {"find", ".", "-name", "*.*",NULL};
char *argv2[] = {"sed", "s/.*\\.//", NULL};
char *argv3[] = {"sort", NULL};
//creates a pipe
pipe(old_pipe);
pipe(new_pipe);
test = fork();
if(test == 0){
dup2(old_pipe[1], 1);
close(old_pipe[0]);
execvp(argv1[0], argv1);
perror("exec");
return 1;
}
test2 = fork();
if(test2 == 0){
dup2(old_pipe[0], 0);
close(old_pipe[1]);
dup2(new_pipe[1], 1);
close(new_pipe[0]);
execvp(argv2[0], argv2);
perror("exec");
return 1;
}
test3 = fork();
if(test3 == 0){
dup2(new_pipe[0], 0);
close(new_pipe[1]);
execvp(argv3[0], argv3);
perror("exec");
return 1;
}
close(old_pipe[0]);
close(old_pipe[1]);
close(new_pipe[0]);
close(new_pipe[1]);
waitpid(test);
waitpid(test2);
waitpid(test3);
return 0;
}
your 3rd exec (starting "sort") does not close old_pipe[1] and keeps it open. sed will not see the EOF and stays alive. You should call pipe(2) very late.
I suggest to look into /proc/<pid>/fd resp. use lsof to see which filedescriptors are open. E.g. after
dup2(old_pipe[1], 1);
close(old_pipe[0]);
you should close old_pipe[1] when it is not 1.
Some more explanations (as asked in comment): You program has
pipe(old_pipe);
pipe(new_pipe);
test = fork();
/* --> 'find'; writes into old_pipe[1] */
test2 = fork();
/* --> 'sed'; reads from old_pipe[0], writes into new_pipe[1] */
test3 = fork();
/* --> 'sort'; reads from new_pipe[0] */
The test2 program does not exit as long as its input (old_pipe[0]) is open. Because this pipe is alive for test3 (which waits for test2 to finish), it will deadlock.
Closing fds in child branches does not close them in the parent process.
Related
I have the following code that forks and execvpe's a shell script and redirects its STDERR and STDOUT to the parent process.
#define _GNU_SOURCE
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#define BUFLEN 65536
int main () {
int pipes[2];
pipe(pipes);
pid_t pid = fork();
if (pid == -1) {return 1;}
else if (pid > 0) {
close(pipes[1]);
char buf[BUFLEN];
size_t n = read(pipes[0], buf, sizeof(buf));
int stat = 0;
waitpid(pid, &stat, 0);
printf("%.*s", n, buf);
} else {
dup2(pipes[1], STDOUT_FILENO);
dup2(pipes[1], STDERR_FILENO);
close(pipes[0]);
char *const argv[] = {"sh", "test", NULL};
execvpe(argv[0], argv, environ);
}
return 0;
}
As a minimal working example "test" is:
#!/bin/bash
cat file.txt
echo "Hello!"
echo "Goodbye!"
The output of the C program is the contents of file.txt and then the output from the echos are lost. If it's three echo statements then all of them get seen.
My best guess is that echo is a shell builtin and the shell will fork for cat and my pipes will be lost. In my project it seems the first command called in the script and the rest are lost.
If my assumption is correct how can I collect all the output from any and all children of what execvpe spawned?
I think the problem is simply a combination of timing and not checking for EOF before stopping reading from the pipe. If you wrap the read() call into a loop to read everything, all the data is read. When the cat completes, the read() returns everything that's available. The output from the echo commands is added to the pipe afterwards, but simply not read.
This code demonstrates, I believe. Note that execvpe() is not POSIX standard (and not available on macOS specifically) so I used my own surrogate header #include "execvpe.h" and implementation execvpe.c to obtain an implementation of it. Also, POSIX does not define a header that declares environ, so I declared it too. You're probably using Linux and the system headers there fix some gaps that POSIX leaves as holes.
Here's working code and data.
pipe17.c
/* SO 6412-3757 */
#define _GNU_SOURCE
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <stdio.h>
#define BUFLEN 65536
#include "execvpe.h" /* execvpe() is not in POSIX */
extern char **environ; /* No POSIX header declares environ */
int main(void)
{
int pipes[2];
pipe(pipes);
pid_t pid = fork();
if (pid == -1)
{
return 1;
}
else if (pid > 0)
{
close(pipes[1]);
char buffer[BUFLEN];
char *b_str = buffer;
size_t b_len = sizeof(buffer);
size_t t_len = 0;
ssize_t n_len;
while (b_len > 0 && (n_len = read(pipes[0], b_str, b_len)) > 0)
{
b_str += n_len;
b_len -= n_len;
t_len += n_len;
}
close(pipes[0]);
int status = 0;
int corpse = waitpid(pid, &status, 0);
printf("%d 0x%.4X: [[%.*s]]\n", corpse, status, (int)t_len, buffer);
}
else
{
dup2(pipes[1], STDOUT_FILENO);
dup2(pipes[1], STDERR_FILENO);
close(pipes[0]);
close(pipes[1]);
char *argv[] = {"sh", "test", NULL};
execvpe(argv[0], argv, environ);
fprintf(stderr, "failed to execute '%s'\n", argv[0]);
exit(1);
}
return 0;
}
test
#!/bin/bash
cat file.txt
echo "Hello!"
echo "Goodbye!"
echo "Errors go to standard error" >&2
file.txt
Line 1 of file1.txt
The very last line of file1.txt
Sample output
14171 0x0000: [[Line 1 of file1.txt
The very last line of file1.txt
Hello!
Goodbye!
Errors go to standard error
]]
Note that the code closes both ends of the pipe before calling execvpe(). It isn't crucial here, but it often can be crucial to do so. Your original code passed a size_t value, n, to printf() for use by the * in the format. You might get away with that on a 64-bit machine where sizeof(int) == sizeof(size_t), but it yields compilation warnings on 64-bit machines where sizeof(int) < sizeof(size_t).
Rule of thumb: If you
dup2()
one end of a pipe to standard input or standard output, close both of the
original file descriptors returned by
pipe()
as soon as possible.
In particular, you should close them before using any of the
exec*()
family of functions.
The rule also applies if you duplicate the descriptors with either
dup()
or
fcntl()
with F_DUPFD or F_DUPFD_CLOEXEC.
Im having big troubles in doing what i said in title.
Basically, i want a program, say broadcast.c, that accepts input from the user and then sends that input to the input of two processes.
So if would run this command:
./broadcast prog1 prog2
It would block awaiting the input from the user and then sending that input to prog1 and prog2.
Now, i want to use pipes, thing is, i dont know if i have to use 1 pipe or 2 pipes.
broadcast.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char* argv[]) {
int fds1[2], fds2[2];
char buffer[120];
pipe(fds1);
pipe(fds2);
if (fork() == 0) {
close(0);
dup(fds1[0]);
close(fds1[0]);
close(fds1[1]);
execl(argv[1], argv[1], NULL);
}
if (fork() == 0) {
close(0);
dup(fds2[0]);
close(fds2[0]);
close(fds2[1]);
execl(argv[2], argv[2], NULL);
}
while(read(0, buffer, 120) != 0) {
printf("lido: %s\n", buffer);
write(fds1[0],buffer,120);
write(fds2[0],buffer,120);
}
close(1);
dup(fds1[1]);
dup(fds2[1]);
exit(0);
}
I know this doesnt work and it may be messed up, so if you guys could help me out that would be great.
For now i just want that when i do:
./broadcast prog1 prog2
The user enters: Hello
The output is:
prog1 says: Hello!
prog2 says: Hello!
Basically prog1 and prog2 are just print using read on fd 0.
It can be easily done in shell:
FIFO_FILE=/tmp/fifo$$
mkfifo $FIFO_FILE
cat $FIFO_FILE | prog1 &
cat | tee $FIFO_FILE | prog2
wait # wait for everything to finish
rm -f $FIFO_FILE
If you insist on the C code... There are so many problems I have found in your code:
duping the other end of pipe (0 instead of 1)
close the other pipe in the child processes
you should handle return value of read, the actual number of bytes read - and pass it to the write function
you must close the child's ends of pipes in the parent
parent should close its pipes afterwards
unnecesasry dup2 calls at the end of the program
From the number of mistakes I see you don't understand it (sorry...). But basicly I must commend you - you created the 2 pipes and the while loop, this core of the program was almost correct. I recommend you to start learning step by step on small examples:
Linux Documentation Project - pipes in C
This valuable resource will teach you how to do pipes, how to redirect etc.
Here is my attempt to fix your code:
int main(int argc, char* argv[]) {
int fds1[2], fds2[2];
char buffer[120];
int size;
pipe(fds1);
pipe(fds2);
if (fork() == 0) {
close(0);
dup(fds1[1]);
close(fds1[0]);
close(fds1[1]);
close(fds2[0]);
close(fds2[1]);
execl(argv[1], argv[1], NULL);
}
if (fork() == 0) {
close(0);
dup(fds2[1]);
close(fds1[0]);
close(fds1[1]);
close(fds2[0]);
close(fds2[1]);
execl(argv[2], argv[2], NULL);
}
close(fds1[1]);
close(fds2[1]);
while((size = read(0, buffer, 120)) != 0) {
printf("lido: %s\n", buffer);
write(fds1[0],buffer,size);
write(fds2[0],buffer,size);
}
close(fds1[0]);
close(fds1[0]);
exit(0);
}
Note that you should handle all syscalls by checking for -1 return value and the ERRNO by perror!
i want a program, say broadcast.c, that accepts input from the user and then sends that input to the input of two processes. So if would run this command: ./broadcast prog1 prog2
You could implement broadcast command using bash:
$ tee >/dev/null >(prog1) >(prog2)
tee reads from stdin and sends it to prog1 and prog2. tee by default duplicates stdin to stdout therefore >/dev/null is used to suppress it.
There is also pee command from moreutils package:
$ pee prog1 prog2
It does exactly what you want. It uses popen() to run child processes. The implementation is very simple, here's the whole source code (pee.c from git://git.kitenet.net/moreutils):
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/wait.h>
/* Licensed under the GPL
* Copyright (c) Miek Gieben, 2006
*/
/* like tee(1), but then connect to other programs using
* pipes _and_ output to standard output
*/
int
close_pipes(FILE **p, size_t i)
{
int ret=EXIT_SUCCESS;
size_t j;
for (j = 0; j < i; j++) {
int r = pclose(p[j]);
if (WIFEXITED(r))
ret |= WEXITSTATUS(r);
else
ret |= 1;
}
return ret;
}
int
main(int argc, char **argv) {
size_t i, r;
FILE **pipes;
char buf[BUFSIZ];
pipes = malloc(((argc - 1) * sizeof *pipes));
if (!pipes)
exit(EXIT_FAILURE);
for (i = 1; i < argc; i++) {
pipes[i - 1] = popen(argv[i], "w");
if (!pipes[i - 1]) {
fprintf(stderr, "Can not open pipe to '%s\'\n", argv[i]);
close_pipes(pipes, argc);
exit(EXIT_FAILURE);
}
}
argc--;
while(!feof(stdin) && (!ferror(stdin))) {
r = fread(buf, sizeof(char), BUFSIZ, stdin);
for(i = 0; i < argc; i++) {
if (fwrite(buf, sizeof(char), r, pipes[i]) != r) {
fprintf(stderr, "Write error to `%s\'\n", argv[i + 1]);
close_pipes(pipes, argc);
exit(EXIT_FAILURE);
}
}
}
exit(close_pipes(pipes, argc));
}
The problem is that streams can only go to one destination. You would need to open two streams and send one to each destination and there is no easy way I'm aware of to do that.
What may be simplest is to make prog1 just pass through any input it receives and send it on it's output, then you can just add it in the middle of the chain and it is effectively invisible to the other processes.
To expand:
in prog 1 - whenever it receives input on stdin - it as well as processing it outputs the same data on stdout. This means that you can add it into a chain broadcast | prog1 | prog2 and the data will get passed through to prog2 as well as being processed by prog1.
I need to fork a process, redirect output (stdout and stderr) in buffer. My code seems to work with most of binary but not all. For example I can run my code with a very long "ls" like ls -R /proc/ and it is working perfectly. When I run mke2fs process, my code does not work anymore.
If I run mke2fs in a fork and wait for it, it is working perfectly. Now if I add redirect stuff, my programs never finish to run.
I wrote a little main to test this specific trouble :
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main ()
{
pid_t pid;
int status = -42;
int pipefd_out[2];
int pipefd_err[2];
char buf_stderr[1024];
char buf_stdout[1024];
int count;
int ret;
pipe(pipefd_out);
pipe(pipefd_err);
memset (buf_stdout, 0, 1024);
memset (buf_stderr, 0, 1024);
pid = fork ();
if (pid == -1)
{
fprintf (stderr, "Error when forking process : /usr/sbin/mke2fs\n");
return 1;
}
if (pid == 0)
{
close(pipefd_out[0]);
close(pipefd_err[0]);
dup2(pipefd_out[1], 1);
dup2(pipefd_err[1], 2);
close(pipefd_out[1]);
close(pipefd_err[1]);
char **args;
args = malloc (sizeof (1024));
args[0] = strdup("/usr/sbin/mke2fs");
args[1] = strdup("/dev/sda4");
args[2] = strdup("-t");
args[3] = strdup("ext4");
args[4] = NULL;
execvp ("/usr/sbin/mke2fs", args);
/*
args = malloc (sizeof (1024));
args[0] = strdup("/bin/ls");
args[1] = strdup("-R");
args[2] = strdup("/proc/irq");
args[3] = NULL;
execvp ("/bin/ls", args);
*/
perror ("execv");
fprintf (stderr, "Error when execvp process /usr/sbin/mke2fs\n");
return 1;
}
close(pipefd_out[1]);
close(pipefd_err[1]);
if (waitpid(pid, &status, 0) == -1)
{
fprintf (stderr, "Error when waiting pid : %d\n", pid);
return 1;
}
do
{
count = read(pipefd_out[0], buf_stdout, sizeof(buf_stdout));
}
while (count != 0);
do
{
count = read(pipefd_err[0], buf_stderr, sizeof(buf_stderr));
}
while (count != 0);
ret = WEXITSTATUS(status);
FILE* file = NULL;
file = fopen("/root/TUTU", "w");
if (file != NULL)
{
fwrite(buf_stdout, 1, sizeof(buf_stdout), file);
fwrite(buf_stderr, 1, sizeof(buf_stdout), file);
fclose(file);
}
return 0;
}
If I run ps, I could see my child process running :
# ps | grep sda4
936 root 2696 S {mke2fs} /dev/sda4 -t ext4
I am not able to understand why I got this strange behavior. Not sure if its related, but output of mke2fs is not classic. Instead of print output and move forward the prompt, the process seems to update the output during the computing. It is a kind of progress bar. Not sure if my explanation is really clear.
Thanks,
Eva.
You can't wait for the program to finish (what you do with waitpid) before reading its stdout/stderr from the pipe. When the program writes to the pipe and its full it will sleep until you read from the pipe to make space in it. So the program waits until there's more space in the pipe before it can continue and exit, while you're waiting for the program to exit before you read from the pipe to make space in it.
The simplest solution in this case would be to just move waitpid until after you're done reading from the pipes. It should be fine since the program you execute will close the pipes when exiting.
I'm trying to run a program with a specific standard input. I succeed by using a file descriptor of a file where there is what I want to put in the stdin, but I fail to write directly on the stdin :
$cat input.test
echo Hello
$
Code C :
int main(int argc, char **argv)
{
int fd = 0;
fd = open("input.test", O_CREAT);
close(STDIN_FILENO);
dup2(fd, STDIN_FILENO);
char *const args[] = { "bash", NULL };
execvp("bash", args);
}
That works :
$./a.out
Hello
$
But if I try to write directly on the STDIN using pipe the program displays nothing and keeps running :
int main(int argc, char **argv)
{
int fds[2];
pipe(fds);
close(STDIN_FILENO);
dup2(fds[1], STDIN_FILENO);
write(fds[1], "echo Hello;", 11); // Résults are identics with fds[0]
char *const args[] = { "bash", NULL };
execvp("bash", args);
}
Thanks for your help
Cordially,
Bastien.
EDIT Problem solved:
Thanks for your answers, here the code which works :
int main(void)
{
int fd[2];
pid_t pid;
if (pipe(fd) < 0)
return EXIT_FAILURE;
if ((pid = fork()) < 0)
return EXIT_FAILURE;
else if (pid != 0) { /* father */
close(fd[1]);
dup2(fd[0], STDIN_FILENO);
execlp("bash", "bash", (char *)0);
} else { /* son */
close(fd[0]);
write(fd[1], "echo hello\n", 11);
}
return EXIT_SUCCESS;
}
You need to dup the read side of the pipe to stdin, not the write side. (And write to the write side, obviously.)
#include <unistd.h>
#include <string.h>
#include <stdio.h>
int main(int argc, char **argv)
{
int fds[2];
char cmd[] = "echo hello\nexit\n";
pipe(fds);
close(STDIN_FILENO);
dup2(fds[0], STDIN_FILENO);
write(fds[1], cmd, strlen(cmd));
char *const args[] = { "bash", NULL };
execvp("bash", args);
return 0;
}
Make sure you check the return values of all those functions though, you'll never manage to debug your code if you don't.
execv and friends replace the current running program with the specified one; they do not return - execution continues at the start of new program instead.
So what you normally do is fork and, in one of the forks, call execv. You then read and write through the pipe from your program continuing in the other fork. There are usually popen functions to do this in most languages; sadly in POSIX the popen() is strictly read or write and not bidirectional.
Luckily, I've made, tested and published a popen3 function. This gives you back three file descriptors - one for stdin to the process, and two for stdout and stderr. You can then use write() on the stdin.
When you call pipe, fd[ 0 ] is open for reading, and fd[ 1 ] is open for writing. You should be dup'ing stdin on the read side ( fd[ 0 ]) and writing to the write side( fd[ 1 ]). Check the return value of write: it is probably -1.
But there is a larger issue. You never close either side of the pipe. bash may block on a read and never do anything until the write side of the pipe is closed. You should close both sides of the pipe after you dup and write. (Or set FD_CLOEXEC).
Also note that doing it the way you do, you're dependent on pipe buffer size. If you write too much, write will be blocked as there's no reader. Do do it reliably, you should fork(), do exec in the child and write to the pipe in the parent. This way the pipe will have a reader and you will be able to write as much data as you want into it.
Trying to implement a shell, mainly piping. I've written this test case which I expect to simply pipe ls to wc...it definitely doesn't work as expected. It prints ls to the terminal then prints memory exhausted.
I'm very lost in how to fix this and get it to work. find_path works in all of my tests.
Edit - I have to use execv for the project, its a class thing, but I've tried it with execvp just in case and it does the exact same thing. Also this is just an example, a test to see why it does not work, I call fork twice once for both commands and waitpid because I have nothing else to do.
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
int find_path(char* execname, char** dst)
{
char *path = getenv("PATH");
path = strdup(path);
char *pos;
path = strtok_r(path, ":", &pos);
char *originalpath = path;
do
{
char* test = (char*)calloc(strlen(path) + strlen(execname) + 2, sizeof(char));
test = strcpy(test, path);
int testlen = strlen(test);
(*(test+testlen)) = '/';
strcpy(test + testlen + 1,execname);
struct stat buf;
int result = stat(test, &buf);
if (result == 0)
{
*dst = test;
free (originalpath);
return 1;
}
else
{
free(test);
}
} while ((path = strtok_r(NULL, ":", &pos)) != NULL);
free(originalpath);
return 0;
}
int main()
{
char *cmd1 = "ls";
char *cmd2 = "wc";
int filedes[2];
pipe(filedes);
char** argv = (char**)calloc(1, sizeof(char*));
argv[0] = (char*)malloc(sizeof(char*));
argv[0] = NULL;
pid_t pid = fork();
if (pid == 0)
{
char *path;
find_path(cmd1, &path);
dup2(filedes[1],stdout);
execv(path,argv);
}
pid = fork();
if (pid == 0)
{
dup2(filedes[0], stdin);
char *path;
find_path(cmd2, &path);
execv(path, argv);
}
else
waitpid(pid);
}
Often when it is hard to debug a program, it is best to simplify it a little to eliminate sources of error. Here is your program, simplified to remove find_path as a source of errors:
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
int main(void)
{
int filedes[2];
pipe(filedes);
/* Run LS. */
pid_t pid = fork();
if (pid == 0) {
/* Set stdout to the input side of the pipe, and run 'ls'. */
dup2(filedes[1], 1);
char *argv[] = {"ls", NULL};
execv("/bin/ls", argv);
} else {
/* Close the input side of the pipe, to prevent it staying open. */
close(filedes[1]);
}
/* Run WC. */
pid = fork();
if (pid == 0) {
dup2(filedes[0], 0);
char *argv[] = {"wc", NULL};
execv("/usr/bin/wc", argv);
}
/* Wait for WC to finish. */
waitpid(pid);
}
This should behave as you expect.
During simplification, a few errors came out:
argv[] wasn't being setup correctly, in particular, argv[0] was being set to NULL;
The program was not closing the input side of the pipe that was being given to ls. When ls finished, the pipe wasn't being closed (because the wc process still had it open), preventing wc from ever finishing.
The program was confusing the values stdout and stdin (which are of type FILE*) with the file descriptor numbers 0 and 1 (used by dup, pipe, etc.)
There is a lot you can do to improve this code (e.g. breaking this into smaller functions would be a start), but I suspect your out of memory issue is from the code in find_path(), which you could avoid entirely by using execvp which will locate the executable using the standard PATH mechanism for you. It is probably a good idea to install a signal handler using sigaction to handle SIGCHLD and invoke waitpid from the signal handler, instead of just invoking waitpid() ad-hoc like you are doing. You appear to be forking more times than you want, and you aren't checking for errors. Hope these suggestions help.