I want to do simple thing: my_process | proc2 | proc3, but programatically - without using shell, that can do this pretty easy. Is this possible? I cannot find anything :(
EDIT:
Well, without code, nobody will know, what problem I'm trying to resolve. Actually, no output is going out (I'm using printfs)
int pip1[2];
pipe(pip1);
dup2(pip1[1], STDOUT_FILENO);
int fres = fork();
if (fres == 0) {
close(pip1[1]);
dup2(pip1[0], STDIN_FILENO);
execlp("wc", "wc", (char*)0);
}
else {
close(pip1[0]);
}
Please learn about file descriptors and the pipe system call. Also, check read and write.
Your 'one child' code has some major problems, most noticeably that you configure the wc command to write to the pipe, not to your original standard output. It also doesn't close enough file descriptors (a common problem with pipes), and isn't really careful enough if the fork() fails.
You have:
int pip1[2];
pipe(pip1);
dup2(pip1[1], STDOUT_FILENO); // The process will write to the pipe
int fres = fork(); // Both the parent and the child will…
// Should handle fork failure
if (fres == 0) {
close(pip1[1]);
dup2(pip1[0], STDIN_FILENO); // Should close pip1[0] too
execlp("wc", "wc", (char*)0);
}
else { // Should duplicate pipe to stdout here
close(pip1[0]); // Should close pip1[1] too
}
You need:
fflush(stdout); // Print any pending output before forking
int pip1[2];
pipe(pip1);
int fres = fork();
if (fres < 0)
{
/* Failed to create child */
/* Report problem */
/* Probably close both ends of the pipe */
close(pip1[0]);
close(pip1[1]);
}
else if (fres == 0)
{
dup2(pip1[0], STDIN_FILENO);
close(pip1[0]);
close(pip1[1]);
execlp("wc", "wc", (char*)0);
}
else
{
dup2(pip1[1], STDOUT_FILENO);
close(pip1[0]);
close(pip1[1]);
}
Note that the amended code follows the:
Rule of thumb: If you use dup2() to duplicate one end of a pipe to standard input or standard output, you should close both ends of the original pipe.
This also applies if you use dup() or fcntl() with F_DUPFD.
The corollary is that if you don't duplicate one end of the pipe to a standard I/O channel, you typically don't close both ends of the pipe (though you usually still close one end) until you're finished communicating.
You might need to think about saving your original standard output before running the pipeline if you ever want to reinstate things.
As Alex answered, you'll need syscalls like pipe(2), dup2(2), perhaps poll(2) and some other syscalls(2) etc.
Read Advanced Linux Programming, it explains that quite well...
Also, play with strace(1) and study the source code of some simple free software shell.
See also popen(3) -which is not enough in your case-
Recall that stdio(3) streams are buffered. You probably need to fflush(3) at appropriate places (e.g. before fork(2))
Related
I'm trying to do a little C program that realize a pipeline of two bash commands : echo $arithmeticOperation | bc
$arithmeticOperation is a string taken as input.
The program works fine executing first command, but when i run the second one, i get the right output but the child process executing bc remains stuck preventing the child from ending.
So in this line father process is blocked :
waitpid(pid2,NULL,0);
Where do you think the problem may be ?
Sorry if i asked the question incorrectly, it's my first one. Thanks.
#define SYSCALL(r,c,e) if((r=c)==-1) { perror(e);exit(EXIT_FAILURE);}
int main(){
char buf[128];
int pfd[2],err;
pid_t pid1,pid2;
SYSCALL(err,pipe(pfd),"pipe");
switch (pid1=fork()) {
case -1: { perror("fork"); exit(EXIT_FAILURE);}
case 0 : {
scanf("%s",buf);
SYSCALL(err,dup2(pfd[1],1),"dup");
close(pfd[1]);
close(pfd[0]);
execl("/bin/echo","echo",buf,(char *)NULL);
return 1;
}
}
switch (pid2=fork() ){
case -1 : { perror("fork"); exit(EXIT_FAILURE);}
case 0 : {
SYSCALL(err,dup2(pfd[0],0),"dup");
close(pfd[1]);
close(pfd[0]);
// execl("/usr/bin/bc","bc",(char *)NULL);
execlp("bc","bc",(char *)NULL);
return 1;
}
}
printf("waiting . . . \n");
waitpid(pid1,NULL,0);
printf("wait\n");
waitpid(pid2,NULL,0);
close(pfd[1]);
close(pfd[0]);
return 0;
}
So if i digit "1+1" as a input string i get the right output but then the process executing bc never exit
As I noted in a comment, your parent process must close the file descriptors for the pipe before waiting for bc (and you've agreed that this fixes the problem).
This arises because bc has the pipe open for reading, and the parent has the pipe open for writing, and the kernel thinks that the parent could therefore send data to bc. It won't, but it could.
You have to be very careful when managing pipes. You carefully avoided the usual problem of not closing enough file descriptors in the children.
Rule of thumb: If you
dup2()
one end of a pipe to standard input or standard output, close both of the
original file descriptors returned by
pipe()
as soon as possible.
In particular, you should close them before using any of the
exec*()
family of functions.
The rule also applies if you duplicate the descriptors with either
dup()
or
fcntl()
with F_DUPFD
I need to extend that to cover parent processes too.
If the parent process is not going to communicate with any of its children via the pipe, it must ensure that it closes both ends of the pipe so that its children can receive EOF indications on read (or get SIGPIPE signals or write errors on write), rather than blocking indefinitely. The parent should normally close at least one end of the pipe — it would be extremely unusual for a program to read and write on both ends of a single pipe.
My task is to write a C program that executes the command "ls -l /bin/?? | grep rwxr-xr-x | sort". There are 3 child processes where each of them executes one of the commands separately and sends the result through a pipe to the next child process. I'm using a Swedish modified verision of debian so the error message is in Swedish, but i'll translate the error i get, it's something along the lines of: sort: failed to status -: unknown fileidentifier.
Maybe it's my pipes that do not work as intended, I'm not too sure about the close() commands. I'm pretty sure the error comes from the pipes. Would be grateful if someone could run the program and get the english error message.
#include <stdio.h>
#include <sys/types.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <assert.h>
#include <errno.h>
#include <string.h>
int main()
{
int ret;
int fds1[2], fds2[2], fds3[2];
char buf[20];
pid_t pid;
///initiating pipes
ret=pipe(fds1);
if(ret == -1){
perror("could not pipe");
exit(1);
}
ret=pipe(fds2);
if( ret == -1){
perror("could not pipe");
exit(1);
}
ret=pipe(fds3);
if (ret == -1){
perror("could not pipe");
exit(1);
}
pid=fork();
if(pid==-1){
fprintf(stderr,"fork failed");
exit(0);
}
if(pid==0){
///CHILD 1
close(1);
dup(fds1[1]);
close(fds1[0]);
close(fds1[1]);
close(0);
execlp("/bin/sh","bin/sh", "ls-l /bin/??", (char *)NULL);
}
else{
wait(0);
}
pid=fork();
if(pid==-1){
fprintf(stderr,"fork failed");
exit(0);
}
if(pid==0){
close(0);
dup(fds1[0]);
close(fds1[0]);
close(fds1[1]);
close(1);
dup(fds2[1]);
close(fds2[0]);
close(fds2[1]);
execlp("/usr/share/grep/", "grep", "rwxr-xr-x", NULL);
}
else{
wait(0);
}
close(fds1[0]);
close(fds1[1]);
pid=fork();
if(pid==-1){
fprintf(stderr,"fork failed");
exit(0);
}
if(pid==0){
close(0);
dup(fds2[0]);
close(fds2[0]);
close(fds2[1]);
execlp("sort", "sort", NULL);
}
else{
wait(0);
}
close(fds2[0]);
close(fds2[1]);
}
Your code has several problems, but before I discuss them, let me introduce you to a flavor of one of my favorite preprocessor macros:
#define DO_OR_DIE(x, s) do { \
if ((x) < 0) { \
perror(s); \
exit(1); \
} \
} while (0)
Using this macro where it is applicable can clarify your code by replacing all the boilerplate error checking. For example, this:
ret=pipe(fds1);
if(ret == -1){
perror("could not pipe");
exit(1);
}
becomes just
DO_OR_DIE(pipe(fds1), "pipe");
That makes it a lot easier to see and focus on the key parts of the code, and it's easier to type, too. As a result, it also reduces the temptation to skip error checks, such as those for your calls to dup().
Now, as to your code. For me, it exhibits not just the one misbehavior you now describe in your question, but three:
It emits an error message "bin/sh: ls-l /bin/??: No such file or directory".
It emits the error message you describe, "sort: stat failed: -: Bad file descriptor".
It does not terminate.
The first error message pertains to multiple problems in the arguments to your first execlp() call. If you want to launch a shell and specify a command for it to run, as opposed to a file from which to read commands, then you must pass the -c option to it. Additionally, you've omitted mandatory whitespace between the ls and its arguments. It looks like you want this:
execlp("/bin/sh","sh", "-c", "ls -l /bin/??", (char *)NULL);
Setting aside the second problem for the moment, let's turn to the failure to terminate. You have several problems in this area, falling into these categories:
Holding pipe ends open where you should ensure them closed
Calling wait() at the wrong points
When you set up a pipe between two processes, you generally want to make sure that there are no open file descriptors on either end of the pipe other than one on the write end held by one process, and one on the read end held by the other process. Each end should be open exactly once, in exactly one process. Since the processes being connected invariably inherit these file descriptors from their parent, it is essential that the parent close its copies (except that the parent will want to keep one open in the event that it itself is one of the communicating processes).
The process on the read end of a pipe will not see EOF on that pipe until all open file descriptors on the write end are closed. Child processes running programs such as grep and sort that read their input to its end will hang indefinitely if the write end of the pipe is not completely closed.
That can be a particularly perverse problem when the child reading the pipe also has a copy of the write end of that pipe, unused, or if one of its siblings does.
Additionally, the whole point of a pipeline is that the processes involved run concurrently. If you wait() after starting one before starting the next, then at minimum you prevent such concurrency. Worse, however, that can also cause your program to hang, because a pipe has finite buffer capacity. If the child is writing output to a pipe, but no one is reading it, then the pipe's buffer can fill to capacity, at which point the child blocks. If the parent is waiting for the child to finish before launching the process that will drain the pipe, then you have a deadlock. Therefore, you should start all the processes in the pipeline first, then wait for them all.
Having fixed such problems in your code, I find that the program emits a different error for me:
execlp: No such file or directory
(The specifics of this message derive from the nature of my fixes.) This should be especially concerning, because if execlp() fails then it returns in the process in which it was called. In your cases, control will then fall right out of your if statement, into the code intended only for the parent to execute. For this reason, it is essential to handle errors from execlp(). At minimum, add a call to exit() or _Exit() immediately after.
But what's failing? Well, it's the grep this time. Note that you specify the command to execute as "/usr/share/grep/" -- that trailing / is erroneous, and the path itself is suspect. On my system, the correct path is /usr/bin/grep, but since we're using execlp, which resolves the executable in the path, we might as well omit the path altogether:
execlp("grep", "grep", "rwxr-xr-x", (char *) NULL);
Et voilà! After making that correction as well, your program runs for me.
Additional advice: do not use dup() when you care what file descriptor number you want the duplicate to have, such as when you're trying to dup onto one of the standard streams. Use dup2() for that, which has the additional advantage that you don't need to close the specified file descriptor first.
I am creating a C program and with it I am setting up a pipe between separately forked process for interprocess communication.
The first process has written the data I need into the pipe.
However, with the second process reading from the pipe, I am trying to exec the process to become the UNIX sort command. I want to somehow call sort on the data in the pipe.
How can I call sort on a pipe? On the commandline, I can sort by supplying the filename to sort as a commandline argument e.g. "sort -r MyFileToSort". I know that pipes are essentially considered files, but they are only described by their file descriptor, and as far as I know, sort won't know what to do with a fd.
Thanks for any help/feedback
int p[2];
if (pipe(p) != 0) ...report error and do not continue...
pid_t pid = fork();
if (pid < 0) ...report error, close pipe descriptors, and do not continue...
if (pid == 0)
{
/* Child - becomes sort */
dup2(p[0], 0);
close(p[0]);
close(p[1]);
int fd = open("output-file", O_CREAT | O_EXCL | O_WRONLY, 0644);
if (fd < 0) ...report error and exit...
dup2(fd, 1);
close(fd);
execlp("sort", "sort", (char *)0);
...report error and exit...
}
else
{
/* Parent - writes data to sort */
close(fd[0]);
...write data to fd[1]...
close(fd[1]);
int status;
int corpse;
while ((corpse = wait(&status)) > 0 && corpse != pid)
...consider reporting which child died...
...consider reporting sort status...
...continue with the rest of the program...
}
You can decide whether to report errors related to dup2() failing, or close() failing. There isn't much you can do in either case except report the problem and exit. Unless someone has subjected your program to cruel and unusual punishment by not supplying it with standard input, standard output and standard error (or something elsewhere in the program has closed any of the standard channels), then the pipe and file descriptors can't be the standard I/O descriptors, so the closes are safe. If you're not sure how sick your users are, you might protect the closes:
if (p[0] > FILENO_STDERR)
close(p[0]);
That is normally unnecessarily paranoid (but it can be fun trying programs with missing standard I/O).
You don't need to pass sort any arguments to specify input source or output sink at all in this case. Instead, before execing it, you should make attach your pipeline's file descriptors to its stdin (FD 0, if receiving data from a pipe) or stdout (FD 1, if writing data to a pipe), as appropriate.
See the dup2() call, which lets you set the destination to which you're copying a FD, for this purpose. As #JonathanLeffler points out, you'll want to be sure to close the original FDs (after duplicating them to the numbers you want) before your exec call.
Since you've clarified, in comments, that your goal is to write to a file, you would attach FD 1 to that destination file before calling exec, with FD 0 attached to the output side of the pipeline containing input.
I'm writing my own shell. Below is part of it. I want to implement pipe with two child processes.
But when I execute below code, some commands work, some don't. 'who | sort', 'ls | sort' work but 'll | sort', 'cat file | sort' make my shell stop. I have to 'cntrl + c' to escape from my shell. I don't understand why some commands work and some don't. Please somebody point out what I'm missing?
int pipefd[2];
int pipePos = checkPipePos(argc, argv); //find '|' command position in argv
if (pipe(pipefd) == -1) {
printf("Creating pipe failed\n");
return;
}
if (fork() == 0) {
close(pipefd[0]);
dup2(pipefd[1], 1);
execlp(argv[0], argv[0], NULL);
}
if (fork() == 0) {
close(pipefd[1]);
dup2(pipefd[0], 0);
execlp(argv[pipePos + 1], argv[pipePos + 1], NULL);
}
close(pipefd[0]);
close(pipefd[1]);
while (wait(NULL) >= 0);
I see one possible problem here, which may or may not explain the non-deterministic results.
In your child process, after you're dup-ing one of the pipe's file descriptor, you're not closing the original ones.
dup2(oldfd, newfd) duplicates oldfd onto newfd, but oldfd is still open. Both oldfd and newfd file descriptors now point to the same file. See the dup2 man page.
Typically, aftering dup2()-ing a file descriptor, you'll want to close() the oldfd.
So, what's going to happen here is that one of the child processes will have the same file descriptor on stdout and on some other, random file descriptor, and the other child process will have the same situation with stdin.
This may or may not cause a problem, depending on what the child process does. Can't really say for sure, but you'll definitely want to fix this, and see if it helps.
Your pipe logic is correct.
ll | sort doesn't work because ll is not a executable file.
cat file | sort doesn't work because you are not passing the file argument.
Sam Varshavchik's improvement is valid. It's a good-to-fix, but it won't cause any serious problem. All it causes is wasting one more file descriptor slot.
This seems to be a fairly common thing to do, and I've managed to teach myself everything that I need to make it work, except that I now have a single problem, which is defying my troubleshooting.
int nonBlockingPOpen(char *const argv[]){
int inpipe;
pid_t pid;
/* open both ends of pipe nonblockingly */
pid = fork();
switch(pid){
case 0: /*child*/
sleep(1); /*child should open after parent has open for reading*/
/*redirect stdout to opened pipe*/
int outpipe = open("./fifo", O_WRONLY);
/*SHOULD BLOCK UNTIL MAIN PROCESS OPENS FOR WRITING*/
dup2(outpipe, 1);
fcntl(1, F_SETFL, fcntl(1, F_GETFL) | O_NONBLOCK);
printf("HELLO WORLD I AM A CHILD PROCESS\n");
/*This seems to be written to the pipe immediately, blocking or not.*/
execvp(*argv, argv);
/*All output from this program, which outputs "one" sleeps for 1 second
*outputs "two" sleeps for a second, etc, is captured only after the
*exec'd program exits!
*/
break;
default: /*parent*/
inpipe = open("./fifo", O_RDONLY | O_NONBLOCK);
sleep(2);
/*no need to do anything special here*/
break;
}
return inpipe;
}
Why won't the child process write its stdout to the pipe each time a line is generated? Is there something I'm missing in the way execvp or dup2 work? I'm aware that my approach to all this is a bit strange, but I can't find another way to capture output of closed-source binaries programatically.
I would guess you only get the exec'd program's output after it exits because it does not flush after each message. If so, there is nothing you can do from the outside.
I am not quite sure how this is supposed to relate to the choice between blocking and nonblocking I/O in your question. A non-blocking write may fail completely or partially: instead of blocking the program until room is available in the pipe, the call returns immediately and says that it was not able to write everything it should have. Non-blocking I/O neither makes the buffer larger nor forces output to be flushed, and it may be badly supported by some programs.
You cannot force the binary-only program that you are exec'ing to flush. If you thought that non-blocking I/O was a solution to that problem, sorry, but I'm afraid it is quite orthogonal.
EDIT: Well, if the exec'd program only uses the buffering provided by libc (does not implement its own) and is dynamically linked, you could force it to flush by linking it against a modified libc that flushes every write. This would be a desperate measure. to try only if everything else failed.
When a process is started (via execvp() in your example), the behaviour of standard output depends on whether the output device is a terminal or not. If it is not (and a FIFO is not a terminal), then the output will be fully buffered, rather than line buffered. There is nothing you can do about that; the (Standard) C library does that.
If you really want to make it work line buffered, then you will have to provide the program with a pseudo-terminal as its standard output. That gets into interesting realms - pseudo-terminals or ptys are not all that easy to handle. For the POSIX functions, see:
grantpt() - grant access to the slave pseudo-terminal device
posix_openpt() - open a pseudo-terminal device
ptsname() - get name of the slave pseudo-terminal device
unlockpt() - unlock a pseudo-terminal master/slave pair
Why won't the child process write its stdout to the pipe each time a line is generated?
How do you know that? You do not even try to read the output from the fifo.
N.B. by the file name I presume that you are using the fifo. Or is it a plain file?
And the minor bug in the child: after dup2(), you need to close(outpipe).
fcntl(1, F_SETFL, fcntl(1, F_GETFL) | O_NONBLOCK);
Depending on what program you exec(), you might either lose some output or cause the program to fail since write to stdout now might fail with EWOULDBLOCK.
IIRC fifos has the same buffer size as pipes. Per POSIX minimum is 512 bytes, commonly 4K or 8K.
You probably want to explain why you need that at all. Non-blocking IO has different semantics compared to blocking IO and unless your child process expects that you will run into various problems.
printf("HELLO WORLD I AM A CHILD PROCESS\n");
stdout is buffered, I would have after that fflush(stdout). (Can't find documentation whether exec() on its own would flush stdout or not.)
Is there something I'm missing in the way execvp or dup2 work? I'm aware that my approach to all this is a bit strange, but I can't find another way to capture output of closed-source binaries programatically.
I wouldn't toy with non-blocking IO - and leave it as it is in blocking mode.
And I would use pipe() instead of the fifo. Linux's man pipe has a convenient example with the fork().
Otherwise, that is a pretty normal practice.
The sleep()s do not guarantee that the parent will open the pipe first - as Dummy00001 says, you should be using a pipe() pipe, not a named pipe. You should also check for execvp() and fork() failing, and you shouldn't be setting the child side to non-blocking - that's a decision for the child process to make.
int nonBlockingPOpen(char *const argv[])
{
int childpipe[2];
pid_t pid;
pipe(childpipe);
pid = fork();
if (pid == 0)
{
/*child*/
/*redirect stdout to opened pipe*/
dup2(childpipe[1], 1);
/* close leftover pipe file descriptors */
close(childpipe[0]);
close(childpipe[1]);
execvp(*argv, argv);
/* Only reached if execvp fails */
perror("execvp");
exit(1);
}
/*parent*/
/* Close leftover pipe file descriptor */
close(childpipe[1]);
/* Check for fork() failing */
if (pid < 0)
{
close(childpipe[0]);
return -1;
}
/* Set file descriptor non-blocking */
fcntl(childpipe[0], F_SETFL, fcntl(childpipe[0], F_GETFL) | O_NONBLOCK);
return childpipe[0];
}