I'm writing my own shell. Below is part of it. I want to implement pipe with two child processes.
But when I execute below code, some commands work, some don't. 'who | sort', 'ls | sort' work but 'll | sort', 'cat file | sort' make my shell stop. I have to 'cntrl + c' to escape from my shell. I don't understand why some commands work and some don't. Please somebody point out what I'm missing?
int pipefd[2];
int pipePos = checkPipePos(argc, argv); //find '|' command position in argv
if (pipe(pipefd) == -1) {
printf("Creating pipe failed\n");
return;
}
if (fork() == 0) {
close(pipefd[0]);
dup2(pipefd[1], 1);
execlp(argv[0], argv[0], NULL);
}
if (fork() == 0) {
close(pipefd[1]);
dup2(pipefd[0], 0);
execlp(argv[pipePos + 1], argv[pipePos + 1], NULL);
}
close(pipefd[0]);
close(pipefd[1]);
while (wait(NULL) >= 0);
I see one possible problem here, which may or may not explain the non-deterministic results.
In your child process, after you're dup-ing one of the pipe's file descriptor, you're not closing the original ones.
dup2(oldfd, newfd) duplicates oldfd onto newfd, but oldfd is still open. Both oldfd and newfd file descriptors now point to the same file. See the dup2 man page.
Typically, aftering dup2()-ing a file descriptor, you'll want to close() the oldfd.
So, what's going to happen here is that one of the child processes will have the same file descriptor on stdout and on some other, random file descriptor, and the other child process will have the same situation with stdin.
This may or may not cause a problem, depending on what the child process does. Can't really say for sure, but you'll definitely want to fix this, and see if it helps.
Your pipe logic is correct.
ll | sort doesn't work because ll is not a executable file.
cat file | sort doesn't work because you are not passing the file argument.
Sam Varshavchik's improvement is valid. It's a good-to-fix, but it won't cause any serious problem. All it causes is wasting one more file descriptor slot.
Related
I'm writing a C program where I fork(), exec(), and wait(). I'd like to take the output of the program I exec'ed to write it to file or buffer.
For example, if I exec ls I want to write file1 file2 etc to buffer/file. I don't think there is a way to read stdout, so does that mean I have to use a pipe? Is there a general procedure here that I haven't been able to find?
For sending the output to another file (I'm leaving out error checking to focus on the important details):
if (fork() == 0)
{
// child
int fd = open(file, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
dup2(fd, 1); // make stdout go to file
dup2(fd, 2); // make stderr go to file - you may choose to not do this
// or perhaps send stderr to another file
close(fd); // fd no longer needed - the dup'ed handles are sufficient
exec(...);
}
For sending the output to a pipe so you can then read the output into a buffer:
int pipefd[2];
pipe(pipefd);
if (fork() == 0)
{
close(pipefd[0]); // close reading end in the child
dup2(pipefd[1], 1); // send stdout to the pipe
dup2(pipefd[1], 2); // send stderr to the pipe
close(pipefd[1]); // this descriptor is no longer needed
exec(...);
}
else
{
// parent
char buffer[1024];
close(pipefd[1]); // close the write end of the pipe in the parent
while (read(pipefd[0], buffer, sizeof(buffer)) != 0)
{
}
}
You need to decide exactly what you want to do - and preferably explain it a bit more clearly.
Option 1: File
If you know which file you want the output of the executed command to go to, then:
Ensure that the parent and child agree on the name (parent decides name before forking).
Parent forks - you have two processes.
Child reorganizes things so that file descriptor 1 (standard output) goes to the file.
Usually, you can leave standard error alone; you might redirect standard input from /dev/null.
Child then execs relevant command; said command runs and any standard output goes to the file (this is the basic shell I/O redirection).
Executed process then terminates.
Meanwhile, the parent process can adopt one of two main strategies:
Open the file for reading, and keep reading until it reaches an EOF. It then needs to double check whether the child died (so there won't be any more data to read), or hang around waiting for more input from the child.
Wait for the child to die and then open the file for reading.
The advantage of the first is that the parent can do some of its work while the child is also running; the advantage of the second is that you don't have to diddle with the I/O system (repeatedly reading past EOF).
Option 2: Pipe
If you want the parent to read the output from the child, arrange for the child to pipe its output back to the parent.
Use popen() to do this the easy way. It will run the process and send the output to your parent process. Note that the parent must be active while the child is generating the output since pipes have a small buffer size (often 4-5 KB) and if the child generates more data than that while the parent is not reading, the child will block until the parent reads. If the parent is waiting for the child to die, you have a deadlock.
Use pipe() etc to do this the hard way. Parent calls pipe(), then forks. The child sorts out the plumbing so that the write end of the pipe is its standard output, and ensures that all other file descriptors relating to the pipe are closed. This might well use the dup2() system call. It then executes the required process, which sends its standard output down the pipe.
Meanwhile, the parent also closes the unwanted ends of the pipe, and then starts reading. When it gets EOF on the pipe, it knows the child has finished and closed the pipe; it can close its end of the pipe too.
Since you look like you're going to be using this in a linux/cygwin environment, you want to use popen. It's like opening a file, only you'll get the executing programs stdout, so you can use your normal fscanf, fread etc.
After forking, use dup2(2) to duplicate the file's FD into stdout's FD, then exec.
You could also use the linux sh command and pass it a command that includes the redirection:
string cmd = "/bin/ls > " + filepath;
execl("/bin/sh", "sh", "-c", cmd.c_str(), 0);
For those such as myself who like a complete example with includes, here's this fantastic answer with a runnable example (still without error handling, left as an exercise):
#include <fcntl.h>
#include <sys/wait.h>
#include <unistd.h>
int main() {
if (fork() == 0) { // child
int fd = open("test.txt", O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
dup2(fd, 1); // make stdout go to file
dup2(fd, 2); // make stderr go to file - you may choose to not do this
// or perhaps send stderr to another file
close(fd); // fd no longer needed - the dup'ed handles are sufficient
execlp("ls", "ls", NULL);
}
else {
while (wait(NULL) > 0) {} // wait for each child process
}
return 0;
}
I want to communicate with a child process like the following:
int main(int argc, char *argv[])
{
int bak, temp;
int fd[2];
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[0]);
dup2(STDOUT_FILENO, fd[1]);
fflush(stdout);
bak = dup(1);
temp = open("/dev/null", O_WRONLY);
dup2(temp, 1);
close(temp );
Mat frame;
std::vector<uchar> buf;
namedWindow( "Camera", WINDOW_AUTOSIZE );
VideoCapture cam(0 + CAP_V4L);
sleep(1);
if (!cam.isOpened())
{
cout << "\nCould not open reference " << 0 << endl;
return -1;
}
for (int i=0; i<30; i++)
{
cam>>frame;
}
//cout<<"\nCamera initialized\n";
/*Set the normal STDOUT back*/
fflush(stdout);
dup2(bak, 1);
close(bak);
imencode(".png",frame, buf);
cout<<buf.size()<<endl;
ssize_t written= 0;
size_t s = 128;
while (written<buf.size())
{
written += write(fd[1], buf.size()+written, s);
}
cout<<'\0';
return 0;
}
The process corresponding to the compilation of the source code above is called from the parent with popen.
Note that I am writing to the std out that has been duplicated with a pipe.
The parent will read the data and resend them to UDP socket.
If I do something like this:
#define BUFLEN 128
FILE *fp;
char buf[BUFLEN];
if ((fp = popen("path/to/exec", "r")) != NULL)
{
while((fgets(buf, BUFLEN, fp)!=NULL))
{
sendto(sockfd, buf, strlen(buf),0, addr, alen);
}
}
the program is working i.e. the receiver of sendto will receive the data.
I tried to use a pipe as done in the child process:
int fd[2];
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[1]);
dup2(STDIN_FILENO, fd[0]);
if ((fp = popen("path/to/exec", "r")) != NULL)
{
while((read(fd[0], buf, BUFLEN) > 0)
{
sendto(sockfd, buf, strlen(buf),0, addr, alen);
}
}
but with this are not sent.
So how to use pipe in this case to achieve the same behaviour of the first case? Should I do dup2(STDIN_FILENO, fd[0]); or dup2(STDOUT_FILENO, fd[0]);?
I am using the sandard(s) since the file descriptors are inherited by the child process so should not require any other effort. That is why I thought I can use pipe but is that so?
In the parent:
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[0]);
you get a pipe, and then immediately close one end of it. This pipe is now useless, because no-one will ever be able to recover the closed end, and so no data can flow through it. You have converted a pipe into a hollow cylinder sealed at one end.
Then in the child:
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[1]);
you create another unrelated pipe, and seal this at the other end. The two pipes are not connected, and now you have two separate hollow cyclinders, each sealed at one end. Nothing can flow through either of them.
If putting something in the first cylinder made it appear in the other, that'd be a pretty good magic trick. Without sleight of hand or cleverly arranged mirrors, the solution is to create one pipe, keep both ends open and push data through it.
The usual way to manually set up a pipe from which a parent process can read a child process's standard output has these general steps:
parent creates a pipe by calling pipe()
parent fork()s
parent closes (clarification: its copy of) the write end of the pipe
child dupes the write end of the pipe onto its standard output via dup2()
child closes the original file descriptor for the write end of the pipe
(optional) child closes (clarification: its copy of) the read end of the pipe
child execs the desired command, or else performs the wanted work directly
The parent can then read the child's output from the read end of the pipe.
The popen() function does all of that for you, plus wraps the parent's pipe end in a FILE. Of course, it can and will set up a pipe going in the opposite direction instead if that's what the caller requests.
You need to understand and appreciate that in the procedural scheme presented above, it is important which actions are performed by which process, and in what order relative to other actions in the same process. In particular, the parent must not close the write end of the pipe before the child is launched, because that renders the pipe useless. The child inherits the one-end-closed pipe, through which no data can be conveyed.
With respect to your latter example, note also that redirecting the standard input to the read end of the pipe is not part of the process for either parent or child. The fact that your pipe is half-closed, so that nothing can ever be read from it anyway, is just icing on the cake. Moreover, the parent clobbers its own standard input this way. That's not necessarily wrong, but the parent does not even rely on it.
Overall, however, there is a bigger picture that you seem not to appreciate. Even if you performed the redirection you seem to want in the parent, so that it could be inherited by the child, popen() performs its own redirection to a pipe of its own creation. The FILE * it returns is the means by which you can read the child's output. No previous output redirection you may have performed is relevant (clarification: of the child's standard output).
In principle, an approach similar to yours could be used to create a second redirection going the other way, but at that point the convenience factor of popen() is totally lost. It would be better go take the direct pipe / fork / dup2 / exec route all the way through if you want to redirect the child's input and output.
Applying all that to your first example, you have to appreciate that although a process can redirect its own standard streams, it cannot establish a pipe to its parent process that way. The parent needs to provide the pipe, else it has no knowledge of it. And when a process dupes one file descriptor onto another, that replaces the original with the new, closing the original if it is open. It does not redefine the original. And of course, in this case, too, a pipe is useless once either end is no longer open anywhere.
I am creating a C program and with it I am setting up a pipe between separately forked process for interprocess communication.
The first process has written the data I need into the pipe.
However, with the second process reading from the pipe, I am trying to exec the process to become the UNIX sort command. I want to somehow call sort on the data in the pipe.
How can I call sort on a pipe? On the commandline, I can sort by supplying the filename to sort as a commandline argument e.g. "sort -r MyFileToSort". I know that pipes are essentially considered files, but they are only described by their file descriptor, and as far as I know, sort won't know what to do with a fd.
Thanks for any help/feedback
int p[2];
if (pipe(p) != 0) ...report error and do not continue...
pid_t pid = fork();
if (pid < 0) ...report error, close pipe descriptors, and do not continue...
if (pid == 0)
{
/* Child - becomes sort */
dup2(p[0], 0);
close(p[0]);
close(p[1]);
int fd = open("output-file", O_CREAT | O_EXCL | O_WRONLY, 0644);
if (fd < 0) ...report error and exit...
dup2(fd, 1);
close(fd);
execlp("sort", "sort", (char *)0);
...report error and exit...
}
else
{
/* Parent - writes data to sort */
close(fd[0]);
...write data to fd[1]...
close(fd[1]);
int status;
int corpse;
while ((corpse = wait(&status)) > 0 && corpse != pid)
...consider reporting which child died...
...consider reporting sort status...
...continue with the rest of the program...
}
You can decide whether to report errors related to dup2() failing, or close() failing. There isn't much you can do in either case except report the problem and exit. Unless someone has subjected your program to cruel and unusual punishment by not supplying it with standard input, standard output and standard error (or something elsewhere in the program has closed any of the standard channels), then the pipe and file descriptors can't be the standard I/O descriptors, so the closes are safe. If you're not sure how sick your users are, you might protect the closes:
if (p[0] > FILENO_STDERR)
close(p[0]);
That is normally unnecessarily paranoid (but it can be fun trying programs with missing standard I/O).
You don't need to pass sort any arguments to specify input source or output sink at all in this case. Instead, before execing it, you should make attach your pipeline's file descriptors to its stdin (FD 0, if receiving data from a pipe) or stdout (FD 1, if writing data to a pipe), as appropriate.
See the dup2() call, which lets you set the destination to which you're copying a FD, for this purpose. As #JonathanLeffler points out, you'll want to be sure to close the original FDs (after duplicating them to the numbers you want) before your exec call.
Since you've clarified, in comments, that your goal is to write to a file, you would attach FD 1 to that destination file before calling exec, with FD 0 attached to the output side of the pipeline containing input.
I want to do simple thing: my_process | proc2 | proc3, but programatically - without using shell, that can do this pretty easy. Is this possible? I cannot find anything :(
EDIT:
Well, without code, nobody will know, what problem I'm trying to resolve. Actually, no output is going out (I'm using printfs)
int pip1[2];
pipe(pip1);
dup2(pip1[1], STDOUT_FILENO);
int fres = fork();
if (fres == 0) {
close(pip1[1]);
dup2(pip1[0], STDIN_FILENO);
execlp("wc", "wc", (char*)0);
}
else {
close(pip1[0]);
}
Please learn about file descriptors and the pipe system call. Also, check read and write.
Your 'one child' code has some major problems, most noticeably that you configure the wc command to write to the pipe, not to your original standard output. It also doesn't close enough file descriptors (a common problem with pipes), and isn't really careful enough if the fork() fails.
You have:
int pip1[2];
pipe(pip1);
dup2(pip1[1], STDOUT_FILENO); // The process will write to the pipe
int fres = fork(); // Both the parent and the child will…
// Should handle fork failure
if (fres == 0) {
close(pip1[1]);
dup2(pip1[0], STDIN_FILENO); // Should close pip1[0] too
execlp("wc", "wc", (char*)0);
}
else { // Should duplicate pipe to stdout here
close(pip1[0]); // Should close pip1[1] too
}
You need:
fflush(stdout); // Print any pending output before forking
int pip1[2];
pipe(pip1);
int fres = fork();
if (fres < 0)
{
/* Failed to create child */
/* Report problem */
/* Probably close both ends of the pipe */
close(pip1[0]);
close(pip1[1]);
}
else if (fres == 0)
{
dup2(pip1[0], STDIN_FILENO);
close(pip1[0]);
close(pip1[1]);
execlp("wc", "wc", (char*)0);
}
else
{
dup2(pip1[1], STDOUT_FILENO);
close(pip1[0]);
close(pip1[1]);
}
Note that the amended code follows the:
Rule of thumb: If you use dup2() to duplicate one end of a pipe to standard input or standard output, you should close both ends of the original pipe.
This also applies if you use dup() or fcntl() with F_DUPFD.
The corollary is that if you don't duplicate one end of the pipe to a standard I/O channel, you typically don't close both ends of the pipe (though you usually still close one end) until you're finished communicating.
You might need to think about saving your original standard output before running the pipeline if you ever want to reinstate things.
As Alex answered, you'll need syscalls like pipe(2), dup2(2), perhaps poll(2) and some other syscalls(2) etc.
Read Advanced Linux Programming, it explains that quite well...
Also, play with strace(1) and study the source code of some simple free software shell.
See also popen(3) -which is not enough in your case-
Recall that stdio(3) streams are buffered. You probably need to fflush(3) at appropriate places (e.g. before fork(2))
I'm writing a C program where I fork(), exec(), and wait(). I'd like to take the output of the program I exec'ed to write it to file or buffer.
For example, if I exec ls I want to write file1 file2 etc to buffer/file. I don't think there is a way to read stdout, so does that mean I have to use a pipe? Is there a general procedure here that I haven't been able to find?
For sending the output to another file (I'm leaving out error checking to focus on the important details):
if (fork() == 0)
{
// child
int fd = open(file, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
dup2(fd, 1); // make stdout go to file
dup2(fd, 2); // make stderr go to file - you may choose to not do this
// or perhaps send stderr to another file
close(fd); // fd no longer needed - the dup'ed handles are sufficient
exec(...);
}
For sending the output to a pipe so you can then read the output into a buffer:
int pipefd[2];
pipe(pipefd);
if (fork() == 0)
{
close(pipefd[0]); // close reading end in the child
dup2(pipefd[1], 1); // send stdout to the pipe
dup2(pipefd[1], 2); // send stderr to the pipe
close(pipefd[1]); // this descriptor is no longer needed
exec(...);
}
else
{
// parent
char buffer[1024];
close(pipefd[1]); // close the write end of the pipe in the parent
while (read(pipefd[0], buffer, sizeof(buffer)) != 0)
{
}
}
You need to decide exactly what you want to do - and preferably explain it a bit more clearly.
Option 1: File
If you know which file you want the output of the executed command to go to, then:
Ensure that the parent and child agree on the name (parent decides name before forking).
Parent forks - you have two processes.
Child reorganizes things so that file descriptor 1 (standard output) goes to the file.
Usually, you can leave standard error alone; you might redirect standard input from /dev/null.
Child then execs relevant command; said command runs and any standard output goes to the file (this is the basic shell I/O redirection).
Executed process then terminates.
Meanwhile, the parent process can adopt one of two main strategies:
Open the file for reading, and keep reading until it reaches an EOF. It then needs to double check whether the child died (so there won't be any more data to read), or hang around waiting for more input from the child.
Wait for the child to die and then open the file for reading.
The advantage of the first is that the parent can do some of its work while the child is also running; the advantage of the second is that you don't have to diddle with the I/O system (repeatedly reading past EOF).
Option 2: Pipe
If you want the parent to read the output from the child, arrange for the child to pipe its output back to the parent.
Use popen() to do this the easy way. It will run the process and send the output to your parent process. Note that the parent must be active while the child is generating the output since pipes have a small buffer size (often 4-5 KB) and if the child generates more data than that while the parent is not reading, the child will block until the parent reads. If the parent is waiting for the child to die, you have a deadlock.
Use pipe() etc to do this the hard way. Parent calls pipe(), then forks. The child sorts out the plumbing so that the write end of the pipe is its standard output, and ensures that all other file descriptors relating to the pipe are closed. This might well use the dup2() system call. It then executes the required process, which sends its standard output down the pipe.
Meanwhile, the parent also closes the unwanted ends of the pipe, and then starts reading. When it gets EOF on the pipe, it knows the child has finished and closed the pipe; it can close its end of the pipe too.
Since you look like you're going to be using this in a linux/cygwin environment, you want to use popen. It's like opening a file, only you'll get the executing programs stdout, so you can use your normal fscanf, fread etc.
After forking, use dup2(2) to duplicate the file's FD into stdout's FD, then exec.
You could also use the linux sh command and pass it a command that includes the redirection:
string cmd = "/bin/ls > " + filepath;
execl("/bin/sh", "sh", "-c", cmd.c_str(), 0);
For those such as myself who like a complete example with includes, here's this fantastic answer with a runnable example (still without error handling, left as an exercise):
#include <fcntl.h>
#include <sys/wait.h>
#include <unistd.h>
int main() {
if (fork() == 0) { // child
int fd = open("test.txt", O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
dup2(fd, 1); // make stdout go to file
dup2(fd, 2); // make stderr go to file - you may choose to not do this
// or perhaps send stderr to another file
close(fd); // fd no longer needed - the dup'ed handles are sufficient
execlp("ls", "ls", NULL);
}
else {
while (wait(NULL) > 0) {} // wait for each child process
}
return 0;
}