im trying to execute md5sume command in my programm using pipe,fork and dup.i found sum code that run succesfully but i cant understand some line of code. Here is my code:
int infp, outfp;
char buf[128];
if (popen2("md5sum", &infp, &outfp) <= 0)
{
printf("Unable to exec sort\n");
exit(1);
}
write(infp, "hello\n", 2);
close(infp);
*buf = '\0';
read(outfp, buf, 128);
printf("buf = '%s'\n", buf);
return 0;
}
int p_stdin[2], p_stdout[2];
pid_t pid;
if (pipe(p_stdin) != 0 || pipe(p_stdout) != 0)
return -1;
pid = fork();
if (pid < 0)
return pid;
if (pid == 0)
{
close(p_stdin[WRITE]);
dup2(p_stdin[READ], READ);
close(p_stdout[READ]);
dup2(p_stdout[WRITE], WRITE);
execl("/bin/sh", "sh", "-c", command, NULL);
perror("execl");
exit(1);
}
else
{
if (infp == NULL)
close(p_stdin[WRITE]);
else
*infp = p_stdin[WRITE];
if (outfp == NULL)
close(p_stdout[READ]);
else
*outfp = p_stdout[READ];
}
return pid;
}
i dont understand the popen function. What does this line exactly do?
*infp = p_stdin[WRITE];
how can pipes comunicate with each other?
i dont understand the popen function.
how can pipes comunicate with each other?
pipe() : A pipe is unidirectional and a byte stream buffer in kernel. As it is of type byte stream, a writer can write in arbitrary number of bytes and reader can read out arbitrary number of bytes. However, note that sequential reads are possible , but seek (like lseek) is not possible. Since pipe is uni-directinal, the data that is written into pipe shall be buffered in kernel, until it is read from the read-end of the pipe. Also, if pipe gets full, the write blocks.
Let's consider that fd is an integer array of 2 file descriptors (int fd[2]), then the pipe(fd) system call shall create a pipe and return a pair of file descriptors such that fd[1] (stdout is 1) shall be the write-end of the pipe and the fd[0] (stdin is 0) shall be the read-end of the pipe. Unlike named pipe(like FIFO - a pipe with name in File system), the anonymous pipes can be used only between related processes like parent-child. So, fork shall be done to duplicate these 2 parent file descriptors in child, thereby parent shares the pipe with child so that the child shall write in write-end and parent shall read from read-end of pipe or Parent shall write into write-end and child shall read from read-end of pipe. Care should be taken to ensure to close the unused read(fd[0]) file descriptor / unused write(fd[1]) file descriptor by the parent or child as per the scenario.
popen() : popen enables you to invoke another program as a new process and thereby transmit data to it or receive data from it. In case of popen, note that the direction of data flow is based on the 2nd argument. We need not manually create a child process as popen automatically forks for creating a child process, starts a shell and executes the command argument passed via popen. It also establishes appropriate read or write stream between parent and child automatically based on the type argument.
Thus, popen() simplifies things, as it avoids the need to manually call/invoke pipe,fork,exec and simplifies establishment of appropriate streams between parent / child automatically as per argument type. However, the other side of popen is that, it should be noted that every invocation of the popen() shall result in creation of extra process - that is, the shell is invoked every time apart from the program that is being invoked which in-turn leads to high resource consumption.
Related
A pipe connects the stdout of one process to the stdin of another: https://superuser.com/a/277327
Here is a simple program to take input from stdin and print it:
int main( ) {
char str[100];
gets( str );
puts( str );
return 0;
}
I can use a unix pipe to pass the input from another process:
echo "hi" | ./a.out
My question is, what is the difference between the simple code above and using the pipe() system call? Does the system call essentially do the same job without writing to the terminal? More on Pipes: https://tldp.org/LDP/lpg/node11.html
The pipe() system call allows you to get file descriptors (one for reading and one for writing) for a channel (a pipe) that allows to stream bytes through multiple processes. This is an example where a parent process creates a pipe and its child writes to it so the parent can read from it:
int main() {
int fd[2];
pipe(fd);
int pid = fork();
if (pid == 0) { // Child:
close(fd[0]); // Close reading descriptor as it's not needed
write(fd[1], "Hello", 5);
} else { // Parent:
char buf[5];
close(fd[1]); // Close writing descriptor as it's not needed
read(fd[0], buf, 5); // Read the data sent by the child through the pipe
write(1, buf, 5); // print the data that's been read to stdout
}
}
When a shell encounters the pipe (|) operator, it does use the pipe() system call, but also does additional things, in order to redirect the left operand's stdout and the right operand's stdin to the pipe. Here's a simplified example of what the shell would do for the command echo "hi" | ./a.out (keep in mind that when duplicating a file descriptor it gets duplicated to the first index available in the open files structure of the process):
int main() {
int fd[2];
pipe(fd);
int pid_echo = fork();
if (pid_echo == 0) {
// Close reading descriptor as it's not needed
close(fd[0]);
// Close standard output
close(1);
// Replace standard output with the pipe by duplicating its writing descriptor
dup(fd[1]);
// Execute echo;
// now when echo prints to stdout it will actually print to the pipe
// because now file descriptor 1 belongs to the pipe
execlp("echo", "echo", "hi", (char*)NULL);
exit(-1);
}
int pid_aout = fork();
if (pid_aout == 0) {
// Close standard input
close(0);
// Replace standard input with the pipe by duplicating its reading descriptor
dup(fd[0]);
// Execute a.out;
// Now when a.out reads from stdin it will actually read from the pipe
// because now file descriptor 0 belongs to the pipe
execl("./a.out", "./a.out", (char*)NULL);
exit(-1);
}
}
A pipe is an inter-process communication mechanism that leverages I/O redirection. However, pipes are not involved in all I/O redirection.
Since child processes may inherit file descriptors from their parent process, a parent process may change what files the child's standard streams point to, unbeknownst to the child process. This is I/O redirection.
I want to communicate with a child process like the following:
int main(int argc, char *argv[])
{
int bak, temp;
int fd[2];
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[0]);
dup2(STDOUT_FILENO, fd[1]);
fflush(stdout);
bak = dup(1);
temp = open("/dev/null", O_WRONLY);
dup2(temp, 1);
close(temp );
Mat frame;
std::vector<uchar> buf;
namedWindow( "Camera", WINDOW_AUTOSIZE );
VideoCapture cam(0 + CAP_V4L);
sleep(1);
if (!cam.isOpened())
{
cout << "\nCould not open reference " << 0 << endl;
return -1;
}
for (int i=0; i<30; i++)
{
cam>>frame;
}
//cout<<"\nCamera initialized\n";
/*Set the normal STDOUT back*/
fflush(stdout);
dup2(bak, 1);
close(bak);
imencode(".png",frame, buf);
cout<<buf.size()<<endl;
ssize_t written= 0;
size_t s = 128;
while (written<buf.size())
{
written += write(fd[1], buf.size()+written, s);
}
cout<<'\0';
return 0;
}
The process corresponding to the compilation of the source code above is called from the parent with popen.
Note that I am writing to the std out that has been duplicated with a pipe.
The parent will read the data and resend them to UDP socket.
If I do something like this:
#define BUFLEN 128
FILE *fp;
char buf[BUFLEN];
if ((fp = popen("path/to/exec", "r")) != NULL)
{
while((fgets(buf, BUFLEN, fp)!=NULL))
{
sendto(sockfd, buf, strlen(buf),0, addr, alen);
}
}
the program is working i.e. the receiver of sendto will receive the data.
I tried to use a pipe as done in the child process:
int fd[2];
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[1]);
dup2(STDIN_FILENO, fd[0]);
if ((fp = popen("path/to/exec", "r")) != NULL)
{
while((read(fd[0], buf, BUFLEN) > 0)
{
sendto(sockfd, buf, strlen(buf),0, addr, alen);
}
}
but with this are not sent.
So how to use pipe in this case to achieve the same behaviour of the first case? Should I do dup2(STDIN_FILENO, fd[0]); or dup2(STDOUT_FILENO, fd[0]);?
I am using the sandard(s) since the file descriptors are inherited by the child process so should not require any other effort. That is why I thought I can use pipe but is that so?
In the parent:
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[0]);
you get a pipe, and then immediately close one end of it. This pipe is now useless, because no-one will ever be able to recover the closed end, and so no data can flow through it. You have converted a pipe into a hollow cylinder sealed at one end.
Then in the child:
if (pipe(fd) < 0)
{
// pipe error
exit(1);
}
close(fd[1]);
you create another unrelated pipe, and seal this at the other end. The two pipes are not connected, and now you have two separate hollow cyclinders, each sealed at one end. Nothing can flow through either of them.
If putting something in the first cylinder made it appear in the other, that'd be a pretty good magic trick. Without sleight of hand or cleverly arranged mirrors, the solution is to create one pipe, keep both ends open and push data through it.
The usual way to manually set up a pipe from which a parent process can read a child process's standard output has these general steps:
parent creates a pipe by calling pipe()
parent fork()s
parent closes (clarification: its copy of) the write end of the pipe
child dupes the write end of the pipe onto its standard output via dup2()
child closes the original file descriptor for the write end of the pipe
(optional) child closes (clarification: its copy of) the read end of the pipe
child execs the desired command, or else performs the wanted work directly
The parent can then read the child's output from the read end of the pipe.
The popen() function does all of that for you, plus wraps the parent's pipe end in a FILE. Of course, it can and will set up a pipe going in the opposite direction instead if that's what the caller requests.
You need to understand and appreciate that in the procedural scheme presented above, it is important which actions are performed by which process, and in what order relative to other actions in the same process. In particular, the parent must not close the write end of the pipe before the child is launched, because that renders the pipe useless. The child inherits the one-end-closed pipe, through which no data can be conveyed.
With respect to your latter example, note also that redirecting the standard input to the read end of the pipe is not part of the process for either parent or child. The fact that your pipe is half-closed, so that nothing can ever be read from it anyway, is just icing on the cake. Moreover, the parent clobbers its own standard input this way. That's not necessarily wrong, but the parent does not even rely on it.
Overall, however, there is a bigger picture that you seem not to appreciate. Even if you performed the redirection you seem to want in the parent, so that it could be inherited by the child, popen() performs its own redirection to a pipe of its own creation. The FILE * it returns is the means by which you can read the child's output. No previous output redirection you may have performed is relevant (clarification: of the child's standard output).
In principle, an approach similar to yours could be used to create a second redirection going the other way, but at that point the convenience factor of popen() is totally lost. It would be better go take the direct pipe / fork / dup2 / exec route all the way through if you want to redirect the child's input and output.
Applying all that to your first example, you have to appreciate that although a process can redirect its own standard streams, it cannot establish a pipe to its parent process that way. The parent needs to provide the pipe, else it has no knowledge of it. And when a process dupes one file descriptor onto another, that replaces the original with the new, closing the original if it is open. It does not redefine the original. And of course, in this case, too, a pipe is useless once either end is no longer open anywhere.
so in this code snippet a process forked a child process. the child process calculated a random number r, and called linux command 'head -r "file' with an exec function which destroys the process itself, but to send the result back to the parent process the child process first duplicated the writing end of a pipe p, shared with the parent process and then closed both ends of the pipe p and closed the stdout file descriptor too...after execlp the parent process could read the result of the command 'head -r "fil2"' from the pipe p.
How is this possible?
if (pid == 0)
{
/* code of child */
srand(time(NULL));
nr=atoi(argv[(i*2)+2]);
r=mia_random(nr); //calc random value
close(1); //closing standard output???
dup(p[1]); //duplicating write end of inherited pipe from parent
close(p[0]);//closing read end of inherited pipe
close(p[1]);//closing write end of inherited pipe
//creating a variable to hold an argument for `head`
sprintf(option, "-%d", r);
//calling head on a file given as argument in main
execlp("head", "head", option, argv[(i*2)+1], (char *)0);
/* must not be here anymore*/
/* using perror to check for errors since stdout is closed? or connected to the pipe?*/
perror("Problem esecuting head by child process");
exit(-1);
}
Why wasn't the result of head written to stderr instead? How come it was written to the dup(p[1])???
The system is guaranteed to open each new file at the lowest possible file descriptor.
Effectively, that means that if fd 0 and 1 are open and p[1] != 1, then
close(1);
dup(p[1]);
in a single-threaded process is equivalent to
dup2(p[1],1);
or in other words, if the dup call in this context succeeds, it will return (filedescriptor) 1.
I am creating a C program and with it I am setting up a pipe between separately forked process for interprocess communication.
The first process has written the data I need into the pipe.
However, with the second process reading from the pipe, I am trying to exec the process to become the UNIX sort command. I want to somehow call sort on the data in the pipe.
How can I call sort on a pipe? On the commandline, I can sort by supplying the filename to sort as a commandline argument e.g. "sort -r MyFileToSort". I know that pipes are essentially considered files, but they are only described by their file descriptor, and as far as I know, sort won't know what to do with a fd.
Thanks for any help/feedback
int p[2];
if (pipe(p) != 0) ...report error and do not continue...
pid_t pid = fork();
if (pid < 0) ...report error, close pipe descriptors, and do not continue...
if (pid == 0)
{
/* Child - becomes sort */
dup2(p[0], 0);
close(p[0]);
close(p[1]);
int fd = open("output-file", O_CREAT | O_EXCL | O_WRONLY, 0644);
if (fd < 0) ...report error and exit...
dup2(fd, 1);
close(fd);
execlp("sort", "sort", (char *)0);
...report error and exit...
}
else
{
/* Parent - writes data to sort */
close(fd[0]);
...write data to fd[1]...
close(fd[1]);
int status;
int corpse;
while ((corpse = wait(&status)) > 0 && corpse != pid)
...consider reporting which child died...
...consider reporting sort status...
...continue with the rest of the program...
}
You can decide whether to report errors related to dup2() failing, or close() failing. There isn't much you can do in either case except report the problem and exit. Unless someone has subjected your program to cruel and unusual punishment by not supplying it with standard input, standard output and standard error (or something elsewhere in the program has closed any of the standard channels), then the pipe and file descriptors can't be the standard I/O descriptors, so the closes are safe. If you're not sure how sick your users are, you might protect the closes:
if (p[0] > FILENO_STDERR)
close(p[0]);
That is normally unnecessarily paranoid (but it can be fun trying programs with missing standard I/O).
You don't need to pass sort any arguments to specify input source or output sink at all in this case. Instead, before execing it, you should make attach your pipeline's file descriptors to its stdin (FD 0, if receiving data from a pipe) or stdout (FD 1, if writing data to a pipe), as appropriate.
See the dup2() call, which lets you set the destination to which you're copying a FD, for this purpose. As #JonathanLeffler points out, you'll want to be sure to close the original FDs (after duplicating them to the numbers you want) before your exec call.
Since you've clarified, in comments, that your goal is to write to a file, you would attach FD 1 to that destination file before calling exec, with FD 0 attached to the output side of the pipeline containing input.
I'm implementing piping on a simulated file system in C++ (with mostly C). It needs to run commands in the host shell but perform the piping itself on the simulated file system.
I could achieve this with the pipe(), fork(), and system() system calls, but I'd prefer to use popen() (which handles creating a pipe, forking a process, and passing a command to the shell). This may not be possible because (I think) I need to be able to write from the parent process of the pipe, read on the child process end, write the output back from the child, and finally read that output from the parent. The man page for popen() on my system says a bidirectional pipe is possible, but my code needs to run on a system with an older version supporting only unidirectional pipes.
With the separate calls above, I can open/close pipes to achieve this. Is that possible with popen()?
For a trivial example, to run ls -l | grep .txt | grep cmds I need to:
Open a pipe and process to run ls -l on the host; read its output back
Pipe the output of ls -l back to my simulator
Open a pipe and process to run grep .txt on the host on the piped output of ls -l
Pipe the output of this back to the simulator (stuck here)
Open a pipe and process to run grep cmds on the host on the piped output of grep .txt
Pipe the output of this back to the simulator and print it
man popen
From Mac OS X:
The popen() function 'opens' a
process by creating a bidirectional
pipe, forking, and invoking the shell.
Any streams opened by previous popen()
calls in the parent process are closed
in the new child process.
Historically, popen() was implemented
with a unidirectional pipe; hence,
many implementations of popen() only
allow the mode argument to specify
reading or writing, not both. Because
popen() is now implemented using a
bidirectional pipe, the mode argument
may request a bidirectional data flow.
The mode argument is a pointer to a
null-terminated string which must be
'r' for reading, 'w' for writing, or
'r+' for reading and writing.
I'd suggest writing your own function to do the piping/forking/system-ing for you. You could have the function spawn a process and return read/write file descriptors, as in...
typedef void pfunc_t (int rfd, int wfd);
pid_t pcreate(int fds[2], pfunc_t pfunc) {
/* Spawn a process from pfunc, returning it's pid. The fds array passed will
* be filled with two descriptors: fds[0] will read from the child process,
* and fds[1] will write to it.
* Similarly, the child process will receive a reading/writing fd set (in
* that same order) as arguments.
*/
pid_t pid;
int pipes[4];
/* Warning: I'm not handling possible errors in pipe/fork */
pipe(&pipes[0]); /* Parent read/child write pipe */
pipe(&pipes[2]); /* Child read/parent write pipe */
if ((pid = fork()) > 0) {
/* Parent process */
fds[0] = pipes[0];
fds[1] = pipes[3];
close(pipes[1]);
close(pipes[2]);
return pid;
} else {
close(pipes[0]);
close(pipes[3]);
pfunc(pipes[2], pipes[1]);
exit(0);
}
return -1; /* ? */
}
You can add whatever functionality you need in there.
You seem to have answered your own question. If your code needs to work on an older system that doesn't support popen opening bidirectional pipes, then you won't be able to use popen (at least not the one that's supplied).
The real question would be about the exact capabilities of the older systems in question. In particular, does their pipe support creating bidirectional pipes? If they have a pipe that can create a bidirectional pipe, but popen that doesn't, then I'd write the main stream of the code to use popen with a bidirectional pipe, and supply an implementation of popen that can use a bidirectional pipe that gets compiled in an used where needed.
If you need to support systems old enough that pipe only supports unidirectional pipes, then you're pretty much stuck with using pipe, fork, dup2, etc., on your own. I'd probably still wrap this up in a function that works almost like a modern version of popen, but instead of returning one file handle, fills in a small structure with two file handles, one for the child's stdin, the other for the child's stdout.
POSIX stipulates that the popen() call is not designed to provide bi-directional communication:
The mode argument to popen() is a string that specifies I/O mode:
If mode is r, when the child process is started, its file descriptor STDOUT_FILENO shall be the writable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the readable end of the pipe.
If mode is w, when the child process is started its file descriptor STDIN_FILENO shall be the readable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the writable end of the pipe.
If mode is any other value, the result is unspecified.
Any portable code will make no assumptions beyond that. The BSD popen() is similar to what your question describes.
Additionally, pipes are different from sockets and each pipe file descriptor is uni-directional. You would have to create two pipes, one configured for each direction.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in popen simultaneous read and write)
Here's the code (C++, but can be easily converted to C):
#include <unistd.h>
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <utility>
// Like popen(), but returns two FILE*: child's stdin and stdout, respectively.
std::pair<FILE *, FILE *> popen2(const char *__command)
{
// pipes[0]: parent writes, child reads (child's stdin)
// pipes[1]: child writes, parent reads (child's stdout)
int pipes[2][2];
pipe(pipes[0]);
pipe(pipes[1]);
if (fork() > 0)
{
// parent
close(pipes[0][0]);
close(pipes[1][1]);
return {fdopen(pipes[0][1], "w"), fdopen(pipes[1][0], "r")};
}
else
{
// child
close(pipes[0][1]);
close(pipes[1][0]);
dup2(pipes[0][0], STDIN_FILENO);
dup2(pipes[1][1], STDOUT_FILENO);
execl("/bin/sh", "/bin/sh", "-c", __command, NULL);
exit(1);
}
}
Usage:
int main()
{
auto [p_stdin, p_stdout] = popen2("cat -n");
if (p_stdin == NULL || p_stdout == NULL)
{
printf("popen2() failed\n");
return 1;
}
const char msg[] = "Hello there!";
char buf[32];
printf("I say \"%s\"\n", msg);
fwrite(msg, 1, sizeof(msg), p_stdin);
fclose(p_stdin);
fread(buf, 1, sizeof(buf), p_stdout);
fclose(p_stdout);
printf("child says \"%s\"\n", buf);
return 0;
}
Possible Output:
I say "Hello there!"
child says " 1 Hello there!"
No need to create two pipes and waste a filedescriptor in each process. Just use a socket instead. https://stackoverflow.com/a/25177958/894520