My end goal is to have a parent process pass lines of text to the child, then the child process will print the text to stdout. The child is to run "permanently" in the back ground while the parent gets user input and pass's it to the child. I prefer the child in a separate program. Differentiating between child and paretn through if statements is messy as fudge.
I was looking into pipes but I'm unsure if it's even possible for pipes to communicate between a parent/child after the child has called exec() to a different program.
Is this possible? If so, is there any example you can point me to? If not, what method of IPC can I use in that case?
The standard scenario is to have the program executed as a child be agnostic of the pipe and just use stdin / stdout. You achieve this by dup2()ing the respective end of the pipe as fd 0 or 1 (or both with two pipes for bidirectional communication), corresponding to STDIN_FILENO and STDOUT_FILENO. After this, exec your child program.
Of course, there are alternatives like e.g. "named pipes" if you need stdin / stdout for a different purpose in the child.
Still if you write both parts yourself, you might want to think about simpler solutions:
Differentiating between child and parent through if statements is messy as fudge.
You have to do this anyways, at least for wiring up the pipes and calling exec(). Just create separate code files and call things like parent_main() and child_main() as appropriate (whatever you like to call them).
After exec*() functions the child shares all the file descriptors of the parent process. So if you create a pipe before fork() you have access to read/write fd in both of them.
Usual method is:
create a pipe (a read fd, a write fd)
fork()
in parent:
close read fd (you will not read from parent here)
write data into write fd
wait/die
in child:
close write fd (you will not write from child)
read data from read fd
wait/die
Can't see why you want to exec a new process. If really needed you have to use standard stdin/stdout (see other answer) or have a program that will accept a fd (filedescriptor, an integer) as parameter in order to know which one is the pipe. Don't seems very nice to me.
Actually it is much easier to do it with popen which manages the communication channels automatically
FILE *popen(const char *command, const char *mode);
The popen() function shall execute the command specified by the string
command. It shall create a pipe between the calling program and the
executed command, and shall return a pointer to a stream that can be
used to either read from or write to the pipe.
The environment of the executed command shall be as if a child process
were created within the popen() call using the fork() function, and
the child invoked the sh utility using the call:
execl(shell path, "sh", "-c", command, (char *)0);
where shell path is an unspecified pathname for the sh utility.
The popen() function shall ensure that any streams from previous
popen() calls that remain open in the parent process are closed in the
new child process.
The mode argument to popen() is a string that specifies I/O mode:
If mode is r, when the child process is started, its file descriptor
STDOUT_FILENO shall be the writable end of the pipe, and the file
descriptor fileno(stream) in the calling process, where stream is the
stream pointer returned by popen(), shall be the readable end of the
pipe.
If mode is w, when the child process is started its file descriptor
STDIN_FILENO shall be the readable end of the pipe, and the file
descriptor fileno(stream) in the calling process, where stream is the
stream pointer returned by popen(), shall be the writable end of the
pipe.
If mode is any other value, the result is undefined.
After popen(), both the parent and the child process shall be capable
of executing independently before either terminates.
Pipe streams are byte-oriented.
RETURN VALUE
Upon successful completion, popen() shall return a pointer to an open
stream that can be used to read or write to the pipe. Otherwise, it
shall return a null pointer and may set errno to indicate the error.
For example suppose you want to send Hello world to the child:
#include<stdio.h>
#include<stdlib.h>
#include <unistd.h>
int main()
{
FILE* toChild;
toChild = popen("./child", "w");//child is executable of the other file: change the name
int res = fputs( "Hello World\n", toChild);
pclose(toChild);
return 0;
}
and the child:
int main()
{
char p[100];
int n;
do{
n = scanf("%s", p);
if (n>0) {
printf("INPUT MESSAGE: \"%s\n\"", p);
//free(p);
}
else {
printf( "%d, No matching characters\n", n);
}
}while(n>0);
return 0;
}
You can also use scanf("%ms", &p) with char* p and then free(p) if you are on a pure POSIX system (not on OSX).
Related
I'm trying to implement a Linux pipe chain in C. For example:
grep file | ls | wc
So, there is a code that splits the arguments into tokens with the pipe as the separator, and sends each part to the following function with an integer specifying whether it precedes a pipe or not:
int control_flow(char** args, int precedes){
int stdin_copy = dup(0);
int stdout_copy = dup(1);
// if the command and its args precedes a pipe
if (precedes){
int fd[2];
if (pipe(fd) == -1){
fprintf(stderr, "pipe failed\n");
}
if (dup2(fd[1], 1)!=1)
perror("dup2 error 1 to p_in\n"); // 1 points to pipe's input
status = turtle_execute(args); // executes the argument list, output should go into the pipe
// Code stops running here
if (dup2(fd[0], 0)!=0)
perror("dup2 error 0 to p_out\n"); // 0 points to pipe's output, any process that reads next will read from the pipe
if (dup2(stdout_copy, 1)!=1)
perror("dup2 error 1 to stdout_copy\n"); // 1 points back to stdout
}
// if the command does not precede a pipe
else{
status = turtle_execute(args); // input to this is coming from pipe
if (dup2(stdin_copy, 0)!=0) // 0 points back to stdin
perror("dup2 error 1 to stdin_copy");
}
return 0;
}
My code stops running after the first command executes. I suspect it is necessary to fork a process before using this pipe, why is that? If so, how do I do that in my code without changing what I intend to do?
Edit:
This is roughly what turtle_execute does:
turtle_execute(args){
if (args[0] is cd or ls or pwd or echo)
// Implement by calling necessary syscalls
else
// Do fork and exec the process
So wherever I have used exec, I have first used fork, so process getting replaced shouldn't be a problem.
The exec system call replaces the current process with the program you are executing. So your process naturally stops working after the turtle_execute, since it was replaced with the new process.
To execute a new process you normally fork to create a copy of the current process and then execute in the copy.
When you are in the shell, normally each command you type is forked and executed. Try typing exec followed by a command into a shell and you will find that the shell terminates once that command has finished executing, since it does not fork in that case.
Edit
I suggest you have a look at the example on the pipe(2) man page (http://man7.org/linux/man-pages/man2/pipe.2.html#EXAMPLE). It shows the usual way of using a pipe:
Calling pipe to get the create the pipe
Calling fork to fork the process
Depending on whether it is child or parent close one end of the pipe and use the other
I think your problem might be that you make the writing end of your pipe the stdout before forking, causing both the parent and the child to have an open writing end. That could prevent an EOF to be sent since one writing end is still open.
I can only guess what happens in most of turtle_execute, but if you fork, exec on one process, and wait for it on the other, without consuming data from the pipe, it might fill the pipe and to the point where writing is blocked. You should always consume data from the pipe while you write to it. It is a pipe after all and not a water tank. For more information have a look at the pipe(7) man page under the 'Pipe capacity' section.
From the below program,
/*****************************************************************************
MODULE: popen.c
*****************************************************************************/
#include <stdio.h>
int main(void)
{
FILE *pipein_fp, *pipeout_fp;
char readbuf[80];
/* Create one way pipe line with call to popen() */
if (( pipein_fp = popen("ls", "r")) == NULL)
{
perror("popen");
exit(1);
}
/* Create one way pipe line with call to popen() */
if (( pipeout_fp = popen("sort", "w")) == NULL)
{
perror("popen");
exit(1);
}
/* Processing loop */
while(fgets(readbuf, 80, pipein_fp))
fputs(readbuf, pipeout_fp);
/* Close the pipes */
pclose(pipein_fp);
pclose(pipeout_fp);
return(0);
}
popen.c is compiled to my_program
Here is my understanding of the file descriptors created/involved after executing my_program, popen() forks & execs the my_program process but child processes does not inherit the pipe file descriptors of my_program.
So, After exec,
1) write file descriptor is only created for ls
2) read file descriptor is only created for sort
3) read and write files descriptors are created in my_program, because ls writes to my_program & sort reads from my_program
As shown above, Are these the only file descriptors involved/created?
Note: 'in' & 'out' are just naming conventions used here
Child processes from a fork() have exactly the same set of open file descriptors as the parent process.
The popen() call uses pipe() to create two file descriptors; it then executes fork(). The parent process arranges that one end of the pipe is closed and the other converted to a file stream (FILE *). The child process closes the other end of the pipe and arranges for the one end to become standard input ("w") or standard output ("r") for the process it executes (using dup() or dup2() for the task).
You use popen() twice; you end up with 2 open descriptors in the parent, and transiently there's a third.
When you say, 'the parent process arranges that one end of the pipe is closed', do you mean read file descriptor (stdin)?
Depending on the mode argument to popen(), one of the two ends of the pipe in the parent is closed immediately; the other is closed by pclose(). The file descriptor is 'never' the one for standard input or standard output — you have to go through extraordinary gyrations to make it so that is one of the standard I/O channels.
Do all process thru popen() make sure they dup() to make sure they use stdout & stdin?
Each pipe has a read end and a write end. Take popen("ls", "r"); your program reads from the ls process. It (popen()) creates a pipe and forks. In the child, the write end of the pipe is connected to stdout (dup2() or perhaps dup()), and the read end of the pipe is closed, before the command is executed. In the parent, the read end of the pipe is 'converted to' or 'attached to' a stream (fdopen(), more or less) and the write end of the pipe is closed. In the parent process, the pipe is never connected to either stdout or stdin.
In the child process, either standard input or standard output is connected to the pipe, depending on the mode argument to popen().
This is the code i found for my own shell. It works fine, but the thing i can't understand is pipe section of the code.
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
char* cmndtkn[256];
char buffer[256];
char* path=NULL;
char pwd[128];
int main(){
//setting path variable
char *env;
env=getenv("PATH");
putenv(env);
system("clear");
printf("\t MY OWN SHELL !!!!!!!!!!\n ");
printf("_______________________________________\n\n");
while(1){
fflush(stdin);
getcwd(pwd,128);
printf("[MOSH~%s]$",pwd);
fgets(buffer,sizeof(buffer),stdin);
buffer[sizeof(buffer)-1] = '\0';
//tokenize the input command line
char* tkn = strtok(buffer," \t\n");
int i=0;
int indictr=0;
// loop for every part of the command
while(tkn!=NULL)
{
if(strcoll(tkn,"exit")==0 ){
exit(0);
}
else if(strcoll(buffer,"cd")==0){
path = buffer;
chdir(path+=3);
}
else if(strcoll(tkn,"|")==0){
indictr=i;
}
cmndtkn[i++] = tkn;
tkn = strtok(NULL," \t\n");
}cmndtkn[i]='\0';
// execute when command has pipe. when | command is found indictr is greater than 0.
if(indictr>0){
char* leftcmnd[indictr+1];
char* rightcmnd[i-indictr];
int a,b;
for(b=0;b<indictr;b++)
leftcmnd[b]=cmndtkn[b];
leftcmnd[indictr]=NULL;
for(a=0;a<i-indictr-1;a++)
rightcmnd[a]=cmndtkn[a+indictr+1];
rightcmnd[i-indictr]=NULL;
if(!fork())
{
fflush(stdout);
int pfds[2];
pipe(pfds);
if(!fork()){
close(1);
dup(pfds[1]);
close(pfds[0]);
execvp(leftcmnd[0],leftcmnd);
}
else{
close(0);
dup(pfds[0]);
close(pfds[1]);
execvp(rightcmnd[0],rightcmnd);
}
}else
wait(NULL);
//command not include pipe
}else{
if(!fork()){
fflush(stdout);
execvp(cmndtkn[0],cmndtkn);
}else
wait(NULL);
}
}
}
What is the purpose of the calls to close() with parameters of 0 and 1 mean and what does the call to dup() do?
On Unix, the dup() call uses the lowest numbered unused file descriptor. So, the close(1) before the call to dup() is to coerce dup() to use file descriptor 1. Similarly for close(0).
So, the aliasing is to get the process to use the write end of the pipe for stdout (file descriptor 1 is used for console output), and the read end of the pipe for stdin (file descriptor 0 is used for console input).
The code may have been more clearly expressed with dup2() instead.
dup2(fd[1], 1); /* alias fd[1] to 1 */
From your question about how ls | sort works, your question is not limited to why the dup() system call is being made. Your question is actually how pipes in Unix work, and how a shell command pipeline works.
A pipe in Unix is a pair of file descriptors that are related in that writing data on tje writable descriptor allows that data to be read from the readable descriptor. The pipe() call returns this pair in an array, where the first array element is readable, and second array element is writable.
In Unix, a fork() followed by some kind of exec() is the only way to produce a new process (there are other library calls, such as system() or popen() that create processes, but they call fork() and do an exec() under the hood). A fork() produces a child process. The child process sees the return value of 0 from the call, while the parent sees a non-zero return value that is either the PID of the child process, or a -1 indicating that an error has occurred.
The child process is a duplicate of the parent. This means that when a child modifies a variable, it is modifying a copy of the variable that resides in its own process. The parent does not see the modification occur, as the parent has the original copy). However, a duplicated pair of file descriptors that form a pipe can be used to allow a child process its parent to communicate with each other.
So, ls | sort means that there are two processes being spawned, and the output written by ls is being read as input by sort. Two processes means two calls to fork() to create two child processes. One child process will exec() the ls command, the other child process will exec() the sort command. A pipe is used between them to allow the processes to talk to each other. The ls process writes to the writable end of the pipe, the sort process reads from the readable end of the pipe.
The ls process is coerced into writing into the writable end of the pipe with the dup() call after issuing close(1). The sort process is coerced into reading the readable end of the pipe with the dup() call after close(0).
In addition, the close() calls that close the pipe file descriptors are used to make sure that the ls process is the only process to have an open reference to the writable fd, the the sort process is the only process to have an open reference to the readable fd. That step is important because after ls exits, it will close the writable end of the fd, and the sort process will expect to see an EOF as a result. However, this will not occur if some other process still has the writable fd open.
http://en.wikipedia.org/wiki/Standard_streams#Standard_input_.28stdin.29
stdin is file descriptor 0.
stdout is file descriptor 1.
In the !fork section, the process closes stdout then calls dup on pfds[1] which according to:
http://linux.die.net/man/2/dup
Creates a duplicate of the specified file descriptor at the lowest available position, which will be 1, since it was just closed (and stdin hasn't been closed yet). This means everything sent to stdout will really go to pfds[1].
So, basically, it's setting up the two new processes to talk to each other. the !fork section is for the new child which will send data to stdout (file descriptor 1), the parent (the else block) closes stdin, so it really reads from pfds[0] when it tries to read from stdout.
Each process has to close the file descriptor in pfds it's not using, as there are two open handles to the file now that the process has forked. Each process now execs to left/right-cmnd, but the new stdin and stdout mappings remain for the new processes.
Forking twice is explained here: Why fork() twice
Is it necessary to replace stdin with a pipe end when using pipes?
I have an application that:-
Creates a pipe,
Forks a child process, and then
execl() a new process image within new child process,
But I'm running into two conceptual issues.
Is it necessary to use dup() or dup2() to replace stdin? It would obviously be easier to just use the fd from the pipe. (I need little insight about this)
If you can just use the fd from the pipe, how do you pass an integer fd using execl() when execl takes char * arguments?
I'm having trouble figuring out exactly what remains open after execl() is performed, and how to access that information from the newly execl'd process.
It depends on the commands you're running. However, many Unix commands read from standard input and write to standard output, so if the pipes are not set up so that the write end is the output of one command and the read end is the input of the next command, nothing happens (or, more accurately, programs read from places where the input isn't coming from, or write to places which will not be read from, or hang waiting for you to type the input at the terminal, or otherwise do not work as intended).
If your pipe is on file descriptors 3 and 4, the commands you execute must know to read from 3 and write to 4. You could handle that with shell, but it is moderately grotesque overkill to do so compared with using dup2().
No; you're not obliged to use dup2(), but it is generally easier to do so. You could close standard output and then use plain dup() instead of dup2().
If you use dup2() for a pipe, don't forget to close both of the original file descriptors.
You are probably trying to feed data to a subprocess that exists on the system but on the off chance that you are also writing the child process then no you don't need to use dup() and stdin.
execl() keeps all open file descriptors from the parent process open so you could:
int fd[2];
pipe(fd);
if (fork() == 0)
{
char tmp[20];
close(fd[1]);
snprintf(tmp, sizeof(tmp), "%d", fd[0]);
execl("client", tmp, NULL);
exit(1);
}
and in the code for client:
int main(int argc, char** argv)
{
int fd = strtod(argv[1], NULL, 10);
/* Read from fd */
}
I'm implementing piping on a simulated file system in C++ (with mostly C). It needs to run commands in the host shell but perform the piping itself on the simulated file system.
I could achieve this with the pipe(), fork(), and system() system calls, but I'd prefer to use popen() (which handles creating a pipe, forking a process, and passing a command to the shell). This may not be possible because (I think) I need to be able to write from the parent process of the pipe, read on the child process end, write the output back from the child, and finally read that output from the parent. The man page for popen() on my system says a bidirectional pipe is possible, but my code needs to run on a system with an older version supporting only unidirectional pipes.
With the separate calls above, I can open/close pipes to achieve this. Is that possible with popen()?
For a trivial example, to run ls -l | grep .txt | grep cmds I need to:
Open a pipe and process to run ls -l on the host; read its output back
Pipe the output of ls -l back to my simulator
Open a pipe and process to run grep .txt on the host on the piped output of ls -l
Pipe the output of this back to the simulator (stuck here)
Open a pipe and process to run grep cmds on the host on the piped output of grep .txt
Pipe the output of this back to the simulator and print it
man popen
From Mac OS X:
The popen() function 'opens' a
process by creating a bidirectional
pipe, forking, and invoking the shell.
Any streams opened by previous popen()
calls in the parent process are closed
in the new child process.
Historically, popen() was implemented
with a unidirectional pipe; hence,
many implementations of popen() only
allow the mode argument to specify
reading or writing, not both. Because
popen() is now implemented using a
bidirectional pipe, the mode argument
may request a bidirectional data flow.
The mode argument is a pointer to a
null-terminated string which must be
'r' for reading, 'w' for writing, or
'r+' for reading and writing.
I'd suggest writing your own function to do the piping/forking/system-ing for you. You could have the function spawn a process and return read/write file descriptors, as in...
typedef void pfunc_t (int rfd, int wfd);
pid_t pcreate(int fds[2], pfunc_t pfunc) {
/* Spawn a process from pfunc, returning it's pid. The fds array passed will
* be filled with two descriptors: fds[0] will read from the child process,
* and fds[1] will write to it.
* Similarly, the child process will receive a reading/writing fd set (in
* that same order) as arguments.
*/
pid_t pid;
int pipes[4];
/* Warning: I'm not handling possible errors in pipe/fork */
pipe(&pipes[0]); /* Parent read/child write pipe */
pipe(&pipes[2]); /* Child read/parent write pipe */
if ((pid = fork()) > 0) {
/* Parent process */
fds[0] = pipes[0];
fds[1] = pipes[3];
close(pipes[1]);
close(pipes[2]);
return pid;
} else {
close(pipes[0]);
close(pipes[3]);
pfunc(pipes[2], pipes[1]);
exit(0);
}
return -1; /* ? */
}
You can add whatever functionality you need in there.
You seem to have answered your own question. If your code needs to work on an older system that doesn't support popen opening bidirectional pipes, then you won't be able to use popen (at least not the one that's supplied).
The real question would be about the exact capabilities of the older systems in question. In particular, does their pipe support creating bidirectional pipes? If they have a pipe that can create a bidirectional pipe, but popen that doesn't, then I'd write the main stream of the code to use popen with a bidirectional pipe, and supply an implementation of popen that can use a bidirectional pipe that gets compiled in an used where needed.
If you need to support systems old enough that pipe only supports unidirectional pipes, then you're pretty much stuck with using pipe, fork, dup2, etc., on your own. I'd probably still wrap this up in a function that works almost like a modern version of popen, but instead of returning one file handle, fills in a small structure with two file handles, one for the child's stdin, the other for the child's stdout.
POSIX stipulates that the popen() call is not designed to provide bi-directional communication:
The mode argument to popen() is a string that specifies I/O mode:
If mode is r, when the child process is started, its file descriptor STDOUT_FILENO shall be the writable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the readable end of the pipe.
If mode is w, when the child process is started its file descriptor STDIN_FILENO shall be the readable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the writable end of the pipe.
If mode is any other value, the result is unspecified.
Any portable code will make no assumptions beyond that. The BSD popen() is similar to what your question describes.
Additionally, pipes are different from sockets and each pipe file descriptor is uni-directional. You would have to create two pipes, one configured for each direction.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in popen simultaneous read and write)
Here's the code (C++, but can be easily converted to C):
#include <unistd.h>
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <utility>
// Like popen(), but returns two FILE*: child's stdin and stdout, respectively.
std::pair<FILE *, FILE *> popen2(const char *__command)
{
// pipes[0]: parent writes, child reads (child's stdin)
// pipes[1]: child writes, parent reads (child's stdout)
int pipes[2][2];
pipe(pipes[0]);
pipe(pipes[1]);
if (fork() > 0)
{
// parent
close(pipes[0][0]);
close(pipes[1][1]);
return {fdopen(pipes[0][1], "w"), fdopen(pipes[1][0], "r")};
}
else
{
// child
close(pipes[0][1]);
close(pipes[1][0]);
dup2(pipes[0][0], STDIN_FILENO);
dup2(pipes[1][1], STDOUT_FILENO);
execl("/bin/sh", "/bin/sh", "-c", __command, NULL);
exit(1);
}
}
Usage:
int main()
{
auto [p_stdin, p_stdout] = popen2("cat -n");
if (p_stdin == NULL || p_stdout == NULL)
{
printf("popen2() failed\n");
return 1;
}
const char msg[] = "Hello there!";
char buf[32];
printf("I say \"%s\"\n", msg);
fwrite(msg, 1, sizeof(msg), p_stdin);
fclose(p_stdin);
fread(buf, 1, sizeof(buf), p_stdout);
fclose(p_stdout);
printf("child says \"%s\"\n", buf);
return 0;
}
Possible Output:
I say "Hello there!"
child says " 1 Hello there!"
No need to create two pipes and waste a filedescriptor in each process. Just use a socket instead. https://stackoverflow.com/a/25177958/894520