I am trying to implement a simple shell. I fork processes this way:
void forkProcess(char* cmd[]) {
pid_t pid;
char programPath[BUFFERLENGTH] = "/bin/";
strcat(programPath, cmd[0]);
int exitStatus;
pid = fork();
switch (pid) {
case -1:
printf("Fork failed; pid == -1\n");
break;
case 0:
execv(programPath, cmd);
exit(0);
break;
default:
waitpid(pid, &exitStatus, 0);
//printf("Exitstatus = %d\n", WEXITSTATUS(exitStatus));
break;
}
}
Now the cmd parameter might contain a pipe, e.g.:
"ls" "-l" "|" "grep" "whatever" "(char*)NULL";
So how can I implement the pipe functionality? I know there are functions like pipe() and dup(), but I don't know how to use them in this context.
Thank you for any suggestions.
You must fully parse your command line before fork()ing to start the child.
If a pipe operator is being used, you must set up the pipe before calling fork(), so it is inherited.
In general you must also use close() and often dup() to make the pipe replace the forked process' stdin.
Continue reading up on these functions to "get" the big picture, or get a book which covers Unix I/O.
In this case, you may use popen().
Related
I'm trying to do a little C program that realize a pipeline of two bash commands : echo $arithmeticOperation | bc
$arithmeticOperation is a string taken as input.
The program works fine executing first command, but when i run the second one, i get the right output but the child process executing bc remains stuck preventing the child from ending.
So in this line father process is blocked :
waitpid(pid2,NULL,0);
Where do you think the problem may be ?
Sorry if i asked the question incorrectly, it's my first one. Thanks.
#define SYSCALL(r,c,e) if((r=c)==-1) { perror(e);exit(EXIT_FAILURE);}
int main(){
char buf[128];
int pfd[2],err;
pid_t pid1,pid2;
SYSCALL(err,pipe(pfd),"pipe");
switch (pid1=fork()) {
case -1: { perror("fork"); exit(EXIT_FAILURE);}
case 0 : {
scanf("%s",buf);
SYSCALL(err,dup2(pfd[1],1),"dup");
close(pfd[1]);
close(pfd[0]);
execl("/bin/echo","echo",buf,(char *)NULL);
return 1;
}
}
switch (pid2=fork() ){
case -1 : { perror("fork"); exit(EXIT_FAILURE);}
case 0 : {
SYSCALL(err,dup2(pfd[0],0),"dup");
close(pfd[1]);
close(pfd[0]);
// execl("/usr/bin/bc","bc",(char *)NULL);
execlp("bc","bc",(char *)NULL);
return 1;
}
}
printf("waiting . . . \n");
waitpid(pid1,NULL,0);
printf("wait\n");
waitpid(pid2,NULL,0);
close(pfd[1]);
close(pfd[0]);
return 0;
}
So if i digit "1+1" as a input string i get the right output but then the process executing bc never exit
As I noted in a comment, your parent process must close the file descriptors for the pipe before waiting for bc (and you've agreed that this fixes the problem).
This arises because bc has the pipe open for reading, and the parent has the pipe open for writing, and the kernel thinks that the parent could therefore send data to bc. It won't, but it could.
You have to be very careful when managing pipes. You carefully avoided the usual problem of not closing enough file descriptors in the children.
Rule of thumb: If you
dup2()
one end of a pipe to standard input or standard output, close both of the
original file descriptors returned by
pipe()
as soon as possible.
In particular, you should close them before using any of the
exec*()
family of functions.
The rule also applies if you duplicate the descriptors with either
dup()
or
fcntl()
with F_DUPFD
I need to extend that to cover parent processes too.
If the parent process is not going to communicate with any of its children via the pipe, it must ensure that it closes both ends of the pipe so that its children can receive EOF indications on read (or get SIGPIPE signals or write errors on write), rather than blocking indefinitely. The parent should normally close at least one end of the pipe — it would be extremely unusual for a program to read and write on both ends of a single pipe.
I have an app that spawns a child process. That child process outputs information about what it's doing by printing to stdout. The parent process does the same (i.e. prints to stdout).
In the child process I can write to stdout with some text prepended, but I have to add that to every single location I print across many source files.
I thought it might be smarter to have the parent process prepend output from the child process that it forks/exec's. I don't want to redirect the output because seeing the output inline with the parent process is beneficial. How do I do this? I'm using fork/exec in the parent.
Do I have to read the output and prepend each line manually or is there a simpler approach?
Update:
Thanks to Barmar. Here is how I'm doing it. I also could read byte by byte in the parent process from the pipe until line end. But I chose not to use that approach for reasons of complexity in my single threaded lua+C app.
// Crude example of output filtering using sed to
// prepend child process output text
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <libgen.h>
#include <string.h>
pid_t spawn(int fd[2], const char* path)
{
printf("Create child\n");
pid_t pid = fork();
switch(pid){
case -1:
printf("Create process failed");
return -1;
case 0:
dup2(fd[1], STDOUT_FILENO);
close(fd[0]);
close(fd[1]);
execl(path, path, NULL);
return 0;
default:
return pid;
}
}
pid_t spawnOutputFilter(int fd[2])
{
printf("Create sed\n");
pid_t pid = fork();
switch(pid){
case -1:
printf("Create sed failed");
return -1;
case 0:
dup2(fd[0], STDIN_FILENO);
close(fd[0]);
close(fd[1]);
execlp("sed", "sed", "s/^/Engine: /", (char *)NULL);
return -1;
default:
return pid;
}
}
int main(int argc, char* argv[])
{
if (argc > 1){
int options;
int fd[2];
pipe(fd);
pid_t pid = spawn(fd, argv[1]);
pid_t sed_pid = spawnOutputFilter(fd);
close(fd[0]);
close(fd[1]);
waitpid(pid, NULL, 0);
waitpid(sed_pid, NULL, 0);
}
return 0;
}
You could create a second child process that performs
execlp("sed", "sed", "s/^/PREFIX: /", (char *)NULL);
Connect the first child's stdout to this process's stdin with a pipe.
I thought it might be smarter to have the parent process prepend output from the child process.
I guess it depends on how you judge "smart". It might be simpler to just make the child prepend the desired text to its outputs.
I don't want to redirect the output because seeing the output inline with the parent process is beneficial. What's the best way to do this?
When two processes share an open file, both access it independently, regardless of the nature of the relationship between those processes. Thus, if your child inherits the parent's stdout, the parent has no mechanism even to notice that the child is sending output, much less to modify that output.
If you want the parent to handle this, you would need to pass the child's output through the parent. You could do that by creating a pipe, and associating the child's stdout with the write end of that pipe. The parent would then need to monitor the read end, and forward suitably-modified outputs to its own stdout. The parent would probably want to create a separate thread for that purpose.
Additionally, if the child sometimes produces multi-line outputs that you want prefixed as a group, rather than per-line, then you'd probably need to build and use some kind of protocol for demarcating message boundaries, which would make the whole parent-moderation idea pretty pointless.
Couldn't you define #define printf(a) printf("your text:" a).
Other alternative I can think of is using dup
You open the same log file in your child process and dup your stdout to new file descriptor.
My question is extension of this one: popen creates an extra sh process
Motives:
1) My program need to create a child which does tail on a file. I need to process the output line by line. That is why I am using popen because it returns FILE *. I can easily fetch single line, do what I need to do and print it.
One problem with popen is that you do not get pid of child (tail command in my case).
2) My program should not exit before its child is done. So I need to do wait; but without pid, I cannot do it.
How can I achieve both the goals?
A possible (kludge) solution: do execvp("tail -f file > tmpfile") and the keep reading that tmpfile. I am not sure how good this solution is, though.
Why aren't you using pipe/fork/exec method?
pid_t pid = 0;
int pipefd[2];
FILE* output;
char line[256];
int status;
pipe(pipefd); //create a pipe
pid = fork(); //span a child process
if (pid == 0)
{
// Child. Let's redirect its standard output to our pipe and replace process with tail
close(pipefd[0]);
dup2(pipefd[1], STDOUT_FILENO);
dup2(pipefd[1], STDERR_FILENO);
execl("/usr/bin/tail", "/usr/bin/tail", "-f", "path/to/your/file", (char*) NULL);
}
//Only parent gets here. Listen to what the tail says
close(pipefd[1]);
output = fdopen(pipefd[0], "r");
while(fgets(line, sizeof(line), output)) //listen to what tail writes to its standard output
{
//if you need to kill the tail application, just kill it:
if(something_goes_wrong)
kill(pid, SIGKILL);
}
//or wait for the child process to terminate
waitpid(pid, &status, 0);
You can use pipe, a function of the exec* family and fdopen. This is non-standard, but so is popen.
You don't need to wait. Just read the pipe up to EOF.
execvp("tail -f file > tmpfile") won't work, redirection is a feature of the shell and you're not running the shell here. Even if it worked it would be an awful solution. Suppose you have read to the end of the file, but the child process has not ended yet. What do you do?
You can use wait as it doesn't want a PID to wait for but simply waits for the any child process to exit. If you have created other child processes you can keep track of them, and if wait returns an unknown PID you can assume it's from your popen process.
I'm not sure why you need the process ID of the child. When the child exits, your pipe read will return an EOF. If you need to terminate the child, just close the pipe.
I'm implementing piping on a simulated file system in C++ (with mostly C). It needs to run commands in the host shell but perform the piping itself on the simulated file system.
I could achieve this with the pipe(), fork(), and system() system calls, but I'd prefer to use popen() (which handles creating a pipe, forking a process, and passing a command to the shell). This may not be possible because (I think) I need to be able to write from the parent process of the pipe, read on the child process end, write the output back from the child, and finally read that output from the parent. The man page for popen() on my system says a bidirectional pipe is possible, but my code needs to run on a system with an older version supporting only unidirectional pipes.
With the separate calls above, I can open/close pipes to achieve this. Is that possible with popen()?
For a trivial example, to run ls -l | grep .txt | grep cmds I need to:
Open a pipe and process to run ls -l on the host; read its output back
Pipe the output of ls -l back to my simulator
Open a pipe and process to run grep .txt on the host on the piped output of ls -l
Pipe the output of this back to the simulator (stuck here)
Open a pipe and process to run grep cmds on the host on the piped output of grep .txt
Pipe the output of this back to the simulator and print it
man popen
From Mac OS X:
The popen() function 'opens' a
process by creating a bidirectional
pipe, forking, and invoking the shell.
Any streams opened by previous popen()
calls in the parent process are closed
in the new child process.
Historically, popen() was implemented
with a unidirectional pipe; hence,
many implementations of popen() only
allow the mode argument to specify
reading or writing, not both. Because
popen() is now implemented using a
bidirectional pipe, the mode argument
may request a bidirectional data flow.
The mode argument is a pointer to a
null-terminated string which must be
'r' for reading, 'w' for writing, or
'r+' for reading and writing.
I'd suggest writing your own function to do the piping/forking/system-ing for you. You could have the function spawn a process and return read/write file descriptors, as in...
typedef void pfunc_t (int rfd, int wfd);
pid_t pcreate(int fds[2], pfunc_t pfunc) {
/* Spawn a process from pfunc, returning it's pid. The fds array passed will
* be filled with two descriptors: fds[0] will read from the child process,
* and fds[1] will write to it.
* Similarly, the child process will receive a reading/writing fd set (in
* that same order) as arguments.
*/
pid_t pid;
int pipes[4];
/* Warning: I'm not handling possible errors in pipe/fork */
pipe(&pipes[0]); /* Parent read/child write pipe */
pipe(&pipes[2]); /* Child read/parent write pipe */
if ((pid = fork()) > 0) {
/* Parent process */
fds[0] = pipes[0];
fds[1] = pipes[3];
close(pipes[1]);
close(pipes[2]);
return pid;
} else {
close(pipes[0]);
close(pipes[3]);
pfunc(pipes[2], pipes[1]);
exit(0);
}
return -1; /* ? */
}
You can add whatever functionality you need in there.
You seem to have answered your own question. If your code needs to work on an older system that doesn't support popen opening bidirectional pipes, then you won't be able to use popen (at least not the one that's supplied).
The real question would be about the exact capabilities of the older systems in question. In particular, does their pipe support creating bidirectional pipes? If they have a pipe that can create a bidirectional pipe, but popen that doesn't, then I'd write the main stream of the code to use popen with a bidirectional pipe, and supply an implementation of popen that can use a bidirectional pipe that gets compiled in an used where needed.
If you need to support systems old enough that pipe only supports unidirectional pipes, then you're pretty much stuck with using pipe, fork, dup2, etc., on your own. I'd probably still wrap this up in a function that works almost like a modern version of popen, but instead of returning one file handle, fills in a small structure with two file handles, one for the child's stdin, the other for the child's stdout.
POSIX stipulates that the popen() call is not designed to provide bi-directional communication:
The mode argument to popen() is a string that specifies I/O mode:
If mode is r, when the child process is started, its file descriptor STDOUT_FILENO shall be the writable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the readable end of the pipe.
If mode is w, when the child process is started its file descriptor STDIN_FILENO shall be the readable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the writable end of the pipe.
If mode is any other value, the result is unspecified.
Any portable code will make no assumptions beyond that. The BSD popen() is similar to what your question describes.
Additionally, pipes are different from sockets and each pipe file descriptor is uni-directional. You would have to create two pipes, one configured for each direction.
In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.
static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
int p1[2], p2[2];
if (!pid || !infd || !outfd)
return false;
if (pipe(p1) == -1)
goto err_pipe1;
if (pipe(p2) == -1)
goto err_pipe2;
if ((*pid = fork()) == -1)
goto err_fork;
if (*pid) {
/* Parent process. */
*infd = p1[1];
*outfd = p2[0];
close(p1[0]);
close(p2[1]);
return true;
} else {
/* Child process. */
dup2(p1[0], 0);
dup2(p2[1], 1);
close(p1[0]);
close(p1[1]);
close(p2[0]);
close(p2[1]);
execvp(*command, command);
/* Error occured. */
fprintf(stderr, "error running %s: %s", *command, strerror(errno));
abort();
}
err_fork:
close(p2[1]);
close(p2[0]);
err_pipe2:
close(p1[1]);
close(p1[0]);
err_pipe1:
return false;
}
https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46
(I used the same code in popen simultaneous read and write)
Here's the code (C++, but can be easily converted to C):
#include <unistd.h>
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <utility>
// Like popen(), but returns two FILE*: child's stdin and stdout, respectively.
std::pair<FILE *, FILE *> popen2(const char *__command)
{
// pipes[0]: parent writes, child reads (child's stdin)
// pipes[1]: child writes, parent reads (child's stdout)
int pipes[2][2];
pipe(pipes[0]);
pipe(pipes[1]);
if (fork() > 0)
{
// parent
close(pipes[0][0]);
close(pipes[1][1]);
return {fdopen(pipes[0][1], "w"), fdopen(pipes[1][0], "r")};
}
else
{
// child
close(pipes[0][1]);
close(pipes[1][0]);
dup2(pipes[0][0], STDIN_FILENO);
dup2(pipes[1][1], STDOUT_FILENO);
execl("/bin/sh", "/bin/sh", "-c", __command, NULL);
exit(1);
}
}
Usage:
int main()
{
auto [p_stdin, p_stdout] = popen2("cat -n");
if (p_stdin == NULL || p_stdout == NULL)
{
printf("popen2() failed\n");
return 1;
}
const char msg[] = "Hello there!";
char buf[32];
printf("I say \"%s\"\n", msg);
fwrite(msg, 1, sizeof(msg), p_stdin);
fclose(p_stdin);
fread(buf, 1, sizeof(buf), p_stdout);
fclose(p_stdout);
printf("child says \"%s\"\n", buf);
return 0;
}
Possible Output:
I say "Hello there!"
child says " 1 Hello there!"
No need to create two pipes and waste a filedescriptor in each process. Just use a socket instead. https://stackoverflow.com/a/25177958/894520
I already read the man page of the pidfile function family. But I don't really understand it. What is the correct usage? Is there a more elaborate example available? I think I understand pidfile_open. But when should I call pidfile_write and prdfile_close? From which process? Parent or child? What parameters do I have to pass to those functions? I propably lack some *nix fundamentals I guess.
Update:
Below you see the example from man pidfile. Why do they fork twice? Why pidfile_close? When I call pidfile_close I can start another daemon. Isn't that unwanted?
struct pidfh *pfh;
pid_t otherpid, childpid;
pfh = pidfile_open("/var/run/daemon.pid", 0600, &otherpid);
if (pfh == NULL) {
if (errno == EEXIST) {
errx(EXIT_FAILURE, "Daemon already running, pid: %jd.",
(intmax_t)otherpid);
}
/* If we cannot create pidfile from other reasons, only warn. */
warn("Cannot open or create pidfile");
}
if (daemon(0, 0) == -1) {
warn("Cannot daemonize");
pidfile_remove(pfh);
exit(EXIT_FAILURE);
}
pidfile_write(pfh);
for (;;) {
/* Do work. */
childpid = fork();
switch (childpid) {
case -1:
syslog(LOG_ERR, "Cannot fork(): %s.", strerror(errno));
break;
case 0:
pidfile_close(pfh);
/* Do child work. */
break;
default:
syslog(LOG_INFO, "Child %jd started.", (intmax_t)childpid);
break;
}
}
pidfile_remove(pfh);
exit(EXIT_SUCCESS);
The problem is that you want to give an error message before the daemon is spawned, and that you know the PID file after the daemon is spawned.
So you typically do the pidfile_open before the fork, which gives you a possibility to give an error message. After you forked, you know the pidfile and you can do pidfile_write.
You do the pidfile_open(3) before you go into background, so you can immediately report any problems. You don't write PID just yet, because your PID will change after daemon(3). pidfile_open(3) only locks the pidfile. After daemon(3) you can call pidfile_write(3) as you now have your final PID (daemon(3) forks internally). In the main process you cannot call pidfile_close(3), because this is the whole idea - by keeping the pidfile open and locked you let others know that you are still alive. The second fork is totally optional. It illustrates common behaviour that daemons spawn child/worker processes. If you don't use them you don't need this fork(). This fork() is there only to show that in such worker process you should close the pidfile, so it is only kept open and locked by the main process and not by the child.