I'm trying to create a C program that runs printenv | grep [parameters] | sort | more, and so far everything is working, and the program is printing the sorted environment variables to stdout as it's supposed to. The only problem is that there is no pager, even though the last process I'm starting - the only one with a file descriptor to stdout - is more. Here's the relevant code, I've tried editing out stuff that I know works.
#define PIPE_READ 0
#define PIPE_WRITE 1
int main(int argc, char *argv[], char *envp[])
{
int pipe_fd[3][2];
pipe(pipe_fd[0]);
pipe(pipe_fd[1]);
pipe(pipe_fd[2]);
pid[0]=fork();
if(pid[0]==0) /*grep-process*/
{
dup2(pipe_fd[0][PIPE_READ], STDIN_FILENO);
dup2(pipe_fd[1][PIPE_WRITE], STDOUT_FILENO);
/*close all 3 pipes, both ends, then execute grep*/
argv[0]= "grep";
execvp("grep",argv);
}
pid[1]=fork();
if(pid[1]==0) /*sort-process*/
{
dup2(pipe_fd[1][PIPE_READ], STDIN_FILENO);
dup2(pipe_fd[2][PIPE_WRITE], STDOUT_FILENO);
/*close all 3 pipes, both ends, then execute sort*/
execlp("sort","sort", (char*) 0);
}
pid[2]=fork();
if(pid[2]==0) /*more-process*/
{
dup2(pipe_fd[2][PIPE_READ], STDIN_FILENO);
/*close all 3 pipes, both ends, then execute more*/
execlp("more","more", (char*) 0);
}
/*parent-process*/
/*close all pipes except for the write end on the first, then write*/
for(i=0; envp[i]!=NULL; i++)
{
write(pipe_fd[0][PIPE_WRITE],envp[i], strlen(envp[i]));
write(pipe_fd[0][PIPE_WRITE],"\n",1);
}
close(pipe_fd[0][PIPE_WRITE]);
Of course I do check the return values of every call, and where I close all the pipes I simply wrote a comment to simplify reading it.
A call to ./a.out PATH results in this:
INFOPATH=/usr/share/info
MANPATH=/usr/local/vol/matlab/7.4.0/man:/usr/kerberos/man:/usr/local/share/man:/
usr/share/man/en:/usr/share/man:/usr/man
MANPATH=/usr/local/vol/matlab/7.4.0/man:/usr/kerberos/man:/usr/local/share/man:/
usr/share/man/en:/usr/share/man:/usr/man
MODULEPATH=/usr/local/vol/modulefiles:/pkg/modules/modulefiles:/etc/site/modulef
iles:/usr/share/Modules/modulefiles:/etc/modulefiles
MODULEPATH=/usr/local/vol/modulefiles:/pkg/modules/modulefiles:/etc/site/modulef
iles:/usr/share/Modules/modulefiles:/etc/modulefiles
...
...
Related
I'm trying to write a code which manipulates standard input and output and redirect them to files, and then use execvp (also tried other exec's) to run a program that simply uses printf and scanf , but the execvp fails..
Relevant code:
int pid2 = fork();
if (pid2 == 0) {
int fdInput = open("myinputfile", O_RDONLY);
close(0);
dup(fdInput);
int fdOutput = open("results.txt", O_WRONLY | O_CREAT | O_TRUNC);
close(1);
dup(fdOutput);
char* tmp[]={"...somepath/prog"};
execvp("...somepath/prog", tmp);
}
My prog:
int main(){
int x;
scanf("%d",&x);
printf("Hello World! %d",x);
return 0;
}
myinputfile contains only -> 4
I tried two main things:
copying the code from prog and hardcoding it into my code instead of the call to execvp, which works fine and I can see "Hello world! 4" in results.txt
running "mypath" in the terminal manually which also seems to work(with the standard I/O).
I can't understand why it's not working, I tried everything I could think of..
Your array of arguments passed to execvp() is not NULL-terminated.
Per the POSIX exec() documentation:
...
The argument argv is an array of character pointers to null-terminated strings. The application shall ensure that the last member of this array is a null pointer. These strings shall constitute the argument list available to the new process image. The value in argv[0] should point to a filename string that is associated with the process being started by one of the exec functions.
...
Your code should be
int pid2 = fork();
if (pid2 == 0) {
int fdInput = open("myinputfile", O_RDONLY);
close(0);
dup(fdInput);
int fdOutput = open("results.txt", O_WRONLY | O_CREAT | O_TRUNC);
close(1);
dup(fdOutput);
// note the NULL terminator
char* tmp[]={"...somepath/prog", NULL };
execvp("...somepath/prog", tmp);
}
I am trying to implement multiple pipes in C like
ls - al | less | wc
I have trouble with creating the pipeline. I have a loop that is supposed to create the processes and connect them with pipes:
for(i=0;i<num_cmds;i++){
create_commands(cmds[i]);
}
My create_commands() function looks like this
void create_commands (char cmd[MAX_CMD_LENGTH]) // Command be processed
{
int pipeid[2];
pipe(pipeid);
if (childpid = fork())
{
/* this is the parent process */
dup2(pipeid[1], 1); // dup2() the write end of the pipe to standard output.
close(pipeid[1]); // close() the write end of the pipe
//parse the command
parse_command(cmd, argvector);
// execute the command
execvp(argvector[0], argvector);
close(1); // close standard output
}
else
{
/* child process */
dup2( pipeid[0], 0); // the read end of the pipe to standard input
close( pipeid[0] ); // close() the read end of the pipe
}
}
But this doesn't work, I'm getting my stdin and stdout messed up.
Could anyone please point me to what I am doing wrong?
Thank you in advance!
The popen() function executes the command specified by the string command. It creates a pipe between the calling program and the executed command, and returns a pointer to a stream that can be used to either read from or write to the pipe.
#include <stdio.h>
int main(int argc, char *argv[])
{
FILE *fp;
int status;
int PATH_MAX = 1024;
char path[PATH_MAX];
fp = popen("ls -al | less | wc", "r");
if (fp == NULL)
/* Handle error */;
while (fgets(path, PATH_MAX, fp) != NULL)
printf("%s", path);
status = pclose(fp);
if (status == -1) {
/* Error reported by pclose() */
} else {
/* Use macros described under wait() to inspect `status' in order
to determine success/failure of command executed by popen() */
}
}
You can use a preset string to be called within popen(), you can also use your argv[] arguments to be piped in you'ld like.
popen() gives you a pipe, a FIFO First In First Out stream, and popen also feeds the STDOUT back to your program.
Here's the man page for popen():
http://linux.die.net/man/3/popen
I can't think of any way to implement pipelining in c that would actually work. That's why I've decided to write in here. I have to say, that I understand how do pipe/fork/mkfifo work. I've seen plenty examples of implementing 2-3 pipelines. It's easy. My problem starts, when I've got to implement shell, and pipelines count is unknown.
What I've got now:
eg.
ls -al | tr a-z A-Z | tr A-Z a-z | tr a-z A-Z
I transform such line into something like that:
array[0] = {"ls", "-al", NULL"}
array[1] = {"tr", "a-z", "A-Z", NULL"}
array[2] = {"tr", "A-Z", "a-z", NULL"}
array[3] = {"tr", "a-z", "A-Z", NULL"}
So I can use
execvp(array[0],array)
later on.
Untli now, I believe everything is OK. Problem starts, when I'm trying to redirect those functions input/output to eachother.
Here's how I'm doing that:
mkfifo("queue", 0777);
for (i = 0; i<= pipelines_count; i++) // eg. if there's 3 pipelines, there's 4 functions to execvp
{
int b = fork();
if (b == 0) // child
{
int c = fork();
if (c == 0)
// baby (younger than child)
// I use c process, to unblock desc_read and desc_writ for b process only
// nothing executes in here
{
if (i == 0) // 1st pipeline
{
int desc_read = open("queue", O_RDONLY);
// dup2 here, so after closing there's still something that can read from
// from desc_read
dup2(desc_read, 0);
close(desc_read);
}
if (i == pipelines_count) // last pipeline
{
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 0);
close(desc_write);
}
if (i > 0 && i < pipelines_count) // pipeline somewhere inside
{
int desc_read = open("queue", O_RDONLY);
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 1);
dup2(desc_read, 0);
close(desc_write);
close(desc_read);
}
exit(0); // closing every connection between process c and pipeline
}
else
// b process here
// in b process, i execvp commands
{
if (i == 0) // 1st pipeline (changing stdout only)
{
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 1); // changing stdout -> pdesc[1]
close(desc_write);
}
if (i == pipelines_count) // last pipeline (changing stdin only)
{
int desc_read = open("queue", O_RDONLY);
dup2(desc_read, 0); // changing stdin -> pdesc[0]
close(desc_read);
}
if (i > 0 && i < pipelines_count) // pipeline somewhere inside
{
int desc_write = open("queue", O_WRONLY);
dup2(desc_write, 1); // changing stdout -> pdesc[1]
int desc_read = open("queue", O_RDONLY);
dup2(desc_read, 0); // changing stdin -> pdesc[0]
close(desc_write);
close(desc_read);
}
wait(NULL); // it wait's until, process c is death
execvp(array[0],array);
}
}
else // parent (waits for 1 sub command to be finished)
{
wait(NULL);
}
}
Thanks.
Patryk, why are you using a fifo, and moreover the same fifo for each stage of the pipeline?
It seems to me that you need a pipe between each stage. So the flow would be something like:
Shell ls tr tr
----- ---- ---- ----
pipe(fds);
fork();
close(fds[0]); close(fds[1]);
dup2(fds[0],0);
pipe(fds);
fork();
close(fds[0]); close(fds[1]);
dup2(fds[1],1); dup2(fds[0],0);
exex(...); pipe(fds);
fork();
close(fds[0]); etc
dup2(fds[1],1);
exex(...);
The sequence that runs in each forked shell (close, dup2, pipe etc) would seem like a function (taking the name and parameters of the desired process). Note that up until the exec call in each, a forked copy of the shell is running.
Edit:
Patryk:
Also, is my thinking correct? Shall it work like that? (pseudocode):
start_fork(ls) -> end_fork(ls) -> start_fork(tr) -> end_fork(tr) ->
start_fork(tr) -> end_fork(tr)
I'm not sure what you mean by start_fork and end_fork. Are you implying that ls runs to completion before tr starts? This isn't really what is meant by the diagram above. Your shell will not wait for ls to complete before starting tr. It starts all of the processes in the pipe in sequence, setting up stdin and stdout for each one so that the processes are linked together, stdout of ls to stdin of tr; stdout of tr to stdin of the next tr. That is what the dup2 calls are doing.
The order in which the processes run is determined by the operating system (the scheduler), but clearly if tr runs and reads from an empty stdin it has to wait (to block) until the preceding process writes something to the pipe. It is quite possible that ls might run to completion before tr even reads from its stdin, but it is equally possible that it wont. For example if the first command in the chain was something that ran continually and produced output along the way, the second in the pipeline will get scheduled from time to time to prcess whatever the first sends along the pipe.
Hope that clarifies things a little :-)
It might be worth using libpipeline. It takes care of all the effort on your part and you can even include functions in your pipeline.
The problem is you're trying to do everything at once. Break it into smaller steps instead.
1) Parse your input to get ls -al | out of it.
1a) From this you know you need to create a pipe, move it to stdout, and start ls -al. Then move the pipe to stdin. There's more coming of course, but you don't worry about it in code yet.
2) Parse the next segment to get tr a-z A-Z |. Go back to step 1a as long as your next-to-spawn command's output is being piped somewhere.
Implementing pipelining in C. What would be the best way to do that?
This question is a bit old, but here's an answer that was never provided. Use libpipeline. libpipeline is a pipeline manipulation library. The use case is one of the man page maintainers who had to frequently use a command like the following (and work around associated OS bugs):
zsoelim < input-file | tbl | nroff -mandoc -Tutf8
Here's the libpipeline way:
pipeline *p;
int status;
p = pipeline_new ();
pipeline_want_infile (p, "input-file");
pipeline_command_args (p, "zsoelim", NULL);
pipeline_command_args (p, "tbl", NULL);
pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
status = pipeline_run (p);
The libpipeline homepage has more examples. The library is also included in many distros, including Arch, Debian, Fedora, Linux from Scratch and Ubuntu.
I'm just starting to learn C programming and I have some uncertainty about fork(), exec(), pipe(), etc.
I've developed this code, but when I execute it, the variable c remains empty, so I don't know if the child isn't writing to the pipe, or the parent isn't reading from it.
Could you help me please? This is the code:
int main() {
int pid=0;
int pipefd[2];
char* c=(char *)malloc(sizeof(char));
FILE *fp;
pipe(pipefd);
pid=fork();
if (pid==0){
close(pipefd[0]);
dup2(pipefd[1],1);
close(pipefd[1]);
execl("ls -l | cut -c28","ls -l | cut -c28", (char *) 0);
}
else{
close(pipefd[1]);
read(pipefd[0], c, 1);
char* path="/home/random";
char* txt=".txt";
char* root=malloc(strlen(path) + strlen(txt) + sizeof(char));
strcpy(root,path);
strcat(root,c);
strcat(root,txt);
close(pipefd[0]);
fp=fopen(root,"w+");
(...)
}
The problem is that the final root string its only "/home/random.txt" because there is nothing in the char c, and what I want is to open the file "/home/random(number stored in char c).txt".
execl executes a single command, and is not aware of shell concepts such as pipes. If you want to execute a shell command, you will have to execute a shell, as follows:
execl("/bin/sh","/bin/sh","-c","ls -l | cut -c28", (char*) 0);
Always check the return value of the system calls (like execve(2) and derived functions like execl(3)), and use the errno(3) to figure out what went wrong.
In your case the execl line fails.
Using strcpy/strcat seems a bit excessively complex. snprintf can turn those 3 lines into one.
snprintf( root, size_of_buf, "/home/random%s", c );
Additionally, check your error codes. As noted, execl is failing and you don't know it. fork, dup2, ...,can also fail, you want to know sooner rather than later.
Basically I want to do in C (and without buffering) the same as this bash-script:
#!/bin/sh
cat ./fifo_in | myprogram > ./fifo_out
In other words I want to exec "myprogram" and redirect its stdin and stdout to two pipes which have been created previously.
Another program is feeding data into fifo_in and reading out of fifo_out.
Of course it would be easy to just read from ./fifo_in, buffer it in the parent and write to myprogram's stdin (and reverse for stdout and ./fifo_out) but I think there is probably a way to let "myprogram" read/write directly from/to the fifos without buffering in the parent process.
Edit:
Eugen's answer seems to be the correct one, but I cannot get it to work.
I use this function on the C-side, which seems correct to me:
pid_t execpipes(const char *wd, const char *command, const char *pipename)
{
char pipename_in[FALK_NAMESIZE];
char pipename_out[FALK_NAMESIZE];
strcpy(pipename_in, FALKPATH);
strcat(pipename_in, "/");
strcat(pipename_in, FALK_FIFO_PATH);
strcat(pipename_in, "/");
strncat(pipename_in, pipename, FALK_NAMESIZE-2);
strcpy(pipename_out, pipename_in);
strcat(pipename_out, "R");
pid_t pid;
pid = fork();
if (pid < 0)
{ //Error occured
perror("fork");
exit(1);
}
if (pid == 0)
{
chdir(wd);
d("execpipes: pipename_in=\"%s\"\n", pipename_in);
d(" pipename_out=\"%s\"\n", pipename_out);
freopen(pipename_in,"r",stdin);
freopen(pipename_out,"w",stdout);
d("execpipes: command=\"%s\"\n", command);
execl("/bin/sh", "sh", "-c", command, (char *)NULL); // using execv is probably faster
// Should never get here
perror("execl");
exit(1);
}
return pid;
}
I read and write the pipes from a PHP-script (only relevant part posted):
$pipe_in = fopen($fp.$pipename, "w");
$DEBUG .= "Write to pipe_in\n";
$ret = fwrite($pipe_in, $in);
$pipe_out = fopen($fp.$pipename.'R', "r");
$DEBUG .= "Read from pipe_out\n";
$atext = fread($pipe_out, 200000); // Program hangs here
The program is started correctly, receives the input via $pipe_in correctly, processes the data correctly and (because it ran fine for many months) I assume it puts out the data correctly to stdout, but when I try to read from $pipe_out, it hangs. I know that the pipes themselves are set up correctly because if I don't open $pipe_out, the program does not get any input - which makes sense because there is no reader for $pipe_out and therefore the pipeline is not complete. So I can open $pipe_out, but I cannot read anything from it, which is quite strange.
Edit2:
Program works now, thanks guys - For some reason the first pipe has to be closed before you can read from the second pipe:
$pipe_in = fopen($fp.$pipename, "w");
$pipe_out = fopen($fp.$pipename.'R', "r");
$DEBUG .= "Write to pipe_in\n";
$ret = fwrite($pipe_in, $in);
fclose($pipe_in);
$DEBUG .= "Read from pipe_out\n";
$atext = fread($pipe_out, 200000);
fclose($pipe_out);
unlink($fp.$pipename);
unlink($fp.$pipename.'R');
I'd write a small wrapper for myprogram, that does
freopen("./fifo_in","r",stdin)
freopen("./fifo_out","w",stdout)
(Ofcourse not with constant paths!), then execve myprogram
Korn shell supports coprocesses, which I think effectively does what you ask: read from a pipe and write to a pipe (which can be stdout and stdin of a C process)
http://www.dartmouth.edu/~rc/classes/ksh/coprocesses.html
How about
myprogram < ./fifo_in > ./fifo_out
?
As for getting rid of the buffering: Since your program directly reads/writes the pipes, the buffering shouldn't hurt you.
An important point is that the process which writes fifo_in should flush properly so you don't have to wait. The same goes for your output: As soon as a "work unit" is complete, flush your stdout which will make the data available to whoever reads the output pipe.
But you can't do anything in myprogram to make the writer of fifo_in flush its buffers.
[EDIT] To do this from C (without the help of a shell), use code like this:
- Put the names of the two pipes into local variables on the stack
- Call `fork()`. If that returns '0', then open the two fifos with `freopen()` [like Eugen suggested][1]
- Call `execve` to launch the real exec.
That's (in a nutshell) what the shell is doing when it runs commands. Make sure the parent process (the one where fork() returns a PID != 0) handles the signal SIGCHLD
Perhaps you are looking of a named pipe? For example:
mkfifo fifo_in
As a test stub for my_program.c, to read fifo_in via the buffered stdin:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(void) {
char buf[80];
if (!freopen("./fifo_in", "r", stdin)) {
perror("freopen");
exit(EXIT_FAILURE);
}
while (!ferror(stdin)) {
while (fgets(buf, sizeof buf, stdin))
fputs(buf, stdout);
sleep(1);
}
return 0;
}
Then as a test for the writer, using the bash shell:
for x in {1..10}; do
echo $x
echo $x >> fifo_in
sleep 1
done
Notes:
I'd prefer to use unbuffered I/O.
The writer, at least on my machine, blocks until there is a reader.
The reader, in this sample, cannot tell when the writer is finished.