I'm writing a little Linux fifo test program.
I created a pipe using mkfifo mpipe. The program should perform one write per argument sent to it. If no arguments are sent, then it performs one read from the pipe.
Here is my code
int main(int argc, char *argv[])
{
if (argc > 1)
{
int fd = open("./mpipe", O_WRONLY);
int i = 1;
for (i; i < argc; i++)
{
int bytes = write(fd, argv[i], strlen(argv[i]) + 1);
if (bytes <= 0)
{
printf("ERROR: write\n");
close(fd);
return 1;
}
printf("Wrote %d bytes: %s\n", bytes, argv[i]);
}
close(fd);
return 0;
}
/* Else perform one read */
char buf[64];
int bytes = 0;
int fd = open("./mpipe", O_RDONLY);
bytes = read(fd, buf, 64);
if (bytes <= 0)
{
printf("ERROR: read\n");
close(fd);
return 1;
}
else
{
printf("Read %d bytes: %s\n", bytes, buf);
}
close(fd);
return 0;
}
I would expect the behavior to be something like this...
I call ./pt hello i am the deepest guy, and I would expect it to block for 6 reads. Instead one read seems enough to trigger multiple writes. My output looks like
# Term 1 - writer
$: ./pt hello i am the deepest guy # this call blocks until a read, but then cascades.
Just wrote hello
Just wrote i
Just wrote am
Just wrote the # output ends here
# Term 2 - reader
$: ./pt
Read 6 bytes: hello
Could someone help explain this weird behavior? I thought that every read would have to match with a write with regards to pipe communication.
What is happening there is that the kernel blocks the writer process in the open(2) system call until you have a reader opening it for reading. (A fifo requires both ends connected to processes to work) Once the reader does the first read(2) call (either the writer or the reader blocks, who gets first for the system call) The kernel passes all the data from the writer to the reader, and awakens both processes (that's the reason of receiving only the first command line parameter, and not the first 16 bytes from the writer, you get only the six characters {'h', 'e', 'l', 'l', 'o', '\0' } from the blocking writer)
Finally, as the reader just closes the fifo, the writer gets killed with a SIGPIPE signal, as no more readers have the fifo open. If you install a signal handler on the writer process (or ignore the signal) you'll get an error from the write(2) syscall, telling you that no more readers were blocked on the fifo (EPIPE errno value) on the blocking write.
Just notice that this is a feature, and not a bug, a means of knowing that writes will not reach any reader until you close and reopen the fifo.
The kernel blocks the inode of the fifo for the whole read(2) or write(2) calls, so even another process doing another write(2) on the fifo will be blocked, and you'll not get the data on the reader from that second writer (should you have one). You can try if you like, to start two writers and see what happens.
$ pru I am the best &
[1] 271
$ pru
Read 2 bytes: I
Wrote 2 bytes: I
Wrote 3 bytes: am
[1]+ Broken pipe pru I am the best <<< this is the kill to the writer process, announced by the shell.
$ _
Related
I'm writing a program with two processes that communicate through a pipe. The child process reads some parameters from the parent, executes a shell script with them and returns the results to the parent process line by line.
My code worked just fine until I wrote the while(read()) part at the end of the parent process. The child would execute the shell script, read its echo's from popen() and print them to standard output.
Now I tried to write the results to the pipe as well and read them in the while() loop at the parent's end but it blocks and neither will the child process print the result to standard output. Apparently it won't even reach the point after reading the data from the pipe sent by the parent.
If I comment out the while() at the parent process, the child will print the results and return, and the program will end smoothly.
Why does the while(read()) block even if I closed the writing end of the pipe in both parent and child processes?
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
#include <errno.h>
#include <string.h>
#include <fcntl.h>
int read_from_file(char **directory, int *octal) {
FILE *file = fopen("input", "r");
if (file == NULL) {
perror("error opening file");
exit(1);
}
fscanf(file, "%s %d", *directory, octal);
}
int main(int argc, char *argv[]) {
char *directory = malloc(256);
int *octal = malloc(sizeof *octal);
pid_t pid;
int pfd[2];
char res[256];
if (pipe(pfd) < 0) {
perror("Error opening pipe");
return 1;
}
if ((pid = fork()) < 0)
perror("Error forking");
if (pid == 0) {
printf("client here\n");
if (read(pfd[0], directory, 256) < 0)
perror("error reading from pipe");
if (read(pfd[0], octal, sizeof(int)) < 0)
perror("error reading from pipe");
// This won't get printed:
printf("client just read from pipe\n");
// close(pfd[0]);
char command[256] = "./asd.sh ";
strcat(command, directory);
char octal_c[5];
sprintf(octal_c, " %d", *octal);
strcat(command, octal_c);
FILE *f = popen(command, "r");
while (fgets(res, 256, f) != NULL) {
printf("%s", res);
if (write(pfd[1], res, 256) < 0)
perror("Error writing res to pipe");
}
fclose(f);
close(pfd[1]);
close(pfd[0]);
fflush(stdout);
return 1;
}
read_from_file(&directory, octal);
if (write(pfd[1], directory, 256) < 0)
perror("Error writing dir to pipe");
if (write(pfd[1], octal, sizeof(int)) < 0)
perror("error writing octal to pipe");
int r;
close(pfd[1]);
while (r = read(pfd[0], res, 256)) {
if (r > 0) {
printf("%s", res);
}
}
close(pfd[0]);
while (wait(NULL) != -1 || errno != ECHILD);
}
Since the child demonstrably reaches ...
printf("client here\n");
... but seems not to reach ...
printf("client just read from pipe\n");
... we can suppose that it blocks indefinitely on one of the two read() calls between. With the right timing, that explains why the parent blocks on its own read() from the pipe. But how and why does that blocking occur?
There are at least three significant semantic errors in your program:
pipes do not work well for bidirectional communication. It is possible, for example, for a process to read back the bytes that it wrote itself and intended for a different process. If you want bidirectional communication then use two pipes. In your case, I think that would have avoided the apparent deadlock, though it would not, by itself, have made the program work correctly.
write and read do not necessarily transfer the full number of bytes requested, and short reads and writes are not considered erroneous. On success, these functions return the number of bytes transferred, and if you want to be sure to transfer a specific number of bytes then you need to run the read or write in a loop, using the return values to track progress through the buffer being transferred. Or use fread() and fwrite() instead.
Pipes convey undifferentiated streams of bytes. That is, they are not message oriented. It is not safe to assume that reads from a pipe will be paired with writes to the pipe, so that each read receives exactly the bytes written by one write. Yet your code depends on that to happen.
Here's a plausible failure scenario that could explain your observations:
The parent:
fork()s the child.
after some time performs two writes to the pipe, one from variable directory and the other from variable octal. At least the first of those is a short write.
closes its copy of the write end of the pipe.
blocks attempting to read from the pipe.
The child:
reads all the bytes written via its first read (into its copy of directory).
blocks on its second read(). It can do this despite the parent closing its copy of the write end, because the write end of the pipe is still open in the child.
You then have a deadlock. Both ends of the pipe are open in at least one process, the pipe is empty, and both processes are blocked trying to read bytes that can never arrive.
There are other possibilities that arrive at substantially the same place, too, some of them not relying on a short write.
The parent process was trying to read from the pipe before the child could have read from it and write the results to it. Using two different pipes for the two-way communication solved the problem.
There are many questions related to reading and writing of pipes in this forum, but i am unable to resolve my issue.
The code snippet below, does the following things:
Through command line argument filename is passed to the child process through pipe_p
Child process opens the file specified, and writes its content to pipe_c for parent process to read and display on the screen.
Everything is working fine, but parent process is unable to read data from the pipe (since it is not printing anything).
I observed that data is successfully written by child process since i am able to print the contents through pipe in child process block but not in parent process.
NOTE : STEP 4 is not working
Anyone please help me this.
Code:
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char **argv){
int pipe_p[2], pipe_c[2];
int childpid, c, k = 0;
char buffer[1000] = {0};
FILE *file;
pipe(pipe_p);
pipe(pipe_c);
childpid = fork();
if(childpid){
//parent process block
//STEP 1 -------
close(pipe_p[0]); //closing reading side of pipe
write(pipe_p[1], argv[1], strlen(argv[1]));
close(pipe_p[1]);
//--------------
wait(NULL);
//--------------
//printf("%s\n", "Its working");
//STEP 4 -------
close(pipe_c[1]);
read(pipe_c[0], buffer, sizeof(buffer));
close(pipe_c[0]);
printf("%s\n", buffer);
//--------------
}
else{
//child process block
//sleep(1);
//STEP 2 -------
close(pipe_p[1]);
read(pipe_p[0], buffer, sizeof(buffer));
close(pipe_p[0]);
//printf("%s\n", buffer);
//--------------
//STEP 3 -------
file = fopen(buffer, "r");
while((c = getc(file)) != EOF){
buffer[k++] = c;
}
buffer[k] = 0;
//printf("%s", buffer);
close(pipe_c[0]);
write(pipe_c[1], buffer, strlen(buffer));
close(pipe_c[1]);
//--------------
}
return 0;
}
I see five bugs in this code. I'm going to list them from least to most important. I haven't tried to fix any of the bugs, so there may be more that are hidden behind these.
You forgot to include sys/wait.h. The compiler should have complained about an implicit declaration of wait. (If your compiler did not make any complaints, turn on all the warnings.)
You are not checking whether any of your system calls are failing. Every system call should be followed by a check for failure. When one does fail, print to stderr a full description of the failure, including the name of the system call that failed, the names of all files involved (if any), and strerror(errno) and then exit the program with a nonzero (unsuccessful) exit code. If you had done this you would have discovered for yourself that, in fact, certain things were not "working fine".
Relatedly, you are not checking whether the child exited unsuccessfully. Instead of wait(NULL), the parent should be doing waitpid(childpid, &status, 0) and then decoding the exit status and printing a message to stderr for anything other than WIFEXITED(status) && WEXITSTATUS(status) == 0 and then exiting unsuccessfully itself.
In the parent, you are calling wait in the wrong place. You need to call wait AFTER you have read and processed all of the data from pipe_c. Otherwise, if the child process completely fills up the pipe buffer, the program will deadlock. (Also, you need to read all of the data from the pipe, not just the first 1000 bytes of it.)
In the child, you have a buffer overrun. You are reading an unlimited amount of data from the file into buffer, but buffer has a fixed size. You should either use malloc and realloc to enlarge it as necessary, or copy from the file to the pipe in chunks no bigger than the size of buffer.
I discovered all of these problems by running the program under the strace utility, in -f mode (so it traces both sides of the fork), with input from a large file. This is a valuable debugging technique which you should try for yourself.
In the below program (which is not mine, but one which I have modified), the child process makes two writes to the pipe. When I run the program, I get the following output:
Received string: Hello, world!
This is the child process.
How is it that the read performed by the parent process captures both of these strings from the pipe buffer? What (if anything) prevents the parent process from assuming after it has read the first string (or the first char of the first string for that matter) that there is nothing else to read from the buffer, and exiting?
Program in question:
int main(void)
{
int fd[2], nbytes;
pid_t childpid;
char string[] = "Hello, world!\n";
char string2[] = "This is the child process.\n";
char readbuffer[80];
pipe(fd);
if((childpid = fork()) == -1)
{
perror("fork");
return 1;
}
if(childpid == 0)
{
/* Child process closes pipe's input file descriptor */
close(fd[0]);
/* Send "string" through the output side of pipe */
write(fd[1], string, (strlen(string)));
write(fd[1], string2, (strlen(string2)+1));
return 1;
}
else
{
/* Parent process closes pipe's output file descriptor */
close(fd[1]);
/* Read in a string from the pipe */
nbytes = read(fd[0], readbuffer, sizeof(readbuffer));
printf("Received string: %s", readbuffer);
}
return 0;
}
The key here is the read() buffer size 80.
Usually read() block the process which is calling it (is set in a sleeping status) until certain conditions doesn't happen, for example:
the requested buffer is full
there's less data than requested (EOF) or an error occurred
a signal wake the process (as ^C), this may be your case, the child process exit and the system send a broken pipe signal to the parent (the process wake and read() get the whole buffer)
Note that those conditions depends on the subsystem you are reading from, in your case a pipe. Subsystem which may have different properties, like buffer size. An absurd example: if pipe buffer size on kernel side was less than or equal to your first write, the reading process would have wake up earlier returning a truncated buffer.
I love this question. It makes me remember old time ...
Meta-answer: It does only know, if the stream is closed
Answer: your process reads all it can/has to read, no end stops at '\n'
But I suppose, you want to write/read records.
Despite others I try to answer without distinguishing types of streams. I make a disclaimer here:
it all depends on OS and stream type - and the options opening on both (!) sides
Basic:
Lets say one process writes an other reads - easy enough.
No is not.
Imagine - always ;-) - they put there chars (bytes) into the stream as they want and can - slowly - fast - one by one or all they buffered.
and you read it - no - not you and not your program - in between - byte by byte or on block or how the layer in between likes.
So if you want a record, there a only three possibilities:
make an end char(byte) - block seperator,
make the records fix length,
the old two-byte-length then read the rest
Answer again:
in your case the child puts all chars in the stream the parent read them and over - why should one of them make a break?
Solution:
Depending on the OS and language and libs, you sometimes can tell the OS to make an / accept an end char (like \n) - sometimes your write read functions do that for you
The parent will read a maximum of 80 bytes. So the first write system call (and also the second) will place the first string in the kernel buffer relative to the pipe. The read system call will read a maximum of 80 bytes, so it will ask to the kernel for the data, and the kernel will return the first 80 bytes of data in the buffer (so it is impossible to read only the first character of the string with a single write of the entire string and a single blocking read of 80 bytes).
As you stated there is no synchronization between the child and the parent.
In fact on my linux system your program sometimes print only the first string, sometimes its print both.
The problem is a classic Race Condition problem, sometimes the child will write only the first string, then the parent is scheduled and it will read the first string, then the program will end (or the control will return to the child after the read, the child will write the second string to the pipe buffer but no one will actually read the data).
You should place some synchronization functions in your program, an example can be to place a wait for the child after the close(fd[1]);
[...]
else
{
/* Parent process closes up output side of pipe */
close(fd[1]);
/* Synchronize parent and child */
wait(NULL);
/* Read in a string from the pipe */
nbytes = read(fd[0], readbuffer, sizeof(readbuffer));
printf("Received string: %s", readbuffer);
}
[...]
Then the parent will read the data only when the two strings are correctly placed in the buffer.
this is an example of how to write to a pipe.
This example is for parent writing to child, but the logic is very similar, regardless of the data flow direction.
close(pipefd[0]); /* Close unused read end */
write(pipefd[1], argv[1], strlen(argv[1]));
close(pipefd[1]); /* Reader will see EOF */
wait(NULL); /* Wait for child to exit */
exit(EXIT_SUCCESS);
Amongst other things, this means the caller to read() will get a returned value of 0 when everything is read from the pipe.
this also means the reader to read in a loop until a error condition indication is returned or a 0 is returned.
... how does the read function “know” when there is nothing left to read?
On success read() returns a much as available in the underlying object but not more then requested.
From man 3 read (POSIX):
Upon successful completion, where nbyte is greater than 0, read() shall mark for update the st_atime field of the file, and shall return the number of bytes read. This number shall never be greater
than nbyte. The value returned may be less than nbyte if the number of bytes left in the file is less than nbyte, if the read() request was interrupted by a signal, or if the file is a pipe or FIFO
or special file and has fewer than nbyte bytes immediately available for reading.
I am trying to create a chain of pipes for 3 processes plus the current program such that:
1 - Exec P1 and send output of P1 to input of P2
3 - Send output of P2 to input of P3
4 - Display output of P3 in stdout
5 - Pass stdin of current/main/driver program to input of P3
I am using 2 pipes. For task #5, the main program reads from stdin and writes to a pipe that P3 reads.
I have managed to get the processes communicating with each other. However, what I noticed is there is a large delay between when P1 writes and when P2 detects this write on its STDIN, i.e. P1 may have written hundreds of times before P2 detects the write and misses many of the P1 writes at the beginning. I have confirmed via print messages that P2 is in fact launched on time, however, it does not detect/read the input in time (it is a Python script looping: "for line in sys.stdin:" )
Here is my code:
int pipe1[2];
int pipe2[2];
if (pipe(pipe1) < 0 || pipe(pipe2) < 0 )
{
perror("Error: pipe");
}
pid_t procIDC = fork();
if (procIDC == 0)
{
dup2(pipe2[0], 0);
execv("procC", argv);
}
else
{
pid_t procIDB ;
procIDB = fork();
if (procIDB == 0)
{
dup2( pipe1[0], 0);
dup2( pipe2[1], 1);
if (execl("/usr/bin/python", "/usr/bin/python", "./test.py", (char *)NULL) < 0)
{
perror("execl"); return 0;
}
}
else
{
pid_t procIDA = fork();
if (procIDA ==0)
{
dup2( pipe1[1], 1);
execv("proc1", argv);
}
else
{
dup2( pipe2[1], 1);
//print any input so it sends to p3
ssize_t read;
char *inputLine = NULL;
size_t len = 0;
while ((read = getline(&inputLine, &len, stdin)) != -1)
{
printf(inputLine);
}
}
}
I noticed is there is a large delay between when P1 writes and when P2 detects this write on its STDIN
Yes. Your question is presumably "why is there a delay"?
The answer: stdio (used by most programs) uses fully-buffered output when it detects that the output is going into a pipe.
To prevent that buffering, use fflush or setvbuf.
Some additional reading here.
Actually, the fflush is obviously being done in the writing program
You didn't show us the writing program with fflush in it. If you had, we might be able to point out your mistake (fflush will help, if done correctly).
Anyway, one way to see what is happening is to run strace -p <pid-of-writer>, and observe that immediately after the writer actually executes the write(2) syscall, the reader gets its input. That would prove that it is in fact buffering in the writer that is causing the delay.
This:
setbuf(stdout, NULL);
does not disable buffering. You really need to call setvbuf (or fflush).
There was input buffering on STDIN in P2
I am trying to implement a client-server type of communication system using the poll function in C. The flow is as follows:
Main program forks a sub-process
Child process calls the exec function to execute some_binary
Parent and child send messages to each other alternately, each message that is sent depends on the last message received.
I tried to implement this using poll, but ran into problems because the child process buffers its output, causing my poll calls to timeout. Here's my code:
int main() {
char *buffer = (char *) malloc(1000);
int n;
pid_t pid; /* pid of child process */
int rpipe[2]; /* pipe used to read from child process */
int wpipe[2]; /* pipe used to write to child process */
pipe(rpipe);
pipe(wpipe);
pid = fork();
if (pid == (pid_t) 0)
{
/* child */
dup2(wpipe[0], STDIN_FILENO);
dup2(rpipe[1], STDOUT_FILENO);
close(wpipe[0]); close(rpipe[0]);
close(wpipe[1]); close(rpipe[1]);
if (execl("./server", "./server", (char *) NULL) == -1)
{
fprintf(stderr, "exec failed\n");
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}
else
{
/* parent */
/* close the other ends */
close(wpipe[0]);
close(rpipe[1]);
/*
poll to check if write is good to go
This poll succeeds, write goes through
*/
struct pollfd pfds[1];
pfds[0].fd = wpipe[1];
pfds[0].events = POLLIN | POLLOUT;
int pres = poll(pfds, (nfds_t) 1, 1000);
if (pres > 0)
{
if (pfds[0].revents & POLLOUT)
{
printf("Writing data...\n");
write(wpipe[1], "hello\n", 6);
}
}
/*
poll to check if there's something to read.
This poll times out because the child buffers its stdout stream.
*/
pfds[0].fd = rpipe[0];
pfds[0].events = POLLIN | POLLOUT;
pres = poll(pfds, (nfds_t) 1, 1000);
if (pres > 0)
{
if (pfds[0].revents & POLLIN)
{
printf("Reading data...\n");
int n = read(rpipe[0], buffer, 1000);
buffer[n] = '\0';
printf("child says:\n%s\n", buffer);
}
}
kill(pid, SIGTERM);
return EXIT_SUCCESS;
}
}
The server code is simply:
int main() {
char *buffer = (char *) malloc(1000);
while (scanf("%s", buffer) != EOF)
{
printf("I received %s\n", buffer);
}
return 0;
}
How do I prevent poll calls from timing out because of buffering?
EDIT:
I would like the program to work even when the execed binary is external, i.e., I have no control over the code - like a unix command, e.g., cat or ls.
There seem to be two problems in your code. "stdout" is by default buffered,
so the server should flush it explicitly:
printf("I received %s\n", buffer);
fflush(stdout);
And the main program should not register for POLLOUT when trying to read
(but you may want register for POLLERR):
pfds[0].fd = rpipe[0];
pfds[0].events = POLLIN | POLLERR;
With these modifications you get the expected output:
$ ./main
Writing data...
Reading data...
child says:
I received hello
Generally, you should also check the return value of poll(), and repeat the call if
necessary (e.g. in the case of an interrupted system call or timeout).
You need, as I answered in a related answer to a previous question by you, to implement an event loop; as it name implies, it is looping, so you should code in the parent process:
while (1) { // simplistic event loop!
int status=0;
if (waitpid(pid, &status, WNOHANG) == pid)
{ // clean up, child process has ended
handle_process_end(status);
break;
};
struct pollpfd pfd[2];
memset (&pfd, 0, sizeof(pfd)); // probably useless but dont harm
pfd[0].fd = rpipe[0];
pfd[0].events = POLL_IN;
pfd[1].fd = wpipe[1];
pfd[0].event = POLL_OUT;
#define DELAY 5000 /* 5 seconds */
if (poll(pfd, 2, DELAY)>0) {
if (pfd[0].revents & POLL_IN) {
/* read something from rpipe[0]; detect end of file;
you probably need to do some buffering, because you may
e.g. read some partial line chunk written by the child, and
you could only handle full lines. */
};
if (pfd[1].revents & POLL_OUT) {
/* write something on wpipe[1] */
};
}
fflush(NULL);
} /* end while(1) */
you cannot predict in which order the pipes are readable or writable, and this can happen many times. Of course, a lot of buffering (in the parent process) is involved, I leave the details to you.... You have no influence on the buffering in the child process (some programs detect that their output is or not a terminal with isatty).
What an event polling loop like above gives you is to avoid the deadlock situation where the child process is blocked because its stdout pipe is full, while the parent is blocked writing (to the child's stdin pipe) because the pipe is full: with an event loop, you read as soon as some data is polled readable on the input pipe (i.e. the stdout of the child process), and you write some data as soon as the output pipe is polled writable (i.e. is not full). You cannot predict in advance in which order these events "output of child is readable by parent" and "input of child is writable by parent" happen.
I recommend reading Advanced Linux Programming which has several chapters explaining these issues!
BTW my simplistic event loop is a bit wrong: if the child process terminated and some data remains in its stdout pipe, its reading is not done. You could move the waitpid test after the poll
Also, don't expect that a single write (from the child process) into a pipe would trigger one single read in the parent process. In other words, there is no notion of message length. However, POSIX knows about PIPE_MAX .... See its write documentation. Probably your buffer passed to read and write should be of PIPE_MAX size.
I repeat: you need to call poll inside your event loop and very probably poll will be called several times (because your loop will be repeated many times!), and will report readable or writable pipe ends in an unpredictable (and non-reproducible) order! A first run of your program could report "rpipe[0] readable", you read 324 bytes from it, you repeat the event loop, poll says you "wpipe[1] writable", you can write 10 bytes to it, you repeat the event loop, poll tells that "rpipe[0] readable", you read 110 bytes from it, you repeat the event loop, poll tells again "rpipe[0] readable", you read 4096 bytes from it, etc etc etc... A second run of the same program in the same environment would give different events, like: poll says that "wpipe[1] writable", you write 1000 bytes to it, you repeat the loop, poll says that "rpipe[0] readable, etc....
NB: your issue is not the buffering in the child ("client") program, which we assume you cannot change. So what matters is not the buffered data in it, but the genuine input and output (that is the only thing your parent process can observe; internal child buffering is irrelevant for the parent), i.e. the data that your child program has been able to really read(2) and write(2). And if going thru a pipe(7), such data will become poll(2)-able in the parent process (and your parent process can read or write some of it after POLL_IN or POLL_OUT in the updated revents field after poll). BTW, if you did code the child, don't forget to call fflush at appropriate places inside it.