This question already has answers here:
Determine the size of a pipe without calling read()
(13 answers)
Closed 9 years ago.
I am writing a remote terminal application and i use pipes to take data from a child process' stdout/stderr to then send it to a client. Specifically, I am running Linux, and I don't really care about portability (if the solution is Linux-only, it's OK).
I need to know how many bytes have currently been written to, but not yet read from, the pipe. I was Googling for a long time and couldn't find an answer. I need to read as many bytes from the pipe as possible without blocking.
Is this possible? Thank you for help.
The only way to do this is to attempt a read and count how many bytes you get. recv with MSG_PEEK can do that.
Your best bet will be to set your read end of the pipe to non blocking and then just read ahead. If your file descriptor is set to non blocking, you read on it and the pipe is empty then you well get EAGAIN error in errno. This will indicate to you that the pipe is still open, but just empty at the moment.
You can set your file descriptor to non block with
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) & ~O_NONBLOCK);
Then read:
while (read(fd, buffer, sizeof(buffer))) {
if (errno == EAGAIN) {} // buffer currently empty
else if (errno) {perror("read")} // an error happened
}
Related
I have a C program that receives data from another program over a (Linux) pipe. I want the program to behave differently if the pipe was closed before writing any data.
The natural way to do this is to try to read from the pipe and check if I get EOF, but that consumes some data from the pipe if there is any available, and (as far as I know) there's no way to put data "back" in a pipe.
The part of the program where I want to check if the pipe is empty is pretty far away from where I process the data, so I'd rather not have to deal with saving the data from my first read until then.
Is there any way to check if a pipe is empty (read would return EOF) without consuming any data in the case it's not empty?
Note: I do want this to block if the pipe has not been written to or closed yet.
If you used Unix domain stream sockets instead of pipes – meaning you replace your pipe(fds) calls with socketpair(AF_UNIX, SOCK_STREAM, 0, fds) –, you could use recv(fd, dummybuffer, 1, MSG_PEEK) to read/receive one byte of data, without removing it from the receive buffer.
You can combine MSG_PEEK with MSG_DONTWAIT if you don't want to block, or with MSG_WAITALL if you want to block until the entire buffer can be filled.
The differences between an Unix domain stream socket and a pipe are minimal. The stream socket is bidirectional, but you can use shutdown(fd, SHUT_WR) (or SHUT_RD) to close the "write end" (resp. "read end"), meaning if the other end tries to read from the socket, they'll get an immediate end-of-stream (read(), recv() etc. return 0). (Closing the "read end" means that when the other end tries to write to the socket, they'll get EPIPE.)
Right now, I cannot even think of a reason why a program that works with a pipe would not work with an Unix domain stream socket pair.
If you use named pipes, you do need to change mkfifo() and open() to socket(AF_UNIX, SOCK_STREAM, 0) followed by a bind() to the socket address. read(), write(), and even the higher-level standard I/O facilities work just fine on top of an Unix domain stream socket (use fdopen() to convert the socket descriptor to a FILE handle).
If you cannot modify the readers, you can create a minimal dynamic library that interposes openat() (that's what current C library uses underneath fopen()), calling original openat() for all except the socket path, say named in an environment variable, and instead creates a socket and binds to the socket path for that one. When executing the reader binaries, you just set LD_PRELOAD to point to this interposing library.
In other words, I do believe there are no real obstacles for switching from pipes to Unix domain stream sockets.
You cannot use recv() with pipes, because pipes are implemented in Linux using a special filesystem, not sockets.
No, there is no way to do what you describe. The way to determine whether you have reached the end of a non-seekable file such as a pipe is to attempt to read from it. This is not just the natural way, it is the way.
but that consumes some data from the pipe if there is
any available,
Yes.
and (as far as I know) there's no way to put data
"back" in a pipe.
That depends. If you are reading with POSIX read(), then no. If you are wrapping the the pipe end in a FILE and using stdio functions to read it then there is ungetc().
Nevertheless, this:
The part of the program where I want to check if the
pipe is empty is pretty far away from where I process the data
seems like a design problem. You cannot know whether you will ever get data until you actually do get data or see EOF. The process(es) at the write end of the pipe can delay an arbitrary amount of time before doing anything with the pipe, and even if that process is provided by you, you cannot be fully in control of this aspect of its behavior. Thus, it doesn't make much sense to try to check for EOF before you're ready, in some sense, to consume data, because you cannot rely on getting an answer without blocking.
, so I'd
rather not have to deal with saving the data from my first read until
then.
I suppose you must want to avoid performing some kind of heavyweight initialization in the event that there is no data to process. Ok, but I don't see what the big deal is. You need to provide storage into which to read the data anyway. What's wrong with something like this:
void consume_pipe_data(int fd) {
char buffer[BUFFER_SIZE];
ssize_t count;
count = read(fd, buffer, BUFFER_SIZE);
if (count == 0) {
handle_no_data();
return;
} else if (count > 0) {
perform_expensive_initialization();
}
do {
if (count == -1) {
handle_error();
return;
}
consume_data(buffer);
count = read(fd, buffer, BUFFER_SIZE);
} while (count);
}
The point is not that that's necessarily an appropriate structure for your program, but rather that it is possible to structure the program so that storing the data, if any, from the initial read is pretty clean and natural.
You can do a dummy-write. If your stdin has reached eof, it is a non-blocking mechanism to determine if you've reached EOF, without any more sophisticated tools.
if( write( fileno(stdin), 0, 0 ) != 0 )
return 1; // Is end-of-file.
After ignoring SIGPIPE with a call to signal(), there is a possibility that a write() call will fail with errno set equal to EPIPE. Do I still have to close the file descriptor like in the example below?
if (write(sock_fd, buf, len) < 0) {
if (errno == EPIPE) {
close(sock_fd);
}
}
Remember that when you create a pipe, using the system call of the same name, you get two file descriptors: one for reading and one for writing. You get EPIPE from a write operation, on the write fd, when the read fd has been closed. The write fd is still open. If you try to write to it again, you'll get EPIPE again.
(Often, when this happens, the pipe was set up by a shell or some other parent process, and the read fd was never available to your program, but that doesn't matter to the kernel. It was open in some process(es) and now it isn't.)
Since it's still open, you do need to close it. However, exiting automatically closes all fds that are still open. So if the very next thing you'd do after closing the pipe is to exit, then you don't need to bother closing it first. Since it is a pipe, and you already got EPIPE, there can't be any delayed write errors that close might report.
You always have to close file descriptors. No ifs, no buts.
This question already has an answer here:
How to check if the pipe is opend before writing?
(1 answer)
Closed 8 years ago.
Is it possible to check if a pipe is broken before trying to write/read to it, so I can just skip it and continue with the program?
I'm utilizing a while loop to write to pipes communicating from the Parent to multiple Children. Over the course of the loop, several of the children will close. When the loop comes around and tries to write to them again, my program shuts down because it is killed by SIGPIPE since the pipe is broken. I know the pipe is broken, I programmed the children to close their pipes and exit (necessary). I still want to finish the loop and move on with the program. I'd like it to check if the pipe is broken, skip over it if broken (without an error output), and move on with the program (the other children still need to be written to).
So, is it possible? An example in c would be great.
Here's a simple pseudocode representation of what I'm asking:
int i = 0;
while(i != 5)
{
if (mypipe[WRITE] is BROKEN)
//Do Nothing ---Basically Skip
else if (some condition)
write(mypipe[WRITE]);
i++;
}
This question is related to a previous post of mine, though the context of my question is different. You can review the code for the actual program there.
my program shuts down because it is killed by SIGPIPE
You can change that behavior if you install a signal handler that ignores SIGPIPE, that way the write() call to the pipe returns an error instead of killing your program:
signal(SIGPIPE, SIG_IGN);
Then you simple check
ssize_t rc;
rc = write(...);
if (rc == -1) {
if (errno == EPIPE) {
//it's broken
}
//some other error occured.
}
You can set up a signal handler to ignore SIGPIPE, and then write to the buffer and check the error code:
/* initialization */
signal(SIGPIPE, SIG_IGN);
...
/* try writing to pipe and detect broken pipe */
ssize_t written = write(mypipe[WRITE], buf, size);
if (written == -1 && errno == EPIPE)
... remove pipe from the list and continue processing
Yes, you can use a multiplexing syscall like poll(2) before reading and writing. This will tell you which file descriptors are readable (have something to read) and writable (can be subject of write), amongst a sequence of file descriptors to check for readability or writability.
(You could use the older select(2) multiplexing syscall, but I prefer poll because of the C10K problem)
Beware that some other thread might do something before your own thread.
You want some event loop. See also this answer to a related question.
The following code reads messages from other processes through a pipe. All processes correctly print out all the messages, but then they will never proceed past the while loop. Tried debugging in Eclipse, after reading reading all the messages, it will just stop at that while loop.
The index is a number assigned to each process. The first process would have index == 0.
The message itself is simply the index of the process sending the message.
while((n = read(fd[index][0], &mymsg, sizeof(int))) == sizeof(int))
printf("process%d has received a message from process%d\n", index, mymsg);
Any ideas why this would happen?
Here is how each process writes to another:
// Write to other process
if(write(fd[index2][1], &index, sizeof(int)) != sizeof(int))
sys_error(2);
This is done five times. fd is a table of read-and-write ends for each process.
The call to read() is blocking until more data shows up. From the man page for pipe
If a process attempts to read from an empty pipe, then read(2) will
block until data is available. If a process attempts to write to a
full pipe (see below), then write(2) blocks until sufficient data has
been read from the pipe to allow the write to complete. Nonblocking
I/O is possible by using the fcntl(2) F_SETFL operation to enable the
O_NONBLOCK open file status flag.
After you open each file descriptor before you enter that while loop do this to each one:
fcntl(fd, F_SETFL, O_NONBLOCK);
However, you really should read up on blocking vs. non-blocking I/O, including reading the man pages for pipe, read, fcntl, etc.
I have a file descriptor that is open for reading which may be non-blocking. What I need to do is simply read all data until reaching EOF and write that data to a writable file descriptor. As the process that performs this copying is not "aware" of anything that is going on around it, I don't think that I can do something useful while waiting for data, and I don't want to use a while loop (while errno is not EAGAIN) because I think that it would be wasteful. Is there a way to block or otherwise suspend execution of the copying process until data becomes available?
Your other answer simply unsets O_NONBLOCK, which sets the file back to blocking. That's perfectly fine, if that works in your situation; but if not, you can use select() to block until your non-blocking file descriptor is readable.
Chapter 7 of the Linux SCSI Generic (sg) HOWTO gives an example of how to do this:
int flags = fcntl(fd, F_GETFL);
fcntl(fd, F_SETFL, flags & (~O_NONBLOCK));