I have a popen() function which executes tail -f sometextfile. Aslong as there is data in the filestream obviously I can get the data through fgets(). Now, if no new data comes from tail, fgets() hangs. I tried ferror() and feof() to no avail. How can I make sure fgets() doesn't try to read data when nothing new is in the file stream?
One of the suggestion was select(). Since this is for Windows Platform select doesn't seem to work as anonymous pipes do not seem to work for it (see this post).
In Linux (or any Unix-y OS), you can mark the underlying file descriptor used by popen() to be non-blocking.
#include <fcntl.h>
FILE *proc = popen("tail -f /tmp/test.txt", "r");
int fd = fileno(proc);
int flags;
flags = fcntl(fd, F_GETFL, 0);
flags |= O_NONBLOCK;
fcntl(fd, F_SETFL, flags);
If there is no input available, fgets will return NULL with errno set to EWOULDBLOCK.
fgets() is a blocking read, it is supposed to wait until data is available if there is no data.
You'll want to perform asynchronous I/O using select(), poll(), or epoll(). And then perform a read from the file descriptor when there is data available.
These functions use the file descriptor of the FILE* handle, retrieved by: int fd = fileno(f);
i solved my problems by using threads , specifically _beginthread , _beginthreadex.
You can instead try reading sometextfile using low-level IO functions (open(), read(), etc.), like tail itself does. When there's nothing more to read, read() returns zero, but will still try to read more the next time, unlike FILE* functions.
I you would use POSIX functions for IO instead of those of C library, you could use select or poll.
Related
I have a C program that receives data from another program over a (Linux) pipe. I want the program to behave differently if the pipe was closed before writing any data.
The natural way to do this is to try to read from the pipe and check if I get EOF, but that consumes some data from the pipe if there is any available, and (as far as I know) there's no way to put data "back" in a pipe.
The part of the program where I want to check if the pipe is empty is pretty far away from where I process the data, so I'd rather not have to deal with saving the data from my first read until then.
Is there any way to check if a pipe is empty (read would return EOF) without consuming any data in the case it's not empty?
Note: I do want this to block if the pipe has not been written to or closed yet.
If you used Unix domain stream sockets instead of pipes – meaning you replace your pipe(fds) calls with socketpair(AF_UNIX, SOCK_STREAM, 0, fds) –, you could use recv(fd, dummybuffer, 1, MSG_PEEK) to read/receive one byte of data, without removing it from the receive buffer.
You can combine MSG_PEEK with MSG_DONTWAIT if you don't want to block, or with MSG_WAITALL if you want to block until the entire buffer can be filled.
The differences between an Unix domain stream socket and a pipe are minimal. The stream socket is bidirectional, but you can use shutdown(fd, SHUT_WR) (or SHUT_RD) to close the "write end" (resp. "read end"), meaning if the other end tries to read from the socket, they'll get an immediate end-of-stream (read(), recv() etc. return 0). (Closing the "read end" means that when the other end tries to write to the socket, they'll get EPIPE.)
Right now, I cannot even think of a reason why a program that works with a pipe would not work with an Unix domain stream socket pair.
If you use named pipes, you do need to change mkfifo() and open() to socket(AF_UNIX, SOCK_STREAM, 0) followed by a bind() to the socket address. read(), write(), and even the higher-level standard I/O facilities work just fine on top of an Unix domain stream socket (use fdopen() to convert the socket descriptor to a FILE handle).
If you cannot modify the readers, you can create a minimal dynamic library that interposes openat() (that's what current C library uses underneath fopen()), calling original openat() for all except the socket path, say named in an environment variable, and instead creates a socket and binds to the socket path for that one. When executing the reader binaries, you just set LD_PRELOAD to point to this interposing library.
In other words, I do believe there are no real obstacles for switching from pipes to Unix domain stream sockets.
You cannot use recv() with pipes, because pipes are implemented in Linux using a special filesystem, not sockets.
No, there is no way to do what you describe. The way to determine whether you have reached the end of a non-seekable file such as a pipe is to attempt to read from it. This is not just the natural way, it is the way.
but that consumes some data from the pipe if there is
any available,
Yes.
and (as far as I know) there's no way to put data
"back" in a pipe.
That depends. If you are reading with POSIX read(), then no. If you are wrapping the the pipe end in a FILE and using stdio functions to read it then there is ungetc().
Nevertheless, this:
The part of the program where I want to check if the
pipe is empty is pretty far away from where I process the data
seems like a design problem. You cannot know whether you will ever get data until you actually do get data or see EOF. The process(es) at the write end of the pipe can delay an arbitrary amount of time before doing anything with the pipe, and even if that process is provided by you, you cannot be fully in control of this aspect of its behavior. Thus, it doesn't make much sense to try to check for EOF before you're ready, in some sense, to consume data, because you cannot rely on getting an answer without blocking.
, so I'd
rather not have to deal with saving the data from my first read until
then.
I suppose you must want to avoid performing some kind of heavyweight initialization in the event that there is no data to process. Ok, but I don't see what the big deal is. You need to provide storage into which to read the data anyway. What's wrong with something like this:
void consume_pipe_data(int fd) {
char buffer[BUFFER_SIZE];
ssize_t count;
count = read(fd, buffer, BUFFER_SIZE);
if (count == 0) {
handle_no_data();
return;
} else if (count > 0) {
perform_expensive_initialization();
}
do {
if (count == -1) {
handle_error();
return;
}
consume_data(buffer);
count = read(fd, buffer, BUFFER_SIZE);
} while (count);
}
The point is not that that's necessarily an appropriate structure for your program, but rather that it is possible to structure the program so that storing the data, if any, from the initial read is pretty clean and natural.
You can do a dummy-write. If your stdin has reached eof, it is a non-blocking mechanism to determine if you've reached EOF, without any more sophisticated tools.
if( write( fileno(stdin), 0, 0 ) != 0 )
return 1; // Is end-of-file.
I looked on the Linux man pages for the answer but can't seem to find it. I know that read() is blocking but I'm still not sure about write().
Can anyone point me to any documentation for clarification?
Read POSIX on read() and
write(). See also functions such as open() and pipe().
It depends on the attributes of the file descriptor you're reading from or writing to (think O_NONBLOCK, for example), and on the underlying file type (disk file vs pipe vs FIFO vs socket vs character or block special), and so on.
Succinctly, both read() and write() can be blocking or non-blocking, depending on circumstances.
First I have following macro
#define MSG_UPDATE_DATA 70
Then open a pipe with popen
SensServer = popen("./SensServer", "w") ;
In the following code that uses the putc(...) function to write to pipe, the function makes the program block and the lines of code following do not execute
void requestTempAndPress(int pid) {
printf("Temp and presure requested. msg_type: %d\n", MSG_UPDATE_DATA);
int n = putc(MSG_UPDATE_DATA, SensServer);
printf("Data sent: %d\n", MSG_UPDATE_DATA);
}
It outputs Temp and presure requested. msg_type: 70 fine. But not the "Data sent..." line.
As per the man page,
pipe is a fd, type int.
putc() needs a FILE* stream as argument.
So, most possibly, in your code, you are supplying the wrong type of argument to putc(), creating the issue.
Given the information (and lack of a sample program), this sounds like a question asking how to make pipes non-blocking. This has been discussed before, usually for nonblocking reads, e.g.,
Non-blocking pipe using popen?
Correct Code - Non-blocking pipe with popen (C++)
The first link mentions fcntl and the O_NONBLOCK flag which the manual page says can be applied to both reads and writes.
However, using popen makes the pipe using buffered I/O, while the operations addressed by fcntl are non-buffered read and write (you really cannot mix the two). If the program were changed to use the low-level pipe (as in the example for the first link), and consistently used the non-buffered I/O, it would give the intended behavior.
Here are links to more general discussion on the topic:
Introduction to non-blocking I/O
Blocking and Non-Blocking I/0
On the other hand (noting comments), if the program fragment is for example part of some larger system doing handshaking (expecting a timely response back from the server), that will run into problems. The fragment is writing a single character across the pipe. However, popen opens a (block-)buffered stream. Nothing will be sent directly to the server as single-character writes unless some help is provided. For instance, one could flush the output stream after each putc, e.g.,
fflush(SensServer);
Alternatively, one could make the stream unbuffered by changing it immediately after the successful call to popen, e.g., using setvbuf:
setvbuf(SensServer, NULL, _IONBF, 0);
Here are links for further reading about buffering in pipes:
Turn off buffering in pipe
Unix buffering delays output to stdout, ruins your day
Force line-buffering of stdout when piping to tee
The problem was due to the fact I initialised the variable SensServer in the parent process but not the child. Which meant the pointer was 0 or I guess a random memory location.
I would like to create blocking and non-blocking file in Unix's C. First, blocking:
fd = open("file.txt", O_CREAT | O_WRONLY | O_EXCL);
is that right? Shouldnt I add some mode options, like 0666 for example?
How about non-blocking file? I have no idea for this.
I would like to achieve something like:
when I open it to write in it, and it's opened for writing, it's ok; if not it blocks.
when I open it to read from it, and it's opened for reading, it's ok; if not it blocks.
File descriptors are blocking or non-blocking; files are not. Add O_NBLOCK to the options in the open() call if you want a non-blocking file descriptor.
Note that opening a FIFO for reading or writing will block unless there's a process with the FIFO open for the other operation, or you specify O_NBLOCK. If you open it for read and write, the open() is non-blocking (will return promptly); I/O operations are still controlled by whether you set O_NBLOCK or not.
The updated question is not clear. However, if you're looking for 'exclusive access to the file' (so that no-one else has it open), then neither O_EXCL nor O_NBLOCK is the answer. O_EXCL affects what happens when you create the file; the create will fail if the file already exists. O_NBLOCK affects whether a read() operation will block when there's no data available to read. If you read the POSIX open() description, there is nothing there that allows you to request 'exclusive access' to a file.
To answer the question about file mode: if you include O_CREAT, you need the third argument to open(). If you omit O_CREAT, you don't need the third argument to open(). It is a varargs function:
int open(const char *filename, int options, ...);
I don't know what you are calling a blocking file (blocking IO in Unix means that the IO operations wait for the data to be available or for a sure failure, they are opposed to non-blocking IO which returns immediately if there is no available data).
You always need to specify a mode when opening with O_CREAT.
The open you show will fails if the file already exists (when fixed for the above point).
Unix has no standard way to lock file for exclusive access excepted that. There are advisory locks (but all programs must respect the protocol). Some have mandatory lock extension. The received wisdom is not to rely on either kind of locking when accessing network file system.
Shouldn't I add some mode options?
You should, if the file is write-only and to be created if nonexistent. In this case, open() expects a third argument as well, so omitting it results in undefined behavior.
Edit:
The updated question is even more confusing...
when I open it to write in it, and it's opened for writing, it's ok; if not it blocks.
Why would you need that? See, if you try to write to a file/file descriptor not opened for writing, write() will return -1 and you can check the error code stored in errno. Tell us what you're trying to achieve by this bizarre thing you want instead of overcomplicating and messing up your code.
(Remarks in parentheses:
I would like to create blocking and non-blocking file
What's that?
in unix's C
Again, there's no such thing. There is the C language, which is platform-independent.)
I have a file descriptor that is open for reading which may be non-blocking. What I need to do is simply read all data until reaching EOF and write that data to a writable file descriptor. As the process that performs this copying is not "aware" of anything that is going on around it, I don't think that I can do something useful while waiting for data, and I don't want to use a while loop (while errno is not EAGAIN) because I think that it would be wasteful. Is there a way to block or otherwise suspend execution of the copying process until data becomes available?
Your other answer simply unsets O_NONBLOCK, which sets the file back to blocking. That's perfectly fine, if that works in your situation; but if not, you can use select() to block until your non-blocking file descriptor is readable.
Chapter 7 of the Linux SCSI Generic (sg) HOWTO gives an example of how to do this:
int flags = fcntl(fd, F_GETFL);
fcntl(fd, F_SETFL, flags & (~O_NONBLOCK));