BackgroundI'm playing around with a FIFO and every time I try to write on my fifo it blocks until someone is reading the other side of the fd:
int fd;
char buffer[100] = {0};
char * myfifo = "/tmp/myfifo";
mkfifo(myfifo, 0666);
printf("What would you like to send?\n");
fgets(buffer, 100, stdin);
if((fd = open(myfifo, O_WRONLY)) < 0)
printf("Couldn't open the FIFO for writing!\n");
else {
write(fd, buffer, strlen(buffer));
close(fd);
This code works, but it blocks until I read the /tmp/myfifo side and get the data. When I change the code as such:
if((fd = open(myfifo, O_WRONLY | O_NONBLOCK)) < 0)
Then the open fails with error No such device or address, unless I have someone sitting blocked on the "read" side.
Analysis
As per the man page for fifo():
The kernel maintains exactly one pipe object for each FIFO special file that is opened by at least one process. The FIFO must be opened on both ends (reading and writing) before data can be passed. Normally, opening the FIFO blocks until the other end is opened also.
So this is expected operation.
Question
Based on my experiments, and what I'm reading... I have to assume a FIFO is a non-queued, non-buffered mechanism and only works when there is a process sitting and waiting for data.
Is there a different communication mechanism that works in a non-blocked buffered manor, basically a buffered FIFO, or would I have to make my own message storage/notification system for that?
EDIT
I say I'm "playing around" which is actually pretty descriptive here. I'm trying to learn the ins and outs of the various IPC mechanisms (FIFOs, sockets, and pipes). I'm working towards learning to use select() and understand what can be used to wake up sleeping processes which call select. Reason being it's part of a communications driver I'm analyzing for port to a new platform.
I excluded this from the original post as it is sort of irrelevant. I'm just trying to make sure I can understand (at the moment) FIFO's, how to use them, the limitations on them, and other IPC mechanism. Hence my original assumption/questions about "better" versions of FIFO that will store data and can be written to without blocking.
Not only is a FIFO buffered, but that's basically all a FIFO is. A FIFO is little more than a buffer in the kernel.
Discussion: The kernel has a policy that it refuses to write data to the buffer unless a process has the FIFO open for reading. This behavior is similar to pipes and TCP connections, although if there's no reader for a pipe or a TCP connection, the kernel will actually signal the writing process, terminating it (unless you install a handler). This behavior allows us to string together commands the way we expect, e.g.,
hexdump file.dat | head
The hexdump program gets killed once head reads a few lines. This is what we want 99% of the time, and hexdump doesn't need any special code to achieve this.
Solutions: It would help if you describe some more context about the problem you are trying to solve.
If you want a client/server system where the server queues messages that can be read by the clients, you can achieve this with Unix domain sockets. Unix domain sockets are similar to FIFOs but more flexible in various ways. (Most database servers prefer Unix domain sockets over other types of IPC.)
If you want to be able to store a persistent queue, where applications can independently enqueue and dequeue messages, then you will need to use ordinary files.
Unfortunately, "playing around with a FIFO" is not much to go on. If you want a good exercise in IPC, try writing a chat server that uses Unix domain sockets or TCP (or both at the same time, which isn't much harder). You can use telnet or nc (netcat) as a client. (Make sure you have the "OpenBSD" version of netcat.)
Related
I have a C program that receives data from another program over a (Linux) pipe. I want the program to behave differently if the pipe was closed before writing any data.
The natural way to do this is to try to read from the pipe and check if I get EOF, but that consumes some data from the pipe if there is any available, and (as far as I know) there's no way to put data "back" in a pipe.
The part of the program where I want to check if the pipe is empty is pretty far away from where I process the data, so I'd rather not have to deal with saving the data from my first read until then.
Is there any way to check if a pipe is empty (read would return EOF) without consuming any data in the case it's not empty?
Note: I do want this to block if the pipe has not been written to or closed yet.
If you used Unix domain stream sockets instead of pipes – meaning you replace your pipe(fds) calls with socketpair(AF_UNIX, SOCK_STREAM, 0, fds) –, you could use recv(fd, dummybuffer, 1, MSG_PEEK) to read/receive one byte of data, without removing it from the receive buffer.
You can combine MSG_PEEK with MSG_DONTWAIT if you don't want to block, or with MSG_WAITALL if you want to block until the entire buffer can be filled.
The differences between an Unix domain stream socket and a pipe are minimal. The stream socket is bidirectional, but you can use shutdown(fd, SHUT_WR) (or SHUT_RD) to close the "write end" (resp. "read end"), meaning if the other end tries to read from the socket, they'll get an immediate end-of-stream (read(), recv() etc. return 0). (Closing the "read end" means that when the other end tries to write to the socket, they'll get EPIPE.)
Right now, I cannot even think of a reason why a program that works with a pipe would not work with an Unix domain stream socket pair.
If you use named pipes, you do need to change mkfifo() and open() to socket(AF_UNIX, SOCK_STREAM, 0) followed by a bind() to the socket address. read(), write(), and even the higher-level standard I/O facilities work just fine on top of an Unix domain stream socket (use fdopen() to convert the socket descriptor to a FILE handle).
If you cannot modify the readers, you can create a minimal dynamic library that interposes openat() (that's what current C library uses underneath fopen()), calling original openat() for all except the socket path, say named in an environment variable, and instead creates a socket and binds to the socket path for that one. When executing the reader binaries, you just set LD_PRELOAD to point to this interposing library.
In other words, I do believe there are no real obstacles for switching from pipes to Unix domain stream sockets.
You cannot use recv() with pipes, because pipes are implemented in Linux using a special filesystem, not sockets.
No, there is no way to do what you describe. The way to determine whether you have reached the end of a non-seekable file such as a pipe is to attempt to read from it. This is not just the natural way, it is the way.
but that consumes some data from the pipe if there is
any available,
Yes.
and (as far as I know) there's no way to put data
"back" in a pipe.
That depends. If you are reading with POSIX read(), then no. If you are wrapping the the pipe end in a FILE and using stdio functions to read it then there is ungetc().
Nevertheless, this:
The part of the program where I want to check if the
pipe is empty is pretty far away from where I process the data
seems like a design problem. You cannot know whether you will ever get data until you actually do get data or see EOF. The process(es) at the write end of the pipe can delay an arbitrary amount of time before doing anything with the pipe, and even if that process is provided by you, you cannot be fully in control of this aspect of its behavior. Thus, it doesn't make much sense to try to check for EOF before you're ready, in some sense, to consume data, because you cannot rely on getting an answer without blocking.
, so I'd
rather not have to deal with saving the data from my first read until
then.
I suppose you must want to avoid performing some kind of heavyweight initialization in the event that there is no data to process. Ok, but I don't see what the big deal is. You need to provide storage into which to read the data anyway. What's wrong with something like this:
void consume_pipe_data(int fd) {
char buffer[BUFFER_SIZE];
ssize_t count;
count = read(fd, buffer, BUFFER_SIZE);
if (count == 0) {
handle_no_data();
return;
} else if (count > 0) {
perform_expensive_initialization();
}
do {
if (count == -1) {
handle_error();
return;
}
consume_data(buffer);
count = read(fd, buffer, BUFFER_SIZE);
} while (count);
}
The point is not that that's necessarily an appropriate structure for your program, but rather that it is possible to structure the program so that storing the data, if any, from the initial read is pretty clean and natural.
You can do a dummy-write. If your stdin has reached eof, it is a non-blocking mechanism to determine if you've reached EOF, without any more sophisticated tools.
if( write( fileno(stdin), 0, 0 ) != 0 )
return 1; // Is end-of-file.
I have read the Linux manpage of pipe2, that states that you can use the O_DIRECT flag to create a pipe that performs I/O in packet mode.
I haven't found anything similar for fifos (named pipes) although I have heard that pipes and fifos share a lot of code in Linux, and it could be useful to me in a project (we already pass messages in fifos, but we have to seek for an especial terminator, reading one byte at a time).
Is there anything equivalent to perform fifo I/O in packet mode?
The O_* flags on a file descriptor can usually be changed by fcntl(fd, F_SETFL, ...) but in this case I don't think it will work, because of this:
https://lkml.org/lkml/2015/12/15/480
It's a patch that was submitted just 2 weeks ago to support exactly this use case, and the only replies to it have been a couple of build failures from an automated tester.
So you can try fixing that patch and applying it (should be easy - looks like a typo, the f should be filp)...
or (this is the option I'd prefer) see if your needs can be met by an AF_UNIX, SOCK_SEQPACKET socket instead of a pipe with this new flag. They're more powerful and more portable.
(I assume you already tried passing the O_DIRECT to open when you open the named pipe, since that's the most obvious thing.)
Both pipe & fifo are byte stream, it open a file descriptor, then use read() and write() to do communication.
If u want a packet (I am not sure what u mean, I assume you want to read a block of data without determine the boundary by yourself), POSIX - message queue might be a good choice. It send / receive data in unit of a message instead of byte by byte.
And you can also set priority on the messages which change the order of receiving, if all messages have the same priority (e.g 0), then the send & receive order is the same as FIFO.
If that's what u want, then check man mq_overview for details.
I'm not able to switch to a socket or message queue, and am stuck with using pipe. But good news, the patch refereed to in user2404501's answer eventually got accepted and is in linux v4.5 and newer. So using fnctl() to set O_DIRECT on a named pipe is valid.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dbf5f20652108106cb822ad7662c786baaa03ff
I have to make a simple IRC client/server programs for my IT school. The subject asks us to use select(2) for socket polling but forbids us to use O_NONBLOCK sockets.
Your server will accept multiple simultaneous connections.
Attention, the use of fork is prohibited. So you should imperatively use select
Your server must not be blocking.
This has nothing to do with non-blocking sockets, which are prohibited (so do not use fcntl(s, O_NONBLOCK))
I’m wondering if it is even possible to design a non-blocking server (which does not fork) with blocking sockets even using select(2).
Here is a simple example: let say we have a simple text protocol with one command per line. Each client has a buffer. When select(2) tells us a client is ready for read(2), we read until we found a \n in the client buffer, therefor we process the command. With non-blocking sockets, we would read until EAGAIN.
Let imagine now that we are using blocking sockets and a malicious client sends text with no line-break. select(2) tells us data is available, we then read(2) on the client. But we will never read the expected \n. Instead of returning EAGAIN, the syscall will block indefinitely. This is a denial of service attack.
Is it really possible to design a non-blocking server with blocking-sockets and select(2) (no fork(2))?
Yes, you read once from the socket that select tells you is ready. If the read contains the \n, then process that line. Otherwise, store any data that was received, and immediately go back to the select.
This means of course, that for every open socket, you must maintain state information, and a buffer of data read so far. This allows the code to process each read independently, without the need to finish a full line before going back to the select.
It's impossible.
select() blocks, and therefore so does any program that calls it.
The behaviour defined by Posix for send() in blocking mode is that it blocks until all the data supplied has been transferred to the socket send buffer. Unless you're going to delve into low-water marks and so on, it is impossible to know in advance whether there is enough room in he socket send buffer for any given send() to complete without blocking, and therefore impossible for any program that calls send() not to block.
Note that select() doesn't help you with this. It can tell when you when there is some room, but not when there is enough.
I want to use named fifo channel and I want to implement a timeout when I write in this fifo.
fd = open(pipe, O_WRONLY);
write(fd, msg, len);
Program is blocked by function open, so using the function select will not work.
Thanks.
use select() and its timeout argument.
Read pipe(7), fifo(7), poll(2)
You might setup a timer or or alarm with a signal handler (see time(7) & signal(7)) before your call to open(2) - but I won't do that - or you could use the O_NONBLOCK flag, since fifo(7) says:
A process can open a FIFO in nonblocking mode. In this case, opening
for read-only will succeed even if no-one has opened on the write
side yet, opening for write-only will fail with ENXIO (no such device
or address) unless the other end has already been opened.
However, you need something (some other process reading) on the other side of the FIFO or pipe.
Perhaps you should consider using unix(7) sockets, i.e. the AF_UNIX address family. It looks more relevant to your case: change your code above (trying to open for writing a FIFO) to a AF_UNIX socket on the client side (with a connect), and change the other process to become an AF_UNIX socket server.
As 5gon12eder commented, you might also look into inotify(7). Or even perhaps D-bus !
I'm guessing that FIFOs or pipes are not the right solution in your situation. You should explain more and give a broader picture of your concerns and goals.
I have created a FIFO where I can do non-blocking writes into in this way:
// others, searching for a non-blocking FIFO-writer may copy this ;-)
mkfifo("/tmp/myfifo", S_IRWXU);
int fifo_fd = open("/tmp/myfifo", O_RDWR);
fcntl(fifo_fd, F_SETFL, fcntl(fifo_fd, F_GETFL) | O_NONBLOCK);
// and then in a loop:
LOGI("Writing into fifo.");
if (write(fifo_fd, data, count) < 0) {
LOGE("Failed to write into fifo: %s", strerror(errno));
}
The non-blocking write works perfect.
On the other side, I open the FIFO for read and do the same fcntl() to make the read() non-blocking.
I now would like to make several (cpu-intensive) calculations on the write side, but ONLY if there is a reader attached.
Therefor I need to find a way on the write side, to detect if the FIFO is opended for read somewhere else.
Has anyone an idea how to achieve this?
I now would like to make several (cpu-intensive) calculations on the write side, but ONLY if there is a reader attached.
For that you can simply create a socket, and when a consumer connects to it, do some work and write back.
But I think a better solution is to have calculation results ready for consumers before they connect (or open FIFO). but you don't want the producer working if work is not being consumed. So define N, number of work results you're willing to keep available for consumption and let the producer (or producers) work and save results in a queue of size N until it's full.
You could implement this with threads, one thread listens for connections, pops from the queue and writes to the consumer and one or more producer threads working and pushing to the queue.
Or you could use POSIX message queues to avoid threading headaches. Create a queue of size N, independent producers (multiple processes written in different languages) could push to the queue until its full, and multiple independent consumers pop from it.