Related
I have a small server program that accepts connections on a TCP or local UNIX socket, reads a simple command and (depending on the command) sends a reply.
The problem is that the client may have no interest in the answer and sometimes exits early. So writing to that socket will cause a SIGPIPE and make my server crash.
What's the best practice to prevent the crash here? Is there a way to check if the other side of the line is still reading? (select() doesn't seem to work here as it always says the socket is writable). Or should I just catch the SIGPIPE with a handler and ignore it?
You generally want to ignore the SIGPIPE and handle the error directly in your code. This is because signal handlers in C have many restrictions on what they can do.
The most portable way to do this is to set the SIGPIPE handler to SIG_IGN. This will prevent any socket or pipe write from causing a SIGPIPE signal.
To ignore the SIGPIPE signal, use the following code:
signal(SIGPIPE, SIG_IGN);
If you're using the send() call, another option is to use the MSG_NOSIGNAL option, which will turn the SIGPIPE behavior off on a per call basis. Note that not all operating systems support the MSG_NOSIGNAL flag.
Lastly, you may also want to consider the SO_SIGNOPIPE socket flag that can be set with setsockopt() on some operating systems. This will prevent SIGPIPE from being caused by writes just to the sockets it is set on.
Another method is to change the socket so it never generates SIGPIPE on write(). This is more convenient in libraries, where you might not want a global signal handler for SIGPIPE.
On most BSD-based (MacOS, FreeBSD...) systems, (assuming you are using C/C++), you can do this with:
int set = 1;
setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int));
With this in effect, instead of the SIGPIPE signal being generated, EPIPE will be returned.
I'm super late to the party, but SO_NOSIGPIPE isn't portable, and might not work on your system (it seems to be a BSD thing).
A nice alternative if you're on, say, a Linux system without SO_NOSIGPIPE would be to set the MSG_NOSIGNAL flag on your send(2) call.
Example replacing write(...) by send(...,MSG_NOSIGNAL) (see nobar's comment)
char buf[888];
//write( sockfd, buf, sizeof(buf) );
send( sockfd, buf, sizeof(buf), MSG_NOSIGNAL );
In this post I described possible solution for Solaris case when neither SO_NOSIGPIPE nor MSG_NOSIGNAL is available.
Instead, we have to temporarily suppress SIGPIPE in the current thread that executes library code. Here's how to do this: to suppress SIGPIPE we first check if it is pending. If it does, this means that it is blocked in this thread, and we have to do nothing. If the library generates additional SIGPIPE, it will be merged with the pending one, and that's a no-op. If SIGPIPE is not pending then we block it in this thread, and also check whether it was already blocked. Then we are free to execute our writes. When we are to restore SIGPIPE to its original state, we do the following: if SIGPIPE was pending originally, we do nothing. Otherwise we check if it is pending now. If it does (which means that out actions have generated one or more SIGPIPEs), then we wait for it in this thread, thus clearing its pending status (to do this we use sigtimedwait() with zero timeout; this is to avoid blocking in a scenario where malicious user sent SIGPIPE manually to a whole process: in this case we will see it pending, but other thread may handle it before we had a change to wait for it). After clearing pending status we unblock SIGPIPE in this thread, but only if it wasn't blocked originally.
Example code at https://github.com/kroki/XProbes/blob/1447f3d93b6dbf273919af15e59f35cca58fcc23/src/libxprobes.c#L156
Handle SIGPIPE Locally
It's usually best to handle the error locally rather than in a global signal event handler since locally you will have more context as to what's going on and what recourse to take.
I have a communication layer in one of my apps that allows my app to communicate with an external accessory. When a write error occurs I throw and exception in the communication layer and let it bubble up to a try catch block to handle it there.
Code:
The code to ignore a SIGPIPE signal so that you can handle it locally is:
// We expect write failures to occur but we want to handle them where
// the error occurs rather than in a SIGPIPE handler.
signal(SIGPIPE, SIG_IGN);
This code will prevent the SIGPIPE signal from being raised, but you will get a read / write error when trying to use the socket, so you will need to check for that.
You cannot prevent the process on the far end of a pipe from exiting, and if it exits before you've finished writing, you will get a SIGPIPE signal. If you SIG_IGN the signal, then your write will return with an error - and you need to note and react to that error. Just catching and ignoring the signal in a handler is not a good idea -- you must note that the pipe is now defunct and modify the program's behaviour so it does not write to the pipe again (because the signal will be generated again, and ignored again, and you'll try again, and the whole process could go on for a long time and waste a lot of CPU power).
Or should I just catch the SIGPIPE with a handler and ignore it?
I believe that is right on. You want to know when the other end has closed their descriptor and that's what SIGPIPE tells you.
Sam
What's the best practice to prevent the crash here?
Either disable sigpipes as per everybody, or catch and ignore the error.
Is there a way to check if the other side of the line is still reading?
Yes, use select().
select() doesn't seem to work here as it always says the socket is writable.
You need to select on the read bits. You can probably ignore the write bits.
When the far end closes its file handle, select will tell you that there is data ready to read. When you go and read that, you will get back 0 bytes, which is how the OS tells you that the file handle has been closed.
The only time you can't ignore the write bits is if you are sending large volumes, and there is a risk of the other end getting backlogged, which can cause your buffers to fill. If that happens, then trying to write to the file handle can cause your program/thread to block or fail. Testing select before writing will protect you from that, but it doesn't guarantee that the other end is healthy or that your data is going to arrive.
Note that you can get a sigpipe from close(), as well as when you write.
Close flushes any buffered data. If the other end has already been closed, then close will fail, and you will receive a sigpipe.
If you are using buffered TCPIP, then a successful write just means your data has been queued to send, it doesn't mean it has been sent. Until you successfully call close, you don't know that your data has been sent.
Sigpipe tells you something has gone wrong, it doesn't tell you what, or what you should do about it.
Under a modern POSIX system (i.e. Linux), you can use the sigprocmask() function.
#include <signal.h>
void block_signal(int signal_to_block /* i.e. SIGPIPE */ )
{
sigset_t set;
sigset_t old_state;
// get the current state
//
sigprocmask(SIG_BLOCK, NULL, &old_state);
// add signal_to_block to that existing state
//
set = old_state;
sigaddset(&set, signal_to_block);
// block that signal also
//
sigprocmask(SIG_BLOCK, &set, NULL);
// ... deal with old_state if required ...
}
If you want to restore the previous state later, make sure to save the old_state somewhere safe. If you call that function multiple times, you need to either use a stack or only save the first or last old_state... or maybe have a function which removes a specific blocked signal.
For more info read the man page.
Linux manual said:
EPIPE The local end has been shut down on a connection oriented
socket. In this case the process will also receive a SIGPIPE
unless MSG_NOSIGNAL is set.
But for Ubuntu 12.04 it isn't right. I wrote a test for that case and I always receive EPIPE withot SIGPIPE. SIGPIPE is genereated if I try to write to the same broken socket second time. So you don't need to ignore SIGPIPE if this signal happens it means logic error in your program.
Say that I call select() on a FD_SET containing a bunch of read file descriptors. What happens if during the select() call, one of the file descriptor closes? Assuming that some sort of error occurs, then is it my responsibility to find and remove the closed file descriptor from the set?
I don't believe this is specified anywhere; some systems may immediately return from select while others may continue blocking. Note that the only way this can happen is in a multi-threaded process (otherwise, the close cannot happen during select; even if it happened from a signal handler, select would have already been interrupted by the signal). As such, this situation arising probably indicates you have bigger issues to worry about. If one of the file descriptors you're polling can be closed during select, the bigger issue is that the same file descriptor might be reassigned to a newly opened file (e.g. one opened in another unrelated thread) immediately after the close, and the thread that's polling might then wrongly perform IO on the new file "belonging to" a different thread.
If you have a data object that consists of a set of file descriptors that will be polled with select in a multithreaded program, you almost surely need to be using some sort of synchronization primitive to control access to that set, and adding or removing file descriptors should require a lock that's mutually exclusive with the possibility that select (or any IO on the members) is in progress.
Of course in a multi-threaded program, it may be better not to use select at all and instead let blocking IO in multiple threads achieve the desired result without complicated locking logic.
The select() system call takes three fd_set parameters: Send, Receive, Exception. To check, if an error occurs on a reading file descriptor include it in the read (receive) and in the error (exceprion) set - seeing it in the exception set on return from select() means, an exception has occurred on that socket, giving you the chance to find out what.
In general network sockets with any sort of exception will no longer be fit to send and receive.
Even if you've read all the sent data, a closed socket is always regarded as ready to read. Select will unblock, signaling that socket to be available.
I know that broken pipe error is thrown when the socket on the peer side is closed.
But, in my test I have noted that an immediate 'send' call on this side when the peer side is closed doesn't always lead to a broken pipe error.
E.g.:
After closing the socket on peer side (I have tried clean closing by calling close and also abnormal closing by killing the peer), if I try to send 40 bytes, then I don't get a broken pipe, but, if I try to send 40000 bytes then it immediately gives broken pipe error.
What exactly causes broken pipe and can it's behavior be predicted?
It can take time for the network close to be observed - the total time is nominally about 2 minutes (yes, minutes!) after a close before the packets destined for the port are all assumed to be dead. The error condition is detected at some point. With a small write, you are inside the MTU of the system, so the message is queued for sending. With a big write, you are bigger than the MTU and the system spots the problem quicker. If you ignore the SIGPIPE signal, then the functions will return EPIPE error on a broken pipe - at some point when the broken-ness of the connection is detected.
The current state of a socket is determined by 'keep-alive' activity. In your case, this is possible that when you are issuing the send call, the keep-alive activity tells that the socket is active and so the send call will write the required data (40 bytes) in to the buffer and returns without giving any error.
When you are sending a bigger chunk, the send call goes in to blocking state.
The send man page also confirms this:
When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in non-blocking I/O mode. In non-blocking mode it would return EAGAIN in this case
So, while blocking for the free available buffer, if the caller is notified (by keep-alive mechanism) that the other end is no more present, the send call will fail.
Predicting the exact scenario is difficult with the mentioned info, but I believe, this should be the reason for you problem.
Maybe the 40 bytes fits into the pipe buffer, and the 40000 bytes doesn't?
Edit:
The sending process is sent a SIGPIPE signal when you try to write to a closed pipe. I don't know exactly when the signal is sent, or what effect the pipe buffer has on this. You may be able to recover by trapping the signal with the sigaction call.
When peer close, you just do not know whether it just stop sending or both sending and receiving.Because TCP allows this, btw, you should know the difference between close and shutdown.
If peer both stop sending and receiving, first you send some bytes, it will succeed. But the peer kernel will send you RST. So subsequently you send some bytes, your kernel will send you SIGPIPE signal, if you catch or ignore this signal, when your send returns, you just get Broken pipe error, or if you don't , the default behavior of your program is crashing.
Session timeout settings may be the reason of broken pipe.
For example: Server session timeout 3 hours and load balancer has 1 hour.
Load balancer blocks after 1 hour, but server keeps sending response. In this case, one end of the pipe is broken.
But it can be also a user behavior. User closes the page during download etc.
A situation I have under Windows XP (SP3) has been driving me nuts, and I'm reaching the end of my tether, so maybe someone can provide some inspiration.
I have a C++ networking program (non-GUI). This program is built to compile and run under Windows, MacOS/X, and Linux, so it uses select() and non-blocking I/O as the basis for its event loop.
In addition to its networking duties, this program needs to read text commands from stdin, and exit gracefully when stdin is closed. Under Linux and MacOS/X, that's easy enough -- I just include STDIN_FILENO in my read fd_set to select(), and select() returns when stdin is closed. I check to see that FD_ISSET(STDIN_FILENO, &readSet) is true, try to read some data from stdin, recv() returns 0/EOF, and so I exit the process.
Under Windows, on the other hand, you can't select on STDIN_FILE_HANDLE, because it's not a real socket. You can't do non-blocking reads on STDIN_FILE_HANDLE, either. That means there is no way to read stdin from the main thread, since ReadFile() might block indefinitely, causing the main thread to stop serving its network function.
No problem, says I, I'll just spawn a thread to handle stdin for me. This thread will run in an infinite loop, blocking in ReadFile(stdinHandle), and whenever ReadFile() returns data, the stdin-thread will write that data to a TCP socket. That socket's connection's other end will be select()'d on by the main thread, so the main thread will see the stdin data coming in over the connection, and handle "stdin" the same way it would under any other OS. And if ReadFile() returns false to indicate that stdin has closed, the stdin-thread just closes its end of the socket-pair so that the main thread will be notified via select(), as described above.
Of course, Windows doesn't have a nice socketpair() function, so I had to roll my own using listen(), connect(), and accept() (as seen in the CreateConnectedSocketPair() function here. But I did that, and it seems to work, in general.
The problem is that it doesn't work 100%. In particular, if stdin is closed within a few hundred milliseconds of when the program starts up, about half the time the main thread doesn't get any notification that the stdin-end of the socket-pair has been closed. What I mean by that is, I can see (by my printf()-debugging) that the stdin-thread has called closesocket() on its socket, and I can see that the main thread is select()-ing on the associated socket (i.e. the other end of the socket-pair), but select() never returns as it should... and if it does return, due to some other socket selecting ready-for-whatever, FD_ISSET(main_thread_socket_for_socket_pair, &readSet) returns 0, as if the connection wasn't closed.
At this point, the only hypothesis I have is that there is a bug in Windows' select() implementation that causes the main thread's select() not to notice that the other end of the socket-pair has closed by the stdin-thread. Is there another explanation? (Note that this problem has been reported under Windows 7 as well, although I haven't looked at it personally on that platform)
Just for the record, this problem turned out to be a different issue entirely, unrelated to threading, Windows, or stdin. The actual problem was an inter-process deadlock, where the parent process was blocked, waiting for the child processes to quit, but sometimes the child processes would be simultaneously blocked, waiting on the parent to supply them with some data, and so nothing would move forward.
Apologies to all for wasting your time on a red herring; if there's a standard way to close this case as unwarranted, let me know and I'll do it.
-Jeremy
Is it possible you have a race condition? Eg. Do you ensure that the CreateConnectedSocketPair() function has definitely returned before the stdin-thread has a chance to try closing its socket?
I am studying in your code. In the CreateConnectedSocketPair(), socket1 is used for listen(), and newfd is used for send/recv data. So, why does "socket1 = newfd"? How to close the listenfd then?
Not a solution, but as a workaround, couldn't you send some magic "stdin has closed" message across the TCP socket and have your receiving end disconnect its socket when it sees that and run whatever 'stdin has closed' handler?
Honestly your code is too long and I don't have time right now to spend on it.
Most likely the problem is in some cases closing the socket doesn't cause a graceful (FIN) shutdown.
Checking for exceptions returning from your select may catch the remainder of cases. There is also the (slim) possibility that no notification is actually being sent to the socket that the other end has closed. In that case, there is no way other than timeouts or "keep alive"/ping messages between the endpoints to know that the socket has closed.
If you want to figure out exactly what is happening, break out wireshark and look for FINs and RSTs (and the absence of anything). If you see the proper FIN sequence going across when your socket is closed, then the problem must be in your code. if you see RST, it may be caught by exceptions, and if you don't see anything you'll need to devise a way in your protocol to 'ping' each side of the connection to make sure they are still alive, or set a sufficiently short timeout for more data.
Rather than chasing perceived bugs in select(), I'm going to address your original fallacy that drove you away from simple, reliable, single-threaded design.
You said "You can't do non-blocking reads on STDIN_FILE_HANDLE, either. That means there is no way to read stdin from the main thread, since ReadFile() might block indefinitely" but this simply isn't the whole story. Look at ReadConsoleInput, WSAEventSelect, and WaitForMultipleObjects. The stdin handle will be signalled only when there is input and ReadConsoleInput will return immediately (pretty much the same idea behind select() in Unix).
Or, use ReadFileEx and WaitForMultipleObjectsEx to have the console reads fire off an APC (which isn't all that asynchronous, it runs on the main thread and only during WaitForMultipleObjectsEx or another explicit wait function).
If you want to stick with using a second thread to get async I/O on stdin, then you might try closing the handle being passed to select instead of doing a socket shutdown (via closesocket on the other end). In my experience select() tends to return really quickly when one of the fds it is waiting on gets closed.
Or, maybe your problem is the other way around. The select docs say "For connection-oriented sockets, readability can also indicate that a request to close the socket has been received from the peer". Typically you'd send that "request to close the socket" by calling shutdown(), not closesocket().
I have a small server program that accepts connections on a TCP or local UNIX socket, reads a simple command and (depending on the command) sends a reply.
The problem is that the client may have no interest in the answer and sometimes exits early. So writing to that socket will cause a SIGPIPE and make my server crash.
What's the best practice to prevent the crash here? Is there a way to check if the other side of the line is still reading? (select() doesn't seem to work here as it always says the socket is writable). Or should I just catch the SIGPIPE with a handler and ignore it?
You generally want to ignore the SIGPIPE and handle the error directly in your code. This is because signal handlers in C have many restrictions on what they can do.
The most portable way to do this is to set the SIGPIPE handler to SIG_IGN. This will prevent any socket or pipe write from causing a SIGPIPE signal.
To ignore the SIGPIPE signal, use the following code:
signal(SIGPIPE, SIG_IGN);
If you're using the send() call, another option is to use the MSG_NOSIGNAL option, which will turn the SIGPIPE behavior off on a per call basis. Note that not all operating systems support the MSG_NOSIGNAL flag.
Lastly, you may also want to consider the SO_SIGNOPIPE socket flag that can be set with setsockopt() on some operating systems. This will prevent SIGPIPE from being caused by writes just to the sockets it is set on.
Another method is to change the socket so it never generates SIGPIPE on write(). This is more convenient in libraries, where you might not want a global signal handler for SIGPIPE.
On most BSD-based (MacOS, FreeBSD...) systems, (assuming you are using C/C++), you can do this with:
int set = 1;
setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, (void *)&set, sizeof(int));
With this in effect, instead of the SIGPIPE signal being generated, EPIPE will be returned.
I'm super late to the party, but SO_NOSIGPIPE isn't portable, and might not work on your system (it seems to be a BSD thing).
A nice alternative if you're on, say, a Linux system without SO_NOSIGPIPE would be to set the MSG_NOSIGNAL flag on your send(2) call.
Example replacing write(...) by send(...,MSG_NOSIGNAL) (see nobar's comment)
char buf[888];
//write( sockfd, buf, sizeof(buf) );
send( sockfd, buf, sizeof(buf), MSG_NOSIGNAL );
In this post I described possible solution for Solaris case when neither SO_NOSIGPIPE nor MSG_NOSIGNAL is available.
Instead, we have to temporarily suppress SIGPIPE in the current thread that executes library code. Here's how to do this: to suppress SIGPIPE we first check if it is pending. If it does, this means that it is blocked in this thread, and we have to do nothing. If the library generates additional SIGPIPE, it will be merged with the pending one, and that's a no-op. If SIGPIPE is not pending then we block it in this thread, and also check whether it was already blocked. Then we are free to execute our writes. When we are to restore SIGPIPE to its original state, we do the following: if SIGPIPE was pending originally, we do nothing. Otherwise we check if it is pending now. If it does (which means that out actions have generated one or more SIGPIPEs), then we wait for it in this thread, thus clearing its pending status (to do this we use sigtimedwait() with zero timeout; this is to avoid blocking in a scenario where malicious user sent SIGPIPE manually to a whole process: in this case we will see it pending, but other thread may handle it before we had a change to wait for it). After clearing pending status we unblock SIGPIPE in this thread, but only if it wasn't blocked originally.
Example code at https://github.com/kroki/XProbes/blob/1447f3d93b6dbf273919af15e59f35cca58fcc23/src/libxprobes.c#L156
Handle SIGPIPE Locally
It's usually best to handle the error locally rather than in a global signal event handler since locally you will have more context as to what's going on and what recourse to take.
I have a communication layer in one of my apps that allows my app to communicate with an external accessory. When a write error occurs I throw and exception in the communication layer and let it bubble up to a try catch block to handle it there.
Code:
The code to ignore a SIGPIPE signal so that you can handle it locally is:
// We expect write failures to occur but we want to handle them where
// the error occurs rather than in a SIGPIPE handler.
signal(SIGPIPE, SIG_IGN);
This code will prevent the SIGPIPE signal from being raised, but you will get a read / write error when trying to use the socket, so you will need to check for that.
You cannot prevent the process on the far end of a pipe from exiting, and if it exits before you've finished writing, you will get a SIGPIPE signal. If you SIG_IGN the signal, then your write will return with an error - and you need to note and react to that error. Just catching and ignoring the signal in a handler is not a good idea -- you must note that the pipe is now defunct and modify the program's behaviour so it does not write to the pipe again (because the signal will be generated again, and ignored again, and you'll try again, and the whole process could go on for a long time and waste a lot of CPU power).
Or should I just catch the SIGPIPE with a handler and ignore it?
I believe that is right on. You want to know when the other end has closed their descriptor and that's what SIGPIPE tells you.
Sam
What's the best practice to prevent the crash here?
Either disable sigpipes as per everybody, or catch and ignore the error.
Is there a way to check if the other side of the line is still reading?
Yes, use select().
select() doesn't seem to work here as it always says the socket is writable.
You need to select on the read bits. You can probably ignore the write bits.
When the far end closes its file handle, select will tell you that there is data ready to read. When you go and read that, you will get back 0 bytes, which is how the OS tells you that the file handle has been closed.
The only time you can't ignore the write bits is if you are sending large volumes, and there is a risk of the other end getting backlogged, which can cause your buffers to fill. If that happens, then trying to write to the file handle can cause your program/thread to block or fail. Testing select before writing will protect you from that, but it doesn't guarantee that the other end is healthy or that your data is going to arrive.
Note that you can get a sigpipe from close(), as well as when you write.
Close flushes any buffered data. If the other end has already been closed, then close will fail, and you will receive a sigpipe.
If you are using buffered TCPIP, then a successful write just means your data has been queued to send, it doesn't mean it has been sent. Until you successfully call close, you don't know that your data has been sent.
Sigpipe tells you something has gone wrong, it doesn't tell you what, or what you should do about it.
Under a modern POSIX system (i.e. Linux), you can use the sigprocmask() function.
#include <signal.h>
void block_signal(int signal_to_block /* i.e. SIGPIPE */ )
{
sigset_t set;
sigset_t old_state;
// get the current state
//
sigprocmask(SIG_BLOCK, NULL, &old_state);
// add signal_to_block to that existing state
//
set = old_state;
sigaddset(&set, signal_to_block);
// block that signal also
//
sigprocmask(SIG_BLOCK, &set, NULL);
// ... deal with old_state if required ...
}
If you want to restore the previous state later, make sure to save the old_state somewhere safe. If you call that function multiple times, you need to either use a stack or only save the first or last old_state... or maybe have a function which removes a specific blocked signal.
For more info read the man page.
Linux manual said:
EPIPE The local end has been shut down on a connection oriented
socket. In this case the process will also receive a SIGPIPE
unless MSG_NOSIGNAL is set.
But for Ubuntu 12.04 it isn't right. I wrote a test for that case and I always receive EPIPE withot SIGPIPE. SIGPIPE is genereated if I try to write to the same broken socket second time. So you don't need to ignore SIGPIPE if this signal happens it means logic error in your program.