I have a network daemon (poll()/accept()/fork() style) which is leaking socket file descriptors, one per client in the TIME_WAIT state.
As far as I can see I can shutdown()ing and then close()ing definitely-no-longer-needed sockets. Other sockets (for example the server socket in the client side of the fork) are just close()ed. All sockets have SO_REUSEADDR set and SO_LINGER is off. I am using _exit() to exit the program and I am using non-blocking polling socket operations so as to set a ''dying'' flag in my signal handler -- this allows me to later pick up the dying flag and free(), shutdown(), close(), which would otherwise be dangerous in a signal handler.
But still a fd leak -- What is the best way to debug this kind of problem? It would help to know which socket is loitering at exit, as there are many fds involved in the process.
Cheers!
Sockets in TIME_WAIT mode are NOT leaking -- TIME_WAIT means that the application has finished with the socket and has closed it and cleaned it up, but the kernel is still remembering the socket so as to respond properly to late/orphan/duplicate packets that might be floating around in the network. After a little while, the kernel will automatically delete the TIME_WAIT sockets, but until then, they remain as a reminder to the kernel to not reuse the port unless an app specifically asks for it with SO_REUSEADDR.
I figured this out.
Infact I had fixed the bug already by closing the cli_fd in the server side of the fork; however I did not notice the bug was fixed because i was using natstat wrongly to could open fds.
For the record, the output of netstat -n | grep TIME_WAIT | wc -l should not be used to count file descriptors for sockets which are hanging around -- this is what i was doing wrong. Use lsof or fstat instead.
Anyway - the server is no longer running out of fds under considerable load.
Cheers
Related
In TCP we say one side of the connection performs an "active close" and the other side performs a "passive close".
In terms of the Linux sockets API, how do you differentiate the active close and the passive close?
For example, suppose we have two connected Linux TCP sockets, A and P, that have exchanged information over the application-level protocol and they are both aware that it is time to close their sockets (neither expect to send or receive any more data to or from each other).
We want socket A to perform the active close, and for P to be the passive close.
There are a few things A and P could do. For example:
call shutdown(SHUT_WR)
call recv and expect to get 0 back
call close.
something else
What combination of these things and in what order should A do?... and what combination of these things and in what order should P do?
In terms of the Linux sockets API, how do you differentiate the active
close and the passive close?
The 'active' close is simply whichever side of the socket sends a FIN or RST packet first, typically by calling close().
What combination of these things and in what order should A do?... and
what combination of these things and in what order should P do?
In practice, most of this is application- and application-protocol specific. I will describe the minimum/typical requirement to answer your question, but your mileage may vary depending on what you are specifically trying to accomplish.
You may first call shutdown() on Socket A if you want to terminate communication in one direction or the other (or both) on Socket A. From your description, both programs already know they're done, perhaps due to application protocol messages, so this may not be necessary.
You must call close() on Socket A in order to close the socket and release the file descriptor.
On Socket P, you simply keep reading until recv() returns 0, and then you must call close() to close the socket and release the file descriptor.
For further reading, there are a number of good tutorials out there, and Beej's Guide to Network Programming is quite popular.
Active open is when you issue connect(2) explicitly to make a connection to a remote site. The call blocks until you get the socket opened on the other side (except if you issued O_NONBLOCK fcntl(2) call before calling connect(2).
Passive open is when you have a socket listen(2)ing on a connection and you have not yet issued an accept(2) system call. The accept(2) call normally blocks until you have a completely open connection and gives you a socket descriptor to communicate over it, or gives you inmediately a socket descriptor if the connection handshake has already finished when you issue the accept(2) syscall (this is a passive open). The limit in the number of passively open connections the kernel can accept on your behalf while you prepare yourself to make the accept(2) system call is what is called the listen(2) value.
Active close is what happens when you explicitly call shutdown(2) or close(2) system calls. As with passive open, there's nothing you can do to make a passive close (it's something that happens behind the scenes, product of other side's actions). You detect a passive close when the socket generates an end of file condition (this is, read(2) always returns 0 bytes on reading) meaning the other end has done a shutdown(2) (or close(2)) and the connection is half (or full) closed. When you explicitly shutdown(2) or close(2) your side, it's an active close.
NOTE
if the other end does an explicit close(2) and you continue writing on the socket, you'll get an error due to the impossibility of sending that data (in this case we can talk about a passive close(2) ---one that has occured without any explicit action from our side) but the other end can do a half close calling shutdown(2). This makes the tcp to send a FIN segment only and conserves the socket descriptor to allow the thread to receive any pending data in transit or not yet sent. Only when it receives and acknowledges the other end's FIN segment will it signal you that no more data remains in transit.
I'm programing a small c linux local server. I've read a lot of documentation about threads, select function, nonblocking options, etc.
But I can't find any documentation about how to deal with client-side failures.
More specifically, working with threads and blocking sockets (not the best idea, I know, but I'm just running some tests), what happens when the client connection goes too slow? or what happens when the client doesn't close the connection properly (or not closing it at all)? Will my socket remain blocked? or the thread will never finish?
How to deal with this situations?
When using blocking sockets you have a couple of options.
One is to have one thread per client such that when you wait for information it doesn't matter how long you block. Note that when a connection is closed, the blocked operation will terminate. You can read a much more detailed description of this here.
An alternative to multiple threads is to use select. This allows you to wait on multiple file descriptors until some subset of the file descriptors are ready in the sense that they will not block. So basically instead of blocking on a single file descriptor during a read or write, you instead block on select and then you know you won't later block on the read/write.
Finally, you can use the asynchronous I/O functions, aio_read and aio_write which will perform the read/write asynchronously from the calling thread of execution.
Typically, sockets have some timeout value which can be controlled by the client. If a connection runs too slowly, or the connection dies for some reason (e.g. poor internet connectivity), the socket operation might continue to block until the timeout expires. With Linux sockets, when the timeout expires, you'll get an ETIMEDOUT errno that you can then handle later.
Typical values for the timeout are on the order of 60-300 seconds, but you can set them lower if you want to know about timeouts sooner. You can also set it to infinite, but this isn't recommended if you're using direct blocking calls since you could hang your thread forever.
On Linux (and any other system using the BSD socket API), you may change the socket timeouts using
struct timeval timeout;
timeout.tv_sec = 60;
timeout.tv_usec = 0;
setsockopt(socket, SOL_SOCK, SO_RCVTIMEO, &timeout, sizeof(struct timeval));
I am writing a client that receives UDP datagrams from a single sender. All IO will be done in a single thread. Generally, there will either be no data, or a 30 MBit/s stream. My primary concern is in keeping latency as low as possible.
The plan is to block, waiting for data, in a loop with a short-ish timeout, so that the IO thread can be responsive to shutdown requests, etc.
I am inclined to use a blocking socket, set a timeout on it, and do a recvfrom() call. However, this seems to be much less common than a select()/poll() and recvfrom() combination on a nonblocking socket.
Given that I am only working with a single socket, it seems that the nonblocking approach is needlessly complicated. Am I missing something else? Is there a reason to prefer nonblocking sockets in this particular case?
If you have a dedicated thread for handling the socket then asynchronous I/O, select etc are useless. What you want is simply recvfrom(2) and handle the data as quickly as possible.
Any fancy mechanisms (epoll, libaio, etc.) won't help you get more speed out of your application.
With only a few peers, (and 'one' is surely in this set:), a thread with a blocking socket should be fine. The code is easier to write since state can be maintained in the dedicated thread - no need for the state-machines that are usually required with a non-blocking system.
Short timeout - do you need this? Do you shutdown this subsystem before app close? If not, could you just let it be killed by OS?
If you have to shut down the thread system, you could set some 'terminate' flag and send yourself a UDP message to unblock the thread so it realises it has to die.
Rgds,
Martin
I'm working with some code that needs to be safe against killing the caller due to SIGPIPE, but the only socket writes it's performing are going to datagram sockets (both UDP and Unix domain datagram sockets). Do I need to worry about SIGPIPE? I'm using connect on the socket, but preliminary testing (on Linux) showed that I just get ECONNREFUSED on send if there's nobody listening on the Unix domain socket. Not sure what happens with UDP.
I can wrap the whole thing in hacks to get rid of SIGPIPE, but if it's a non-issue I'd rather save the overhead and keep the code complexity down.
The answer is in the specification for send:
[EPIPE] The socket is shut down for writing, or the socket is connection-mode and is no longer connected. In the latter case, and if the socket is of type SOCK_STREAM or SOCK_SEQPACKET and the MSG_NOSIGNAL flag is not set, the SIGPIPE signal is generated to the calling thread.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/send.html
Thus, no, writes to datagram sockets do not generate SIGPIPE or an EPIPE error.
The open group is one thing, and Apple is another.
It is definitely possible to get a SIGPIPE on iOS when writing to a dead UDP socket, as some of my crash logs revealed lately.
iOS tends to close UDP sockets while the app is in the background, writing to these sockets can pop a SIGPIPE.
From my crash log (courtesy of testflightapp):
Exception Latest Victim Occurrences
SIGPIPE
2 libsystem_c.dylib 0x32df47ec _sigtramp + 48
3 instant talk 0x0005b10e -[IPRSNetDatagramSocket send:size:to:] (iprs_iphone_net.m:671)...
Don't recall this happening on Linux, Solaris or Windows - though I never tried to close a socket and then write to it.
According to man 2 write on my Debian box,
EPIPE: fd is connected to a pipe or socket whose reading end is closed. When this happens the writing process will also receive a SIGPIPE signal. (Thus, the write
return value is seen only if the program catches, blocks or ignores this signal.)
It appears that it is possible to get SIGPIPE when writing to a socket, but it's not clear whether it can happen for UDP sockets specifically.
A situation I have under Windows XP (SP3) has been driving me nuts, and I'm reaching the end of my tether, so maybe someone can provide some inspiration.
I have a C++ networking program (non-GUI). This program is built to compile and run under Windows, MacOS/X, and Linux, so it uses select() and non-blocking I/O as the basis for its event loop.
In addition to its networking duties, this program needs to read text commands from stdin, and exit gracefully when stdin is closed. Under Linux and MacOS/X, that's easy enough -- I just include STDIN_FILENO in my read fd_set to select(), and select() returns when stdin is closed. I check to see that FD_ISSET(STDIN_FILENO, &readSet) is true, try to read some data from stdin, recv() returns 0/EOF, and so I exit the process.
Under Windows, on the other hand, you can't select on STDIN_FILE_HANDLE, because it's not a real socket. You can't do non-blocking reads on STDIN_FILE_HANDLE, either. That means there is no way to read stdin from the main thread, since ReadFile() might block indefinitely, causing the main thread to stop serving its network function.
No problem, says I, I'll just spawn a thread to handle stdin for me. This thread will run in an infinite loop, blocking in ReadFile(stdinHandle), and whenever ReadFile() returns data, the stdin-thread will write that data to a TCP socket. That socket's connection's other end will be select()'d on by the main thread, so the main thread will see the stdin data coming in over the connection, and handle "stdin" the same way it would under any other OS. And if ReadFile() returns false to indicate that stdin has closed, the stdin-thread just closes its end of the socket-pair so that the main thread will be notified via select(), as described above.
Of course, Windows doesn't have a nice socketpair() function, so I had to roll my own using listen(), connect(), and accept() (as seen in the CreateConnectedSocketPair() function here. But I did that, and it seems to work, in general.
The problem is that it doesn't work 100%. In particular, if stdin is closed within a few hundred milliseconds of when the program starts up, about half the time the main thread doesn't get any notification that the stdin-end of the socket-pair has been closed. What I mean by that is, I can see (by my printf()-debugging) that the stdin-thread has called closesocket() on its socket, and I can see that the main thread is select()-ing on the associated socket (i.e. the other end of the socket-pair), but select() never returns as it should... and if it does return, due to some other socket selecting ready-for-whatever, FD_ISSET(main_thread_socket_for_socket_pair, &readSet) returns 0, as if the connection wasn't closed.
At this point, the only hypothesis I have is that there is a bug in Windows' select() implementation that causes the main thread's select() not to notice that the other end of the socket-pair has closed by the stdin-thread. Is there another explanation? (Note that this problem has been reported under Windows 7 as well, although I haven't looked at it personally on that platform)
Just for the record, this problem turned out to be a different issue entirely, unrelated to threading, Windows, or stdin. The actual problem was an inter-process deadlock, where the parent process was blocked, waiting for the child processes to quit, but sometimes the child processes would be simultaneously blocked, waiting on the parent to supply them with some data, and so nothing would move forward.
Apologies to all for wasting your time on a red herring; if there's a standard way to close this case as unwarranted, let me know and I'll do it.
-Jeremy
Is it possible you have a race condition? Eg. Do you ensure that the CreateConnectedSocketPair() function has definitely returned before the stdin-thread has a chance to try closing its socket?
I am studying in your code. In the CreateConnectedSocketPair(), socket1 is used for listen(), and newfd is used for send/recv data. So, why does "socket1 = newfd"? How to close the listenfd then?
Not a solution, but as a workaround, couldn't you send some magic "stdin has closed" message across the TCP socket and have your receiving end disconnect its socket when it sees that and run whatever 'stdin has closed' handler?
Honestly your code is too long and I don't have time right now to spend on it.
Most likely the problem is in some cases closing the socket doesn't cause a graceful (FIN) shutdown.
Checking for exceptions returning from your select may catch the remainder of cases. There is also the (slim) possibility that no notification is actually being sent to the socket that the other end has closed. In that case, there is no way other than timeouts or "keep alive"/ping messages between the endpoints to know that the socket has closed.
If you want to figure out exactly what is happening, break out wireshark and look for FINs and RSTs (and the absence of anything). If you see the proper FIN sequence going across when your socket is closed, then the problem must be in your code. if you see RST, it may be caught by exceptions, and if you don't see anything you'll need to devise a way in your protocol to 'ping' each side of the connection to make sure they are still alive, or set a sufficiently short timeout for more data.
Rather than chasing perceived bugs in select(), I'm going to address your original fallacy that drove you away from simple, reliable, single-threaded design.
You said "You can't do non-blocking reads on STDIN_FILE_HANDLE, either. That means there is no way to read stdin from the main thread, since ReadFile() might block indefinitely" but this simply isn't the whole story. Look at ReadConsoleInput, WSAEventSelect, and WaitForMultipleObjects. The stdin handle will be signalled only when there is input and ReadConsoleInput will return immediately (pretty much the same idea behind select() in Unix).
Or, use ReadFileEx and WaitForMultipleObjectsEx to have the console reads fire off an APC (which isn't all that asynchronous, it runs on the main thread and only during WaitForMultipleObjectsEx or another explicit wait function).
If you want to stick with using a second thread to get async I/O on stdin, then you might try closing the handle being passed to select instead of doing a socket shutdown (via closesocket on the other end). In my experience select() tends to return really quickly when one of the fds it is waiting on gets closed.
Or, maybe your problem is the other way around. The select docs say "For connection-oriented sockets, readability can also indicate that a request to close the socket has been received from the peer". Typically you'd send that "request to close the socket" by calling shutdown(), not closesocket().