I'm programing a small c linux local server. I've read a lot of documentation about threads, select function, nonblocking options, etc.
But I can't find any documentation about how to deal with client-side failures.
More specifically, working with threads and blocking sockets (not the best idea, I know, but I'm just running some tests), what happens when the client connection goes too slow? or what happens when the client doesn't close the connection properly (or not closing it at all)? Will my socket remain blocked? or the thread will never finish?
How to deal with this situations?
When using blocking sockets you have a couple of options.
One is to have one thread per client such that when you wait for information it doesn't matter how long you block. Note that when a connection is closed, the blocked operation will terminate. You can read a much more detailed description of this here.
An alternative to multiple threads is to use select. This allows you to wait on multiple file descriptors until some subset of the file descriptors are ready in the sense that they will not block. So basically instead of blocking on a single file descriptor during a read or write, you instead block on select and then you know you won't later block on the read/write.
Finally, you can use the asynchronous I/O functions, aio_read and aio_write which will perform the read/write asynchronously from the calling thread of execution.
Typically, sockets have some timeout value which can be controlled by the client. If a connection runs too slowly, or the connection dies for some reason (e.g. poor internet connectivity), the socket operation might continue to block until the timeout expires. With Linux sockets, when the timeout expires, you'll get an ETIMEDOUT errno that you can then handle later.
Typical values for the timeout are on the order of 60-300 seconds, but you can set them lower if you want to know about timeouts sooner. You can also set it to infinite, but this isn't recommended if you're using direct blocking calls since you could hang your thread forever.
On Linux (and any other system using the BSD socket API), you may change the socket timeouts using
struct timeval timeout;
timeout.tv_sec = 60;
timeout.tv_usec = 0;
setsockopt(socket, SOL_SOCK, SO_RCVTIMEO, &timeout, sizeof(struct timeval));
Related
I am coding a game-server that allows up to 1100 concurrent connections using thread-per-connection approach. Every time a login packet is read from the client socket I want to be able to give it 5 seconds to connect, otherwise gracefully the connection and release the thread to the pool.
I know about alarm() for sending the process a SIGALRM, but which thread receives the signal is undefined behavior. I also tried the setitimer function, but it also sends the signal to the process. Blocking the signal in all threads but ours is impossible because I need to get the signals in all 5 threads.
Is there any way of doing this without changing the entire server architecture?
Note: This is not a personal project, so changing the thread-per-connection model is not an option, please consider these answers out-of-topic.
Threads and signals don't mix well, for the reasons you found out -- it's indeterminate which thread will receive the signal.
A better way to get a timeout within a thread is to set the socket to non-blocking mode and then run a while-loop around select() and recv(). Use the timeout argument to select() to ensure that select() will wake up at the end of your 5-second deadline, pass your socket in as part of the read-fd_set argument, and keep in mind that if the connection is TCP, the data from your socket may arrive in multiple small chunks (hence the while-loop, to collect all of them into a buffer).
I've seen a few write-ups comparing select() with poll() or epoll(), and I've seen many guides discussing the actual usage of select() with multiple sockets.
However, what I can't seem to find is a comparison to a non-blocking recv() call without select(). In the event of only having 1 socket to read from and 1 socket to write to, is there any justification for using the select() call? The recv() method can be setup to not block and return an error (WSAEWOULDBLOCK) when there is no data available, so why bother to call select() when you have no other sockets to examine? Is the non-blocking recv() call much slower?
You wouldn't want a non-blocking call to recv without some other means for waiting for data on the socket as you poll infinitely eating up cpu time.
If you have no other sockets to examine and nothing else to do in the same thread, a blocking call to read is likely to be the most efficient solution. Although in such a situation, considering the efficiency of this is like to be premature optimisation.
These kinds of considerations only tend to come into play as the socket count increases.
Nonblocking calls are only faster in the context of handling multiple sockets on a single thread.
If there is no data available, and you use non-blocking IO, recv() will return immediately.
Then what should the program do ? You would need to call recv() in a loop until data becomes available - this just uses CPU for pretty much no reason.
Spinning on recv() and burning CPU in that manner is very undesirable; you'd rather want the process to wait until data becomes available and get woken up; that's what select()/poll() and similar does.
And, sleep() in the loop in order to not burn CPU is not a good solution either. You'd introduce high latency in the processing as the program will not be able to process data as soon as the data is available.
select() and friends let you design the workflow in such a way that slowness of one socket does not impede the speed at which you can serve another. Imagine that data arrives fast from the receiving socket and you want to accept it as fast as possible and store in memory buffers. But the sending socket is slow. When you've filled up the sending buffers of the OS and send() gave you EWOULDBLOCK, you can issue select() to wait on both receiving and sending sockets. select() will fall through if either new data on the receiving socket arrived, or some buffers are freed and you can write more data to the sending socket, whichever happens first.
Of course a more realistic use case for select() is when you have multiple sockets to read from and/or to write to, or when you must pass the data between your two sockets in both directions.
In fact, select() tells you when the next read or write operation on a socket is known to succeed, so if you only try to read and write when select allows you, your program will almost work even if you didn't make the sockets non-blocking! It is still unwise to do, because there exist edge cases when the next operation still may block despite select() reported that the socket as "ready".
On the other hand, making the sockets non-blocking and not using select() is almost never advisable because of the reason explained by #Troy.
I'm new to linux programming and not entirely familiar with all the synchronization facilities so I'd like to ask more knowledgeable people how they might go about solving this problem.
I have a single thread that I would like to run through a loop. The stopping point in the loop will be a read operation on a socket. I want the read operation to block for some period of time and then timeout. However, I need a way unblock the thread from the read, if some event needs attention. The "event" could be any one of a number of different things so I need some way to tell the thread what cause the read to unblock.
I know that you can unblock a blocked read with a signal but I'm not sure how that's done.
See the select() system call.
This is especially useful for waiting for multiple file channels.
You can set timeout of socket operation. Example:
struct timeval timeout;
timeout.tv_sec = TIMEOUT_SEC;
timeout.tv_usec = TIMEOUT_MSEC;
setsockopt(sock_fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout));
/* now receive msg */
recvmsg(sock_fd, &msg, 0);
When you want to make your socket blocking, do:
timeout.tv_sec = 0;
timeout.tv_usec = 0;
setsockopt(sock_fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout));
epoll seems to be the way to go:
The epoll API performs a similar task to poll(2): monitoring multiple
file descriptors to see if I/O is possible on any of them. The epoll
API can be used either as an edge-triggered or a level-triggered inter‐
face and scales well to large numbers of watched file descriptors. The
following system calls are provided to create and manage an epoll
instance:
man epoll for more info. You might want to see "Example for Suggested Usage" section on manual.
See also epoll vs select
Sounds like you want to use select() as others have mentioned, but you also want a way to interrupt it when a "message" of some sort is available. A typical way of interrupting a select() is to use the self pipe trick. Basically you create a pipe() and also select() on the read file descriptor of the pipe. When a message arrives in the queue maintained by your program, write a byte to the pipe. This will cause your select call to return and you'll be able to check to see if your pipe is ready for reading. If it is then you know you have a message to process (whatever that is in your context), so you process it and then go back to select(). Better yet, you could have your pipe actually be your message queue. If you just use the pipe as a way to signal that messages are on your queue, make sure you actually read() the bytes out of your pipe each time through, or it will fill up eventually and block you from writing more notifications to it.
Although, as others have mentioned, why not just have one thread service your queue and do your writes to the socket, while another thread does the reads? Probably a lot simpler.
Perhaps these two libraries may be of use to you:
libev
libuv
They both use the event-driven paradigm on one or more threads (if so desired). Of course, you can implement your own event-driven framework using already mentioned APIs and conditional variables, but that might be more work than necessary.
My application has ONLY 1 Unix TCP socket that it uses to recv() and send(). The socket is non-blocking. Given this, is there an advantage in doing a select() before a send()/recv()?
If the underlying TCP pipe is not ready for an I/O, the send()/recv() should immediately return with an EWOULDBLOCK or EAGAIN. So, what's the point of doing a select()? Seems like, it might only cause an additional system call overhead in this case. Am I missing anything?
EDIT: Forgot to mention: The application is single-threaded.
If your socket is non-blocking, then you need select (or preferably poll, which does not have the broken FD_SETSIZE limit and associated dangers) to block for you in place of the blocking that would be taking place (if the socket were not non-blocking) in send and recv. Otherwise you will spin, using 100% cpu time to do-nothing. In most cases, you could just as easily make the socket blocking and do away with select/poll. However, there is one interesting case to consider: blocking IO could deadlock if your program is blocked in send and the program at the other end of the socket is also blocked in send (or the opposite). With non-blocking IO and select/poll, you naturally detect this situation and process the pending input when writing your output is not possible.
You could just do recv() in a loop, but then you'll be consuming a lot of CPU time.
Better to wait on select() to avoid the extra CPU overhead. If you need to be doing background tasks, add a timeout to select() so you can wake periodically, even with no network traffic.
If your application is letency sensitive then in may be justified to spin in a tight recv() loop without select() and give it a dedicated CPU (otherwise scheduler will punish it and you end up having massive latency). If your app cannot afford it but still does gie a thread to serve this socket then just make the socket blocking on read side and let scheduler wake your thread up when data is available. On the sending side again depends on what you need, either make the socket blocking or spin.
Only if your application is single threaded and the logic is "receive-process-reply" you absolutely need a non blocking read/write socket, selector, and a write queue, so that you receive when data is there, process, pit response to the queue, register for writability, flush the queue to the socket when writable, unregister from writability. readability is to be registered for all the time.
I am writing a client that receives UDP datagrams from a single sender. All IO will be done in a single thread. Generally, there will either be no data, or a 30 MBit/s stream. My primary concern is in keeping latency as low as possible.
The plan is to block, waiting for data, in a loop with a short-ish timeout, so that the IO thread can be responsive to shutdown requests, etc.
I am inclined to use a blocking socket, set a timeout on it, and do a recvfrom() call. However, this seems to be much less common than a select()/poll() and recvfrom() combination on a nonblocking socket.
Given that I am only working with a single socket, it seems that the nonblocking approach is needlessly complicated. Am I missing something else? Is there a reason to prefer nonblocking sockets in this particular case?
If you have a dedicated thread for handling the socket then asynchronous I/O, select etc are useless. What you want is simply recvfrom(2) and handle the data as quickly as possible.
Any fancy mechanisms (epoll, libaio, etc.) won't help you get more speed out of your application.
With only a few peers, (and 'one' is surely in this set:), a thread with a blocking socket should be fine. The code is easier to write since state can be maintained in the dedicated thread - no need for the state-machines that are usually required with a non-blocking system.
Short timeout - do you need this? Do you shutdown this subsystem before app close? If not, could you just let it be killed by OS?
If you have to shut down the thread system, you could set some 'terminate' flag and send yourself a UDP message to unblock the thread so it realises it has to die.
Rgds,
Martin