When a socket is signalled as being OK to write by a call to select(), how can I know how much data I can send without blocking? (In the case of full send buffers etc.)
Does inclusion in the set returned by select() signify that the socket is ready for at least one byte of data, and will send() then return a short count of written bytes?
Or will it block when I call send() with a len parameter that is bigger than the available buffer space? If so, how do I know the maximum amount?
I'm using regular C sockets on Linux.
The send call should not block on the first call, and should send at least one byte on the first call -- assuming you are using a stream protocol and assuming it's not interrupted by a signal, etc. However, there are really only two ways to figure out how much data you can send:
Call select after every call to send to see if more data can be sent.
Put the socket in non-blocking mode, and call send until it gives an EAGAIN or EWOULDBLOCK error.
The second option is preferred. (The third option is to do it in a different thread and simply let the thread block, which is also a good solution. In the past, threading implementations weren't as mature so non-blocking mode was seen as necessary for high-performance servers.)
You cannot know. You have to sent the socket to be non-blocking, and then pay attention to the return value that tells you how much it has written.
Related
Two cases are well-documented in the man pages for non-blocking sockets:
If send() returns the same length as the transfer buffer, the entire transfer finished successfully, and the socket may or may not be in a state of returning EAGAIN/EWOULDBLOCK the next call with >0 bytes to transfer.
If send() returns -1 and errno is EAGAIN/EWOULDBLOCK, none of the transfer finished, and the program needs to wait until the socket is ready for more data (EPOLLOUT in the epoll case).
What's not documented for nonblocking sockets is:
If send() returns a positive value smaller than the buffer size.
Is it safe to assume that the send() would return EAGAIN/EWOULDBLOCK on even one more byte of data? Or should a non-blocking program try to send() one more time to get a conclusive EAGAIN/EWOULDBLOCK? I'm worried about putting an EPOLLOUT watcher on the socket if it's not actually in a "would block" state to respond to it coming out of.
Obviously, the latter strategy (trying again to get something conclusive) has well-defined behavior, but it's more verbose and puts a hit on performance.
A call to send has three possible outcomes:
There is at least one byte available in the send buffer →send succeeds and returns the number of bytes accepted (possibly fewer than you asked for).
The send buffer is completely full at the time you call send.
→if the socket is blocking, send blocks
→if the socket is non-blocking, send fails with EWOULDBLOCK/EAGAIN
An error occurred (e.g. user pulled network cable, connection reset by peer) →send fails with another error
If the number of bytes accepted by send is smaller than the amount you asked for, then this consequently means that the send buffer is now completely full. However, this is purely circumstantial and non-authorative in respect of any future calls to send.
The information returned by send is merely a "snapshot" of the current state at the time you called send. By the time send has returned or by the time you call send again, this information may already be outdated. The network card might put a datagram on the wire while your program is inside send, or a nanosecond later, or at any other time -- there is no way of knowing. You'll know when the next call succeeds (or when it doesn't).
In other words, this does not imply that the next call to send will return EWOULDBLOCK/EAGAIN (or would block if the socket wasn't non-blocking). Trying until what you called "getting a conclusive EWOULDBLOCK" is the correct thing to do.
If send() returns the same length as the transfer buffer, the entire transfer finished successfully, and the socket may or may not be in a blocking state.
No. The socket remains in the mode it was in: in this case, non-blocking mode, assumed below throughout.
If send() returns -1 and errno is EAGAIN/EWOULDBLOCK, none of the transfer finished, and the program needs to wait until the socket is isn't blocking anymore.
Until the send buffer isn't full any more. The socket remains in non-blocking mode.
If send() returns a positive value smaller than the buffer size.
There was only that much room in the socket send buffer.
Is it safe to assume that the send() would block on even one more byte of data?
It isn't 'safe' to 'assume [it] would block' at all. It won't. It's in non-blocking mode. EWOULDBLOCK means it would have blocked in blocking mode.
Or should a non-blocking program try to send() one more time to get a conclusive EAGAIN/EWOULDBLOCK?
That's up to you. The API works whichever you decide.
I'm worried about putting an EPOLLOUT watcher on the socket if it's not actually blocking on that.
It isn't 'blocking on that'. It isn't blocking on anything. It's in non-blocking mode. The send buffer got filled at that instant. It might be completely empty a moment later.
I don't see what you're worried about. If you have pending data and the last write didn't send it all, select for writability, and write when you get it. If such a write sends everything, don't select for writability next time.
Sockets are usually writable, unless their send buffer is full, so don't select for writability all the time, as you just get a spin loop.
I've seen a few write-ups comparing select() with poll() or epoll(), and I've seen many guides discussing the actual usage of select() with multiple sockets.
However, what I can't seem to find is a comparison to a non-blocking recv() call without select(). In the event of only having 1 socket to read from and 1 socket to write to, is there any justification for using the select() call? The recv() method can be setup to not block and return an error (WSAEWOULDBLOCK) when there is no data available, so why bother to call select() when you have no other sockets to examine? Is the non-blocking recv() call much slower?
You wouldn't want a non-blocking call to recv without some other means for waiting for data on the socket as you poll infinitely eating up cpu time.
If you have no other sockets to examine and nothing else to do in the same thread, a blocking call to read is likely to be the most efficient solution. Although in such a situation, considering the efficiency of this is like to be premature optimisation.
These kinds of considerations only tend to come into play as the socket count increases.
Nonblocking calls are only faster in the context of handling multiple sockets on a single thread.
If there is no data available, and you use non-blocking IO, recv() will return immediately.
Then what should the program do ? You would need to call recv() in a loop until data becomes available - this just uses CPU for pretty much no reason.
Spinning on recv() and burning CPU in that manner is very undesirable; you'd rather want the process to wait until data becomes available and get woken up; that's what select()/poll() and similar does.
And, sleep() in the loop in order to not burn CPU is not a good solution either. You'd introduce high latency in the processing as the program will not be able to process data as soon as the data is available.
select() and friends let you design the workflow in such a way that slowness of one socket does not impede the speed at which you can serve another. Imagine that data arrives fast from the receiving socket and you want to accept it as fast as possible and store in memory buffers. But the sending socket is slow. When you've filled up the sending buffers of the OS and send() gave you EWOULDBLOCK, you can issue select() to wait on both receiving and sending sockets. select() will fall through if either new data on the receiving socket arrived, or some buffers are freed and you can write more data to the sending socket, whichever happens first.
Of course a more realistic use case for select() is when you have multiple sockets to read from and/or to write to, or when you must pass the data between your two sockets in both directions.
In fact, select() tells you when the next read or write operation on a socket is known to succeed, so if you only try to read and write when select allows you, your program will almost work even if you didn't make the sockets non-blocking! It is still unwise to do, because there exist edge cases when the next operation still may block despite select() reported that the socket as "ready".
On the other hand, making the sockets non-blocking and not using select() is almost never advisable because of the reason explained by #Troy.
I have to write a chat client-server for a class using unix sockets (without O_NONBLOCK) and select for asynchronous I/O on them. At the moment, on the server, i read 1024 bytes from the client, and directly handle it.
For example, in case of a message, i will receive a command formatted as MSG <msg> (representing a client sending a message), i will go through all the sockets of the connected clients and write the message on them.
This approach actually works but i recently found by reading the man of send that it can blocks if the socket buffer is full and is the flag O_NONBLOCK is not set on the socket.
I think this problem could happen when a client does not read for some reasons (crash, bugged etc.) and this would be critical for my server since it will basically blocks until this client read again.
So here is my question:
What is the correct approach on a potentially blocking socket to avoid send to block if the socket buffer is full?
I'm currently using select only to check if there is something to read on sockets but maybe i should use it also to see if i can write on a particular socket too? And also, can i know how many bytes i can read/write when select returns? For example, if select "tells" that i can write on this socket, how can i know how many bytes i can write at most before writing on this socket actually becomes blocking?
Thanks.
You could use setsockopt() together with SO_SNDTIMEO to set up a maximum amount of time send() will try to do its work.
See man setsockoptand man 7 socket for details.
It might be horrible. If you don't go NONBLOCK-ing mode and calling select(), which internally puts the process on sleep for specific timeout value. That means, fd will be blocked for that specific time period.
This approach actually works but i recently found by reading the man of send that it can blocks if the socket buffer is full and is the flag O_NONBLOCK is not set on the socket.
This is why you use select, but it still isn't reliable, as man select states:
Under Linux, select() may report a socket file descriptor as "ready for reading", while nevertheless a subsequent read blocks. This could for example happen when data has
arrived but upon examination has wrong checksum and is discarded. There may be other circumstances in which a file descriptor is spuriously reported as ready. Thus it may
be safer to use O_NONBLOCK on sockets that should not block.
The goal is to read data from a socket without blocking. The Linux manual page says:
The receive calls normally return any data available, up to the
requested amount, rather than waiting for receipt of the full amount
requested.
Does it mean that I don't have to pass MSG_DONTWAIT flag to recv() after polling the socket descriptor with select()/poll()/epoll()?
The behaviour of recv/read depends on the characteristics of the socket itself. If the socket is marked as non-blocking, these calls should immediately return EAGAIN/EWOULDBLOCK rather than blocking the process.
The socket can be marked as non-blocking prior to reading from it, usually via fcntl or ioctl.
What this excerpt from the manual says is that, basically, reads on both blocking and non-blocking sockets are not required to fill the whole buffer that is supplied. That is why it is important to check the result of the recv/read calls in order to know how much of the buffer contains the actual data and how much is garbage.
It is not a good idea at all to use blocking sockets in conjunction with the IO polling calls such as select/poll/epoll. Even if the polling call indicates that a particular socket is ready for reading, a blocking socket would sometimes still block.
What happens if I have one socket, s, there is no data currently available on it, it is a blocking socket, and I call recv on it from two threads at once? Will one of the threads get the data? Will both get it? Will the 2nd call to recv return with an error?
One thread will get it, and there's no way to tell which.
This doesn't seem like a reasonable design. Is there a reason why you need two threads calling recv() on the same socket?
Socket implementations should be thread-safe, so exactly one thread should get the data when it becomes available. The other call should just block.
I can't find a reference for this, but here's my understanding:
A vendor's guarantee of thread-safety may mean only that multiple threads can each safely use their own sockets; it does not guarantee atomicity across a single call, and it doesn't promise any particular allocation of the socket's data among multiple threads.
Suppose thread A calls recv() on a socket that's receiving TCP data streaming in at a high rate. If recv() needs to be an atomic call, then thread A could block all other threads from executing, because it needs to be running continuously to pull in all the data (until its buffer is full, anyway.) That wouldn't be good. Hence, I would not assume that recv() is immune to context switching.
Conversely, suppose thread A makes a blocking call to recv() on a TCP socket, and the data is coming in slowly. Hence the call to recv() returns with errno set to EAGAIN.
In either of these cases, suppose thread B calls recv() on the same socket while thread A is still receiving data. When does thread A stop getting data handed to it so that thread B can start receiving data? I don't know of a Unix implementation that will try to remember that thread A was in the middle of an operation on the socket; instead, it's up to the application (threads A and B) to negotiate their use of it.
Generally, it's best to design the app so that only one of the threads will call recv() on a single socket.
From the man page on recv
A recv() on a SOCK_STREAM socket
returns as much available information
as the size of the buffer supplied can
hold.
Lets assume you are using TCP, since it was not specified in the question. So suppose you have thread A and thread B both blocking on recv() for socket s. Once s has some data to be received it will unblock one of the threads, lets say A, and return the data. The data returned will be of some random size as far as we are concerned. Thread A inspects the data received and decides if it has a complete "message", where a message is an application level concept.
Thread A decides it does not have a complete message, so it calls recv() again. BUT in the meantime B was already blocking on the same socket, and has received the rest of the "message" that was intended for thread A. I am using intended loosely here.
Now both thread A and thread B have an incomplete message, and will, depending on how the code is written, throw the data away as invalid, or cause weird and subtle errors.
I wish I could say I didn't know this from experience.
So while recv() itself is technically thread safe, it is a bad idea to have two threads calling it simultaneously if you are using it for TCP.
As far as I know it is completely safe when you are using UDP.
I hope this helps.