I'm using epoll as level triggered. According to recv(3), if recv returns 0 "no messages are available to be received and the peer has performed an orderly shutdown". Does this mean that the whole request has been received and the socket has been closed on the other side? That the socket has been closed on both ends? Or that simply the whole request has been received and the socket is awaiting a response to be written to it? Any help would be greatly appreciated.
According to What value will recv() return, it sounds like if recv returns 0, the connection is closed on the other side. If this is the case, what needs to be done on the epoll side? Does the socket need to be completely removed with a call to epoll_ctl with EPOLL_CTL_DEL and a call to close?
It means that you have already read all the data that has been sent overthe connection, and the only thing left is a TCP FIN, which could be the result of either a full close by the remote or a shutdown for output by the remote. You can't do much but close the socket at this point, unless you know the peer has only shutdown, in which case you can still write to the socket. Your application protocol determines all that.
There is no such thing as a request or response in TCP. There is just a bidirectional byte-stream.
Related
I have a TCP socket in blocking mode being used for the client side of a request/response protocol. Sometimes I am finding that if a socket was unused for a minute or two a send call succeeds and indicates all bytes sent, but the following recv returns zero, indicating a shutdown. I have seen this on both Windows and Linux clients.
The server guys tell me they always send some response before shutdown if they had received data, but they may close a socket that has not yet received anything if low on server resources.
Is what I am seeing indicative of the server having closed the connection while I was not using it, and then why does send then succeed?
What is the correct way automatically detect this such that the request is resent on a new connection in this case, but bearing in mind that if the server actually received some requests twice could have unintended effects?
//not full code (buffer management, wrapper functions, etc...)
//no special flags/options are being set, just socket then connect
sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
connect(sock, addr, addrlen);
//some time later after many requests/responses, normally if was inactive for a minute
//sending about 50 bytes for requests, never actually seen it loop, or return 0
while (more_to_send) check(send(sock, buffer, len, 0));
//the very first recv returns 0, never seen it happen part way through a response (few KB to a couple of MB)
while (response_not_complete) check(recv(sock, buffer, 4096, 0));
If you don't get an application acknowledgment of the request from the server, re-send it.
Design your transactions to be idempotent so that re-sending them doesn't cause ill-effects.
Is what I am seeing indicative of the server having closed the
connection while I was not using it
Yes.
, and then why does send then succeed?
send()'s succeeding tells you only that some (or all) of the data you passed into send() has been successfully copied into an in-kernel buffer, and that from now on it is the OS's responsibility to try to deliver those bytes to the remote peer.
In particular, it does not indicate that those bytes have actually gone across the network (yet) or been successfully received by the server.
What is the correct way automatically detect this such that the
request is resent on a new connection in this case, but bearing in
mind that if the server actually received some requests twice could
have unintended effects?
As EJP suggests, the best way would be to design your communications protocol such that sending the same request twice has no effect that is different from sending it once. One way to do that would be to add a unique ID to each message you send, and add some logic to the server such that if it receives a message with an ID that is the same as one that it has already processed, it discards the message as a duplicate.
Having the server send back an explicit response to each message (so that you can know for sure your message got through and was processed) might help, but of course then you have to start worrying about the case where your message was received and processed but then the TCP connection broke before the response could be delivered back to you, and so on.
One other thing you could do (if you're not doing it already) is to monitor the state of the TCP socket (via select(), poll(), or similar) so that your program will be immediately notified (by the socket select()-ing as ready-for-read) when the remote peer closes its end of the socket. That way you can deal with the closed TCP connection well before you try to send() a command, rather than only finding out about it afterwards, and that should be a less awkward situation to handle, since in that case there is no question about whether a command "got through" or not.
I'm using send on Linux std socket to write a packet over network. Send call, buffers data and "always" return values greater than 0. Send Pass the problem to the operating system and the lower level.
How can i stop the send call and wait for the delivery of the packet to the other endsystem? Waiting for the TCP ACK or something like that?
if (send(broker->socket, packet, sizeof(packet), 0) < sizeof(packet)){
return -3;
}
The test will like:
start to send packet,remove the ethernet,reattach.
Thanks, sorry for my bad english.
There's no real robust way to find out if the remote peer acked your data. In fact, even if the remote TCP acks your data that doesn't mean the remote process read it. The simplest method is to implement a sort of application-layer ACK where the peer sends back a byte signaling "ok, got it".
You call send and it returns almost immediately
At this point the kernel starts working, trying to push the data
You call recv which blocks
The remote side receives the data and sends some data back, acknowledging it
Your recv unblocks
At this point you can be certain the remote side received your data.
I have a AF_INET/SOCK_STREAM server written in C running on Android/Linux which looks more ore less like this:
...
for (;;) {
client = accept(...);
read(client, &message, sizeof(message));
response = process(&message);
write(client, response, sizeof(*response));
close(client);
}
As far as I know, the call to close should not terminate the connection to the client immediately, but it apparently does: The client reports "Connection Reset by Peer" before it has had a chance to read the server's response.
If I insert a delay between write() and close() the client can read the response as expected.
I got a hint that it might have to do with the SO_LINGER option, but I checked it's value and both members of struct linger (l_onoff, l_linger) have a value of zero.
Any ideas?
Stevens describes a configuration in which this can happen, but it depends on the client sending more data after the server has called close() (after the client should “know” that the connection is being closed). UNP 2nd ed s5.12.
Try tcpdumping the conversation to find out what’s really going on. If there's any possibility that a “clever” gateway (e.g. NAT) is between the two endpoints, tcpdump both ends and look for discrepancies.
Connection gets reset when you call close() on connection with data being sent. Specially for this case the sequence of shutdown() with SHUT_WR flag and then blocking read() is used.
Shutting down the writing end of the socket sends FIN and returns immediately, and the said read() blocks and returns 0 as soon as your peer replies with FIN in due turn. Basically, this is what you need in place of the delay between write() and close() you are talking about.
You do not need do anything with linger options in this case, leave it all to default.
SO_LINGER should be set (i.e. set to 1 not 0) if you want queued data to be sent before a close is effected.
SO_LINGER
Lingers on a close() if data is present. This option controls the
action taken when unsent messages
queue on a socket and close() is
performed. If SO_LINGER is set, the
system shall block the calling thread
during close() until it can transmit
the data or until the time expires. If
SO_LINGER is not specified, and
close() is issued, the system handles
the call in a way that allows the
calling thread to continue as quickly
as possible. This option takes a
linger structure, as defined in the
header, to specify the
state of the option and linger
interval.
In C, I understood that if we close a socket, it means the socket will be destroyed and can be re-used later.
How about shutdown? The description said it closes half of a duplex connection to that socket. But will that socket be destroyed like close system call?
This is explained in Beej's networking guide. shutdown is a flexible way to block communication in one or both directions. When the second parameter is SHUT_RDWR, it will block both sending and receiving (like close). However, close is the way to actually destroy a socket.
With shutdown, you will still be able to receive pending data the peer already sent (thanks to Joey Adams for noting this).
None of the existing answers tell people how shutdown and close works at the TCP protocol level, so it is worth to add this.
A standard TCP connection gets terminated by 4-way finalization:
Once a participant has no more data to send, it sends a FIN packet to the other
The other party returns an ACK for the FIN.
When the other party also finished data transfer, it sends another FIN packet
The initial participant returns an ACK and finalizes transfer.
However, there is another "emergent" way to close a TCP connection:
A participant sends an RST packet and abandons the connection
The other side receives an RST and then abandon the connection as well
In my test with Wireshark, with default socket options, shutdown sends a FIN packet to the other end but it is all it does. Until the other party send you the FIN packet you are still able to receive data. Once this happened, your Receive will get an 0 size result. So if you are the first one to shut down "send", you should close the socket once you finished receiving data.
On the other hand, if you call close whilst the connection is still active (the other side is still active and you may have unsent data in the system buffer as well), an RST packet will be sent to the other side. This is good for errors. For example, if you think the other party provided wrong data or it refused to provide data (DOS attack?), you can close the socket straight away.
My opinion of rules would be:
Consider shutdown before close when possible
If you finished receiving (0 size data received) before you decided to shutdown, close the connection after the last send (if any) finished.
If you want to close the connection normally, shutdown the connection (with SHUT_WR, and if you don't care about receiving data after this point, with SHUT_RD as well), and wait until you receive a 0 size data, and then close the socket.
In any case, if any other error occurred (timeout for example), simply close the socket.
Ideal implementations for SHUT_RD and SHUT_WR
The following haven't been tested, trust at your own risk. However, I believe this is a reasonable and practical way of doing things.
If the TCP stack receives a shutdown with SHUT_RD only, it shall mark this connection as no more data expected. Any pending and subsequent read requests (regardless whichever thread they are in) will then returned with zero sized result. However, the connection is still active and usable -- you can still receive OOB data, for example. Also, the OS will drop any data it receives for this connection. But that is all, no packages will be sent to the other side.
If the TCP stack receives a shutdown with SHUT_WR only, it shall mark this connection as no more data can be sent. All pending write requests will be finished, but subsequent write requests will fail. Furthermore, a FIN packet will be sent to another side to inform them we don't have more data to send.
There are some limitations with close() that can be avoided if one uses shutdown() instead.
close() will terminate both directions on a TCP connection. Sometimes you want to tell the other endpoint that you are finished with sending data, but still want to receive data.
close() decrements the descriptors reference count (maintained in file table entry and counts number of descriptors currently open that are referring to a file/socket) and does not close the socket/file if the descriptor is not 0. This means that if you are forking, the cleanup happens only after reference count drops to 0. With shutdown() one can initiate normal TCP close sequence ignoring the reference count.
Parameters are as follows:
int shutdown(int s, int how); // s is socket descriptor
int how can be:
SHUT_RD or 0
Further receives are disallowed
SHUT_WR or 1
Further sends are disallowed
SHUT_RDWR or 2
Further sends and receives are disallowed
This may be platform specific, I somehow doubt it, but anyway, the best explanation I've seen is here on this msdn page where they explain about shutdown, linger options, socket closure and general connection termination sequences.
In summary, use shutdown to send a shutdown sequence at the TCP level and use close to free up the resources used by the socket data structures in your process. If you haven't issued an explicit shutdown sequence by the time you call close then one is initiated for you.
I've also had success under linux using shutdown() from one pthread to force another pthread currently blocked in connect() to abort early.
Under other OSes (OSX at least), I found calling close() was enough to get connect() fail.
"shutdown() doesn't actually close the file descriptor—it just changes its usability. To free a socket descriptor, you need to use close()."1
Close
When you have finished using a socket, you can simply close its file descriptor with close; If there is still data waiting to be transmitted over the connection, normally close tries to complete this transmission. You can control this behavior using the SO_LINGER socket option to specify a timeout period; see Socket Options.
ShutDown
You can also shut down only reception or transmission on a connection by calling shutdown.
The shutdown function shuts down the connection of socket. Its argument how specifies what action to perform:
0
Stop receiving data for this socket. If further data arrives, reject it.
1
Stop trying to transmit data from this socket. Discard any data waiting to be sent. Stop looking for acknowledgement of data already sent; don’t retransmit it if it is lost.
2
Stop both reception and transmission.
The return value is 0 on success and -1 on failure.
in my test.
close will send fin packet and destroy fd immediately when socket is not shared with other processes
shutdown SHUT_RD, process can still recv data from the socket, but recv will return 0 if TCP buffer is empty.After peer send more data, recv will return data again.
shutdown SHUT_WR will send fin packet to indicate the Further sends are disallowed. the peer can recv data but it will recv 0 if its TCP buffer is empty
shutdown SHUT_RDWR (equal to use both SHUT_RD and SHUT_WR) will send rst packet if peer send more data.
linux: shutdown() causes listener thread select() to awake and produce error. shutdown(); close(); will lead to endless wait.
winsock: vice versa - shutdown() has no effect, while close() is successfully catched.
Pardon if this question has been answered but I couldn't find it.
I'm kinda confused about recv() and recvfrom(). Once the server binds the address (or accepts connection for TCP), recv() is called. Does recv() constantly check for messages that has been sent or does it wait until a message is received? If it does wait, how long is the wait time?
Not sure if I'm making sense, but if someone could enlighten me, I'd be grateful.
If no messages are available at the socket and O_NONBLOCK is not set on the socket's file descriptor, recv() shall block until a message arrives.
If no messages are available at the socket and O_NONBLOCK is set on the socket's file descriptor, recv() shall fail and set errno to [EAGAIN] or [EWOULDBLOCK].
Source: http://www.opengroup.org/onlinepubs/009695399/functions/recv.html
Note that you can implement a timeout using select() or poll(), which also lets you wait on multiple sockets at once.