I'm attempting to write a rudimentary file server that takes a filename from a client and responds by sending the data over TCP to the client. I have a working client and server application for the most part but I'm observing some odd behavior, consider the following
while ((num_read = read (file_fd, file_buffer, sizeof (file_buffer))) > 0)
{
if (num_read != write (conn_fd, article_buffer, num_read))
{
perror ("write");
goto out;
}
}
out:
close(file_fd); close(sub_fd);
file_fd is a file descriptor to the file being sent over the network, conn_fd is a file descriptor to a connect()ed TCP socket.
This seems to work for small files, but when my files get larger(megabyte+) it seems that some non-consistent amount of data at the end of the file will fail to transfer.
I suspected the immediate close() statements after write might have something to do with it so I tried a 1 second sleep() before both close() statements and my client successfully received all of the data.
Is there any better way to handle this than doing a sleep() on the server side?
A successful "write" on a socket does not mean the data has been successfully sent to the peer.
If you are on a unix deriviative, you can perform a "man 7 socket" and examine SO_LINGER" as a potential solution.
edit: Due to EJP's comment (thank you), I reread what Stevens has to say about the subject in "Unix Network Programming" of ensured delivery of all data to a peer. He says the following (in Volume 1 of Second edition, page 189):
... we see that when we close our end of the connection, depending on the function called (close or shutdown) and whether he SO_LINGER socket option is set, the return can occur at three differrent times.
close returns immediately, without waiting at all (the defaults; Figure 7.6)
close lingers until the ACK of our FIN is received (Figure 7.7), or
shutdown followed by a read waits until we receive the peer's FIN (Figure 7.8)
His figures, and his commentary, indicate other than "application level acknowledgement", the combination of shutdown(), followed by a read() waiting for a zero return code (i.e. notification that the socket has been closed), is the only way to ensure the client application has received the data.
If, however, it is only important that the data has been successfully delivered (and acknowledged) the the peer's computer, then SO_LINGER would be sufficient.
Related
In TCP we say one side of the connection performs an "active close" and the other side performs a "passive close".
In terms of the Linux sockets API, how do you differentiate the active close and the passive close?
For example, suppose we have two connected Linux TCP sockets, A and P, that have exchanged information over the application-level protocol and they are both aware that it is time to close their sockets (neither expect to send or receive any more data to or from each other).
We want socket A to perform the active close, and for P to be the passive close.
There are a few things A and P could do. For example:
call shutdown(SHUT_WR)
call recv and expect to get 0 back
call close.
something else
What combination of these things and in what order should A do?... and what combination of these things and in what order should P do?
In terms of the Linux sockets API, how do you differentiate the active
close and the passive close?
The 'active' close is simply whichever side of the socket sends a FIN or RST packet first, typically by calling close().
What combination of these things and in what order should A do?... and
what combination of these things and in what order should P do?
In practice, most of this is application- and application-protocol specific. I will describe the minimum/typical requirement to answer your question, but your mileage may vary depending on what you are specifically trying to accomplish.
You may first call shutdown() on Socket A if you want to terminate communication in one direction or the other (or both) on Socket A. From your description, both programs already know they're done, perhaps due to application protocol messages, so this may not be necessary.
You must call close() on Socket A in order to close the socket and release the file descriptor.
On Socket P, you simply keep reading until recv() returns 0, and then you must call close() to close the socket and release the file descriptor.
For further reading, there are a number of good tutorials out there, and Beej's Guide to Network Programming is quite popular.
Active open is when you issue connect(2) explicitly to make a connection to a remote site. The call blocks until you get the socket opened on the other side (except if you issued O_NONBLOCK fcntl(2) call before calling connect(2).
Passive open is when you have a socket listen(2)ing on a connection and you have not yet issued an accept(2) system call. The accept(2) call normally blocks until you have a completely open connection and gives you a socket descriptor to communicate over it, or gives you inmediately a socket descriptor if the connection handshake has already finished when you issue the accept(2) syscall (this is a passive open). The limit in the number of passively open connections the kernel can accept on your behalf while you prepare yourself to make the accept(2) system call is what is called the listen(2) value.
Active close is what happens when you explicitly call shutdown(2) or close(2) system calls. As with passive open, there's nothing you can do to make a passive close (it's something that happens behind the scenes, product of other side's actions). You detect a passive close when the socket generates an end of file condition (this is, read(2) always returns 0 bytes on reading) meaning the other end has done a shutdown(2) (or close(2)) and the connection is half (or full) closed. When you explicitly shutdown(2) or close(2) your side, it's an active close.
NOTE
if the other end does an explicit close(2) and you continue writing on the socket, you'll get an error due to the impossibility of sending that data (in this case we can talk about a passive close(2) ---one that has occured without any explicit action from our side) but the other end can do a half close calling shutdown(2). This makes the tcp to send a FIN segment only and conserves the socket descriptor to allow the thread to receive any pending data in transit or not yet sent. Only when it receives and acknowledges the other end's FIN segment will it signal you that no more data remains in transit.
As the Title already says im looking for a way, to get notified when a client closes his Session unnormal.
I'm using the freeBSD OS.
The server is running with Xamount threads (depending on CPUcore amount). So I'm not forking, and there isn't a own process for each client.
That's why sending an deathpackage all time_t seconds, to recive a SIGPIPE isn't an option for me.
But i need to remove left clients from the kqueue, because otherwise after too many accept()'s my code will obviously run into memory troubles.
Is there a way, I can check without high performance loose per client, they are connected or not?
Or any event-notification, that would trigger if this happens? Or maybe is there a way of letting a programm send any signal to a port, even in abnormal termination case, before the Client process will exite?
Edit: that answer misses the question, because it's not about using kqueue. But if someone else finds the question by the title, it may be helpful anyway ...
I've often seen the following behaviour: if a client dies, and the server does a select() on the client's socket descriptor, select() returns with return code > 0 and FD_ISSET( fd ) will be true for that descriptor. But when you now try to read form the socket, read() (or recv()) return ERROR.
For a 'normal' connection using that to detect a client's death works fine for us, but there seems to be a different behaviour when the socket connection is tunneled but we haven't yet managed to figure that out completely.
According to the kqueue man page, kevent() should create an event when the socket has shutdown. From the description of th filter EVFILT_READ:
EVFILT_READ
Takes a descriptor as the identifier, and returns whenever there is data available to read. The behavior of the filter is slightly different depending on the descriptor type.
Sockets
Sockets which have previously been passed to listen() return when there is an incoming connection pending. data contains the size of the listen backlog.
Other socket descriptors return when there is data to be read, subject to the SO_RCVLOWAT value of the socket buffer. This may be overridden with a per-filter low water mark at the time the filter is added by setting the NOTE_LOWAT flag in fflags, and specifying the new low water mark in data. On return, data contains the number of bytes of protocol data available to read.
If the read direction of the socket has shutdown, then the filter also sets EV_EOF in flags, and returns the socket error (if any) in fflags. It is possible for EOF to be returned (indicating the connection is gone) while there is still data pending in the socket
buffer.
I have a AF_INET/SOCK_STREAM server written in C running on Android/Linux which looks more ore less like this:
...
for (;;) {
client = accept(...);
read(client, &message, sizeof(message));
response = process(&message);
write(client, response, sizeof(*response));
close(client);
}
As far as I know, the call to close should not terminate the connection to the client immediately, but it apparently does: The client reports "Connection Reset by Peer" before it has had a chance to read the server's response.
If I insert a delay between write() and close() the client can read the response as expected.
I got a hint that it might have to do with the SO_LINGER option, but I checked it's value and both members of struct linger (l_onoff, l_linger) have a value of zero.
Any ideas?
Stevens describes a configuration in which this can happen, but it depends on the client sending more data after the server has called close() (after the client should “know” that the connection is being closed). UNP 2nd ed s5.12.
Try tcpdumping the conversation to find out what’s really going on. If there's any possibility that a “clever” gateway (e.g. NAT) is between the two endpoints, tcpdump both ends and look for discrepancies.
Connection gets reset when you call close() on connection with data being sent. Specially for this case the sequence of shutdown() with SHUT_WR flag and then blocking read() is used.
Shutting down the writing end of the socket sends FIN and returns immediately, and the said read() blocks and returns 0 as soon as your peer replies with FIN in due turn. Basically, this is what you need in place of the delay between write() and close() you are talking about.
You do not need do anything with linger options in this case, leave it all to default.
SO_LINGER should be set (i.e. set to 1 not 0) if you want queued data to be sent before a close is effected.
SO_LINGER
Lingers on a close() if data is present. This option controls the
action taken when unsent messages
queue on a socket and close() is
performed. If SO_LINGER is set, the
system shall block the calling thread
during close() until it can transmit
the data or until the time expires. If
SO_LINGER is not specified, and
close() is issued, the system handles
the call in a way that allows the
calling thread to continue as quickly
as possible. This option takes a
linger structure, as defined in the
header, to specify the
state of the option and linger
interval.
In C, I understood that if we close a socket, it means the socket will be destroyed and can be re-used later.
How about shutdown? The description said it closes half of a duplex connection to that socket. But will that socket be destroyed like close system call?
This is explained in Beej's networking guide. shutdown is a flexible way to block communication in one or both directions. When the second parameter is SHUT_RDWR, it will block both sending and receiving (like close). However, close is the way to actually destroy a socket.
With shutdown, you will still be able to receive pending data the peer already sent (thanks to Joey Adams for noting this).
None of the existing answers tell people how shutdown and close works at the TCP protocol level, so it is worth to add this.
A standard TCP connection gets terminated by 4-way finalization:
Once a participant has no more data to send, it sends a FIN packet to the other
The other party returns an ACK for the FIN.
When the other party also finished data transfer, it sends another FIN packet
The initial participant returns an ACK and finalizes transfer.
However, there is another "emergent" way to close a TCP connection:
A participant sends an RST packet and abandons the connection
The other side receives an RST and then abandon the connection as well
In my test with Wireshark, with default socket options, shutdown sends a FIN packet to the other end but it is all it does. Until the other party send you the FIN packet you are still able to receive data. Once this happened, your Receive will get an 0 size result. So if you are the first one to shut down "send", you should close the socket once you finished receiving data.
On the other hand, if you call close whilst the connection is still active (the other side is still active and you may have unsent data in the system buffer as well), an RST packet will be sent to the other side. This is good for errors. For example, if you think the other party provided wrong data or it refused to provide data (DOS attack?), you can close the socket straight away.
My opinion of rules would be:
Consider shutdown before close when possible
If you finished receiving (0 size data received) before you decided to shutdown, close the connection after the last send (if any) finished.
If you want to close the connection normally, shutdown the connection (with SHUT_WR, and if you don't care about receiving data after this point, with SHUT_RD as well), and wait until you receive a 0 size data, and then close the socket.
In any case, if any other error occurred (timeout for example), simply close the socket.
Ideal implementations for SHUT_RD and SHUT_WR
The following haven't been tested, trust at your own risk. However, I believe this is a reasonable and practical way of doing things.
If the TCP stack receives a shutdown with SHUT_RD only, it shall mark this connection as no more data expected. Any pending and subsequent read requests (regardless whichever thread they are in) will then returned with zero sized result. However, the connection is still active and usable -- you can still receive OOB data, for example. Also, the OS will drop any data it receives for this connection. But that is all, no packages will be sent to the other side.
If the TCP stack receives a shutdown with SHUT_WR only, it shall mark this connection as no more data can be sent. All pending write requests will be finished, but subsequent write requests will fail. Furthermore, a FIN packet will be sent to another side to inform them we don't have more data to send.
There are some limitations with close() that can be avoided if one uses shutdown() instead.
close() will terminate both directions on a TCP connection. Sometimes you want to tell the other endpoint that you are finished with sending data, but still want to receive data.
close() decrements the descriptors reference count (maintained in file table entry and counts number of descriptors currently open that are referring to a file/socket) and does not close the socket/file if the descriptor is not 0. This means that if you are forking, the cleanup happens only after reference count drops to 0. With shutdown() one can initiate normal TCP close sequence ignoring the reference count.
Parameters are as follows:
int shutdown(int s, int how); // s is socket descriptor
int how can be:
SHUT_RD or 0
Further receives are disallowed
SHUT_WR or 1
Further sends are disallowed
SHUT_RDWR or 2
Further sends and receives are disallowed
This may be platform specific, I somehow doubt it, but anyway, the best explanation I've seen is here on this msdn page where they explain about shutdown, linger options, socket closure and general connection termination sequences.
In summary, use shutdown to send a shutdown sequence at the TCP level and use close to free up the resources used by the socket data structures in your process. If you haven't issued an explicit shutdown sequence by the time you call close then one is initiated for you.
I've also had success under linux using shutdown() from one pthread to force another pthread currently blocked in connect() to abort early.
Under other OSes (OSX at least), I found calling close() was enough to get connect() fail.
"shutdown() doesn't actually close the file descriptor—it just changes its usability. To free a socket descriptor, you need to use close()."1
Close
When you have finished using a socket, you can simply close its file descriptor with close; If there is still data waiting to be transmitted over the connection, normally close tries to complete this transmission. You can control this behavior using the SO_LINGER socket option to specify a timeout period; see Socket Options.
ShutDown
You can also shut down only reception or transmission on a connection by calling shutdown.
The shutdown function shuts down the connection of socket. Its argument how specifies what action to perform:
0
Stop receiving data for this socket. If further data arrives, reject it.
1
Stop trying to transmit data from this socket. Discard any data waiting to be sent. Stop looking for acknowledgement of data already sent; don’t retransmit it if it is lost.
2
Stop both reception and transmission.
The return value is 0 on success and -1 on failure.
in my test.
close will send fin packet and destroy fd immediately when socket is not shared with other processes
shutdown SHUT_RD, process can still recv data from the socket, but recv will return 0 if TCP buffer is empty.After peer send more data, recv will return data again.
shutdown SHUT_WR will send fin packet to indicate the Further sends are disallowed. the peer can recv data but it will recv 0 if its TCP buffer is empty
shutdown SHUT_RDWR (equal to use both SHUT_RD and SHUT_WR) will send rst packet if peer send more data.
linux: shutdown() causes listener thread select() to awake and produce error. shutdown(); close(); will lead to endless wait.
winsock: vice versa - shutdown() has no effect, while close() is successfully catched.
In the client, I have a
close(sockfd)
where sockfd is the socket that's connected to the server.
In the server I've got this:
if (sockfd.revents & POLLERR ||
desc_set[i].revents & POLLHUP || desc_set[i].revents & POLLNVAL) {
close(sockfd.fd);
printf("Goodbye (connection closed)\n");
}
Where sockfd is a struct pollfd, and sockfd.fd is the file descriptor of the client's socket.
When the client closes the socket like I put up there, the server doesn't seem to detect it with the second code (desc_set[i].revents & POLLHUP, etc.).
Does anyone know what's the problem?
Sounds like you've managed to half close the connection from the client side. In this state the connection can still send data in one direction, i.e. it operates in half-duplex mode. This is by design and would allow your server to finish replying to whatever the client sent. Typically this would mean completing a file transfer and calling close(), or answering all of the aspects of the query. In the half-closed state you can still quite sensibly send data to the side that has already called close(). In your server you will see eof if you try to read though. close() just means "I'm done sending, finish up whatever I asked for".
POLLHUP, POLLERR and POLLNVAL only checks the output side of the local connection, which is still valid here. There's a POLLRDHUP, which is a GNU extension that should detect the other side closing, but the tests you're doing are only checking if it's still writable, not if it's still readable.
See also this question, which is talking about java, but still very related.
A remote close or output shutdown is neither an error nor a hangup nor an invalid state. It is a read event such that read() will return zero. Just handle it as part of your normal read processing.
BTW your test condition above should read sockfd.revents & (POLLERR|POLLHUP|POLLNVAL).