I'm working on this calendar/reminders server-client thing. I've stumbled across a bug where I may have misunderstood the behaviour of the accept() function while trying to handle new connections.
The client basically specifies a day to view and sends it to the server, then gets some text back and closes the connection.(I have been using telnet and netcat to test this so far though.)
After I hit ctrl+d on netcat after I send the command and receive the message, the server gets and infinite output loop of "New connection\n".
The way I understood accept() was that when it is called, it sets the left hand side to a socket descriptor for a connection on the listen() backlog, or waits until there is a connection before returning. So either I am mistaken or I am doing something wrong:
bind(client_socket, (struct sockaddr*)&server_address, sizeof(server_address));
listen(client_socket, 5);
//start main loop. first level checks for commands
while (1)
{
client_socket = accept(client_socket, NULL, NULL);
printf("New connection.\n");
recv(client_socket, buffer, sizeof(buffer), 0);
/*Bunch of code here that interprets what was sent with some string manipulation
and serves back parts of a text file. No socket functions other than send() twice here*/
recv(client_socket, buffer, sizeof(buffer), 0);
}
The idea I had in mind was, once the job is done, wait until the client closes the connection, which sends a message of length 0(hence the recv() at the end), then loop back to the accept() which accepts or waits for the next connection. What am I missing here?
In your second call to accept, you are passing as the listening socket the descriptor for the first connection rather than the listening socket. You really need to check for errors on all these calls. Otherwise, your code will be impossible to debug.
client_socket = accept(client_socket, NULL, NULL);
This is fine the first time. But it leaks the descriptor for the listening socket. So you can't accept another connection.
Related
I am a newbie in C. I just noticed that the connect() function on the client side can return as long as the TCP three-way hand-shake is finished. I mean connect() can even return before the accept() on the server side is called (correct me if I am wrong). Based on this knowledge, my question is that when I call select() afterwards on the client side, and watch the file descriptor to wait for it to be writeable, when select() successfully returns, that means the server side has already called accept() and now I can safely write to the server side, right? Many thanks for your time.
int flags = fcntl(fd, F_GETFL);
flags |= O_NONBLOCK;
fcntl(fd, F_SETFL, flags);
if (connect(fd, (struct sockaddr *)saptr, salen) < 0)
{
if (errno != EINPROGRESS)
/* error_return */
}
fd_set set;
FD_ZERO (&set);
FD_SET (fd, &set);
select (FD_SETSIZE, NULL, &set, NULL, &timeout)
/* Here, if select returns 1, that means accept() is already called
on the server side, and now I can safely write to the server, right? */
when select() successfully returns, that means the server side has already called accept()
No, not necessarily. connect() returns when the connection attempt is complete, having either succeeded or failed. On the remote side, this is handled by the network stack, outside the context of any application. The subsequent accept() itself produces no additional communication.
and now I can safely write to the server side, right?
There are all kinds of things you could mean by "safely", but if you mean that the local side can write at least one byte without blocking then yes, select() promises you that. Whatever you successfully write will be sent over the wire to the remote side. It may be buffered there for a time, depending on the behavior of the software on the remote end. Whether that software has yet accept()ed the connection or not is not directly relevant to that question.
Update: note also that the network stack maintains a per-socket queue of established connections that have not yet been accept()ed (its backlog). This queuing behavior is one reason why a server might not accept() connections immediately after they are established, especially under heavy load.
'I mean connect() can even return before the accept() on the server
side is called'
Yes, it can, and does.
when I call select() afterwards on the client side, and watch the file
descriptor to wait for it to be writeable, when select() successfully
returns, that means the server side has already called accept() and
now I can safely write to the server side, right?
Sure. Write away:)
I need to know whether a client connected/disconnected and handle it.
This was my only idea:
while(!serverStop)
{
fd_set rfds, wfdsBefore, wfdsAfter;
FD_ZERO(&rfds);
FD_SET(serverFd, &rfds);
FD_ZERO(&wfdsBefore);
fillWithClientFds(&wfdsBefore); // clients only listen for messages
wfdsAfter = wfdsBefore;
while(1)
{
select(notimportant, &rfds, &wfdsAfter, NULL, NULL);
if (FD_ISSET(serverFd, &rfds)) // new client appeared
break;
if (doSetsDiffer(&wfdsBefore, &wfdsAfter)) // some client disconnected (doesn't work)
break;
}
// inform connected clients about disconnected ones
}
Not only busy waiting would occur but also this approach doesn't even work (wfdsAfter doesn't change despite the fact that client closed the socket).
Is there any way to do it? The only requirement is to not use multithreading.
serverFd was made with PF_UNIX and SOCK_STREAM flags.
You should place each client file descriptor in the read descriptors (rfds) set after it connects, and, when the file descriptor is subsequently returned as readable, attempt to read from the socket.
First, if your client is really sending nothing (and isn't yet disconnected), its socket will never be marked as readable. That seems like it would solve your issue since you say the client never actually sends anything: it won't be marked readable then until the client disconnects.
But even if the client sends data, the file descriptor would only be marked readable if there were data available OR the client had disconnected. You can easily then distinguish by attempting to read the socket. The return value would be either number of bytes read (if there are data), or zero if the client has disconnected.
(Servers often add the O_NONBLOCK option to sockets to ensure they get notified when the client has data to send but want to ensure they don't block waiting for data from a client. With that option, read still returns 0 when the client has disconnected. With the option, if the client is still around, but there is no data available, the read call would return -1 with errno set to EAGAIN/EWOULDBLOCK.)
One other nuance I haven't explained is that it is possible to close data delivery in one direction while allowing it to continue in the other (see shutdown(2) if you care about this).
You are putting the client sockets in the write descriptor set. You need to put them in the read descriptor set instead.
When a server socket has at least 1 pending client request, it is readable. You can call accept() to accept a client.
When a socket has data in its inbound buffer, or its connected peer has disconnected, it is readable, not writable. You can call read() to differentiate. read() returns > 0 on inbound data, 0 on graceful disconnect, and -1 on error.
A socket is writable when it has available space in its outbound buffer. If write() fails with an EWOULDBLOCK error, the outbound buffer has filled up, and the socket is no longer writable. When the buffer clears up some space, the socket will become writable again.
Also, select() modifies the fdsets you pass to it, so you need to reset rfds on every loop iteration. To avoid that, you can use (e)poll() instead.
So, you need something more like this instead:
fd_set rfds;
while (!serverStop)
{
FD_ZERO(&rfds);
FD_SET(serverFd, &rfds);
fillWithClientFds(&rfds); // clients only listen for messages
if (select(notimportant, &rfds, NULL, NULL, NULL) < 0)
break;
if (FD_ISSET(serverFd, &rfds)) // new client appeared
{
// call accept(), add client to connected list...
}
// clear disconnected list...
for (each client in connected list)
{
if (FD_ISSET(clientFd, &rfds))
{
int nBytes = read(clientFd, ...);
if (nBytes > 0)
{
// handle client data as needed ...
}
else if (nBytes == 0)
{
// add client to disconnected list
}
else
{
// handle error...
// possibly add client to disconnected list...
}
}
}
for (each client in disconnected list)
{
// remove client from connected list...
}
for (each client in disconnected list)
{
// inform connected clients
}
}
[I asked something similar before. This is a more focused version.]
What can cause a server's select() call on a TCP socket to consistently time-out rather than "see" the client's close() of the socket? On the client's side, the socket is a regular socket()-created blocking socket that successfully connects to the server and successfully transmits a round-trip transaction. On the server's side, the socket is created via an accept() call, is blocking, is passed to a child server process via fork(), is closed by the top-level server, and is successfully used by the child server process in the initial transaction. When the client subsequently closes the socket, the select() call of the child server process consistently times-out (after 1 minute) rather than indicating a read-ready condition on the socket. The select() call looks for read-ready conditions only: the write-ready and exception arguments are NULL.
Here's the simplified but logically equivalent select()-using code in the child server process:
int one_svc_run(
const int sock,
const unsigned timeout)
{
struct timeval timeo;
fd_set fds;
timeo.tv_sec = timeout;
timeo.tv_usec = 0;
FD_ZERO(&fds);
FD_SET(sock, &fds);
for (;;) {
fd_set readFds = fds;
int status = select(sock+1, &readFds, 0, 0, &timeo);
if (status < 0)
return errno;
if (status == 0)
return ETIMEDOUT;
/* This code not reached when client closes socket */
/* The time-out structure, "timeo", is appropriately reset here */
...
}
...
}
Here's the logical equivalent of the sequence of events on the client-side (error-handling not shown):
struct sockaddr_in *raddr = ...;
int sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
(void)bindresvport(sock, (struct sockaddr_in *)0);
connect(sock, (struct sockaddr *)raddr, sizeof(*raddr));
/* Send a message to the server and receive a reply */
(void)close(sock);
fork(), exec(), and system() are never called. The code is considerably more complex than this, but this is the sequence of relevant calls.
Could Nagel's algorithm cause the FIN packet to not be sent upon close()?
Most likely explanation is that you're not actually closing the client end of the connection when you think you are. Probably because you have some other file descriptor that references the client socket somewhere that is not being closed.
If your client program ever does a fork (or related calls that fork, such as system or popen), the forked child might well have a copy of the file descriptor which would cause the behavior you're seeing.
One way to test/workaround the problem is to have the client do an explicit shutdown(2) prior to closing the socket:
shutdown(sock, SHUT_RDWR);
close(sock);
If this causes the problem to go away then that is the problem -- you have another copy of the client socket file descriptor somewhere hanging around.
If the problem is due to children getting the socket, the best fix is probably to set the close-on-exec flag on the socket immediately after creating it:
fcntl(sock, F_SETFD, fcntl(sock, F_GETFD) | FD_CLOEXEC);
or on some systems, use the SOCK_CLOEXEC flag to the socket creation call.
Mystery solved.
#nos was correct in the first comment: it's a firewall problem. A shutdown() by the client isn't needed; the client does close the socket; the server does use the right timeout; and there's no bug in the code.
The problem was caused by the firewall rules on our Linux Virtual Server (LVS). A client connects to the LVS and the connection is passed to the least-loaded of several backend servers. All packets from the client pass through the LVS; all packets from the backend server go directly to the client. The firewall rules on the LVS caused the FIN packet from the client to be discarded. Thus, the backend server never saw the close() by the client.
The solution was to remove the "-m state --state NEW" options from the iptables(8) rule on the LVS system. This allows the FIN packets from the client to be forwarded to the backend server. This article has more information.
Thanks to all of you who suggested using wireshark(1).
select() call of Linux will modify value of timeout argument. From man page:
On Linux, select() modifies timeout to reflect the amount of time not
slept
So your timeo will runs to zero. And when it is zero select will return immediately (mostly with return value zero).
The following change may help:
for (;;) {
struct timeval timo = timeo;
fd_set readFds = fds;
int status = select(sock+1, &readFds, 0, 0, &timo);
As described in Beej's Guide to network Programming, select() monitors a set of file descriptors for reading (using recv()), a set of file descriptor for writing (using send()) and the last one, I don't know. When the server socket receives message from client sockets, read_fds set will be modified and select() return from blocking status. It is the same for sending message to client sockets. For example:
for(;;) {
read_fds = master; // copy it
if (select(fdmax+1, &read_fds, NULL, NULL, NULL) == -1) {
perror("select");
exit(4);
}
//the rest is code for processing ready socket
I guess the read_fds set will contain the only ready socket descriptor at this point (the others are removed), and the ready socket descriptor is the new connected socket or message sent from connected socket. Is my understanding correct?
It seems the ready socket must be handled one by one. When I tried to run it on gdb to understand the behavior, while the program was processing the ready socket (the code after select() return), I tried to send some message and connect to the server by some new clients. How can it recognize the new clients or newly sent message, even if select() is not called?
As described in Beej's Guide to network Programming, select() monitors a set of file descriptors for reading (using recv()), a set of file descriptor for writing (using send())
Yes
and the last one, I don't know.
The last one no longer has any useful meaning.
I guess the read_fds set will contain the only ready socket descriptor at this point (the others are removed), and the ready socket descriptor is the new connected socket or message sent from connected socket. Is my understanding correct?
That's correct.
It seems the ready socket must be handled one by one. When I tried to run it on gdb to understand the behavior, while the program was processing the ready socket (the code after select() return), I tried to send some message and connect to the server by some new clients. How can it recognize the new clients or newly sent message, even if select() is not called?
Normally when you create a polling loop such as this, you'd add new sockets to the loop. That is, you'd add them to the appropriate fd_sets before your next call to select.
When the new socket becomes writable, you'd send on it.
When you're dealing with multiple sockets that may potentially block (in your case reading sockets), you need to determine which sockets have data in them waiting to be read. You can do this by calling select() and adding the sockets to your read_set.
For your listening socket, if you call accept() and there is no pending connection, then your accept will block until a new connection arrives. So you also want to select() this socket. Once you accept that client, you will want to add that to your read_set.
e.g. Pseudo-code
for (;;) {
struct timeval tv = { timeout, 0 };
fd_set read_set;
FD_ZERO(&read_set);
FD_SET(listen_sock, &read_set);
max_fd = max(max_fd, listen_sock);
/* add all your other other client sockets to thread read_set */
n = select(max_fd, &read_set, NULL, NULL, tv);
if (n > 0) {
if (FD_ISSET(listen_sock, &read_set)) {
cli = accept(listen_sock);
/* add to list of clients */
}
else {
for (int i = 0; i < max_clients; i++) {
if (FD_ISSET(clients[i], &read_set)) {
/* data is waiting. recv */
bytes = recv(clients[i], ..)
if (bytes <= 0) {
/* error or EOF, remove client list, so we don't select on this anymore */
}
}
}
}
}
Note that sends can also block, if the other end is not actively reading, and the send buffer is full. So if you're sending, you might want to check if it's "sendable".
In my server I check if any socket is ready to read using select() to determine it. As a result in main loop select() is executed every time it iterates.
To test the server I wrote a simple client that sends only one message and then quits. BTW. I use protocol buffers to send information - message means an object of type class Message in this library.
The test session looks like:
select()
server's socket ready to read
accept() client's socket
read message from client's socket
select()
server's socket not ready to read, client's one ready
read message from client's socket
The last step is wrong because client has already closed connection. As a result protobuf library gets Segmentation fault. I wonder why FD_ISSET says the socket is ready in step 6 when it is closed. How can I check if a socket is closed?
EDIT:
I've found how to check if the socket is open
int error = 0;
socklen_t len = sizeof (error);
int retval = getsockopt (socket_fd, SOL_SOCKET, SO_ERROR, &error, &len );
the socket is "readable" if the remote peer closes it, you need to call recv and handle both the case where it returns an error, and the case where it returns 0, which indicates that the peer shut down the connection in an orderly fashion.
Reading the SO_ERROR sockopt is not the correct way, as it returns the current pending error (from, eg. a non-blocking connect)
The socket used for communication between a client and your server will be flagged as readable (i.e. select() will return) when there is data to read, or when there's an EOF to read (i.e. the peer closed the connection).
Just read() when select() returns and your fd is flagged. If read() returns a positive number, you got data. If it returns 0, you got EOF. If it returns -1, you have a problem (unless errno is EAGAIN).