I am currently using this function in a C client program. Everything seems to work fine but when the server to which this client is connected is shut down, write_all() returns 4 (that's len) instead of the expected -1.
int write_all(int sock, const void *buf, size_t len)
{
int buf_size = len;
while(len > 0)
{
int result = write(sock, buf, len);
if(result < 0)
{
if(errno == EINTR)
continue;
return result;
}
buf += result;
len -= result;
}
return buf_size;
}
Is there anything I am missing in this function? Is there any other function I can call beforehand to make sure the server is still up?
Thanks
You say "shut down", do you mean that you switch the power off, without gracefull TCP closing?
In that case write call returns with success. Data is in TCP sending buffer, and TCP stack does not yet know that peer is down. Program will get EPIPE or other error during later calls.
TCP stack will try retransmission a while, before making decision of connection failure.
To me this looks like you won't get around implementing some sort of hand shake.
As if it's not enough for your sender to know the data it send had been fully received (what I assume is the case), but it also needs to know if any kind on processing had been done on it by the receiver, you expect more from the socket's mechanics than they can provide ...
The sockets are just the transmitter.
Note: I'm assuming TCP here.
From the return value, I gather that the client managed to write 4 bytes to the send buffer before learning that the server closed its end or otherwise disappeared. If it disappeared without proper closing, the only way to know would be a timed-out send. The next write, shutdown or close after that will get the error.
If you want to get prompt notification of disappearing endpoints without having to constantly send data, you can activate the socket keepalive option. In Linux, that would be a setsockopt(..., SOL_SOCKET, SO_KEEPALIVE, ...), and TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT at the SOL_TCP level.
Related
Before I Start
Please don't mark this question as a duplicate. I have already seen the numerous posts on SO about handling multiple clients with socket programming. Most people recommend just multi-threading, but I am trying to avoid that path because I have read it has a few problems:
Bad Scalability
Large Overhead/Inefficient/Memory Hungry
Difficult to Debug
Any posts that I have read that specifically talk about using a single thread either have bad/no answers or have unclear explanations, like people saying "Just use select()!"
The Problem
I am writing code for a server to handle multiple (~1000) clients, and I'm having trouble figuring out how to create an efficient solution. Right now I already have the code for my server that is able to handle 1 client at a time. Both are written in C; the server is on Windows using WinSock and the client is on Linux.
The server and client send several communications back and forth, using send() and blocking recv() calls. Writing this code was pretty simple, and I won't post it here because it is pretty long and I doubt anyone will actually read through all of it. Also the exact implementation is not important, I just want to talk about high level pseudocode. The real difficulty is changing the server to handle multiple clients.
What's Already Out There
I have found a nice PDF tutorial about how to create a WinSock server that handles multiple clients and it can be found here: WinSock Multiple Client Support. It's in C++ but it's easily transferable to C.
From what I understand the server operates something like this:
while (running) {
Sleep(1000);
/* Accept all incoming clients and add to clientArray. */
for (client in clientArray) {
/* Interact with client */
if (recv(...) == "disconnect") {
/* Disconnect from client */
}
}
}
/* Close all connections. */
The problem that I see with using this approach is that you essentially only handle one client at a time (which is obvious because you aren't multithreading), but what if the interaction with each client only needs to happen once? Meaning, what if I just want to send some data back and forth and close the connection? This operation could take anywhere from 5 seconds to 5 minutes depending on the speed of the clients connection, so other clients would be blocking on a connect() call to the server while the server handles a client for 5 minutes. It doesn't seem very efficient, but maybe the best way would be to implement a waiting queue, where clients are connected and told to wait for a while? I'm not sure, but it makes me curious about how large servers send out update downloads concurrently to thousands of clients, and if I should operate the same way.
Also, is there a reason for adding a Sleep(1000) call in the main server loop, if the send() and recv() between the server and client take a while (~1 minute)?
What I'm Asking For
What I want is a solution to handling multiple clients on a single threaded server that is efficient enough for ~1000 clients. If you tell me that the solution in the PDF is fine, that's good enough for me (maybe I'm just too preoccupied with efficiency.)
Please give answers that include a verbal explanation of the implementation, server/client pseudocode, or even a small sample code for the server, if you're feeling sadistic.)
Thanks in advance.
I have written single thread socket pool handling. Im using non-blocking sockets and select call to handle all send, receive and errors.
My class keep all sockets in array, and build 3 fd set's for select call. When something happens it check read or write or error list and handle those events.
For example, non-blocking client socket during connection can trigger write or error event. If error event happens then connection failed. If write happens, connection is established.
All sockets is in read fd set. If you create server socket (with bind and listen) new connection will trigger read event. Then check if socket is server socket then call accept for new connection. If read operation is triggered by regular socket then there is some bytes to read.. just call recv with buffer arge enough to suck all data from that socket.
SOCKET maxset=0;
fd_set rset, wset, eset;
FD_ZERO(&rset);
FD_ZERO(&wset);
FD_ZERO(&eset);
for (size_t i=0; i<readsockets.size(); i++)
{
SOCKET s = readsockets[i]->s->GetSocket();
FD_SET(s, &rset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<writesockets.size(); i++)
{
SOCKET s = writesockets[i]->s->GetSocket();
FD_SET(s, &wset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<errorsockets.size(); i++)
{
SOCKET s = errorsockets[i]->s->GetSocket();
FD_SET(s, &eset);
if (s > maxset) maxset = s;
}
int ret = 0;
if (bBlocking)
ret = select(maxset + 1, &rset, &wset, &eset, NULL/*&tv*/);
else
{
timeval tv= {0, timeout*1000};
ret = select(maxset + 1, &rset, &wset, &eset, &tv);
}
if (ret < 0)
{
//int err = errno;
NetworkCheckError();
return false;
}
if (ret > 0)
{
// loop through eset and check each with FD_ISSET. if you find some socket it means connect failed
// loop through wset and check each with FD_ISSET. If you find some socket check is there any pending connectin on that socket. If there is pending connection then that socket just got connected. Otherwise select just reported that some data has been sent and you can send more.
// finally, loop through rset and check each with FD_ISSET. If you find some socket then check is this socket your server socket (bind and listen). If its server socket then this is signal new client want to connect.. just call accept and new connection is established. If this is not server socket, then just do recv on that socket to collect new data.
}
There is few more things to handle... All sockets must be in non-blocking mode. Each send or recv calls will return -1 (error) but error code is EWOULDBLOCK. Thats normal and ignore error. If recv returns 0 then this connection is dropped. If send return 0 bytes sent then internal buffer is full.
You need to write additional code to serialize and parse data. For example, after recv, message may not be complete (depending on message size) so it may take more than one recv calls to receive complete message. Sometimes if messages is short recv call can deliver several messages in buffer. So, you need to write good parser or design good protocol, easy to parse.
First, regarding single-thread approach: I'd say it's bad idea because your server processing power is limited by performance of single processor core. But other than that it'll work to some extent.
Now about multiclient problem. I'd suggest using WSASend and WSARecv with their compilation routines. It also can be scaled to multiple threads if necessary.
Server core will look something like this:
struct SocketData {
::SOCKET socket;
::WSAOVERLAPPED overlapped;
::WSABUF bufferRef;
char buf [1024];
// other client-related data
SocketData (void) {
overlapped->hEvent = (HANDLE) this;
bufferRef->buf = buf;
bufferRef->len = sizeof (buf);
// ...
}
};
void OnRecv (
DWORD dwError,
DWORD cbTransferred,
LPWSAOVERLAPPED lpOverlapped,
DWORD dwFlags) {
auto data = (SocketData*) lpOverlapped->hEvent;
if (dwError || !cbTransferred) {
::closesocket (data->socket);
delete data;
return;
}
// process received data
// ...
}
// same for OnSend
void main (void) {
// init and start async listener
::SOCKET serverSocket = ::socket (...);
HANDLE hAccept = ::CreateEvent (nullptr, 0, 0, nullptr);
::WSAEventSelect (serverSocket, FD_ACCEPT, hAccept);
::bind (serverSocket, ...);
::listen (serverSocket, ...);
// main loop
for (;;) {
int r = ::WaitForSingleObjectEx (hAccept, INFINITE, 1);
if (r == WAIT_IO_COMPLETION)
continue;
// accept processing
auto data = new SocketData ();
data->socket = ::accept (serverSocket, ...);
// detach new socket from hAccept event
::WSAEventSelect (data->socket, 0, nullptr);
// recv first data from client
::WSARecv (
data->socket,
&data->bufferRef,
1,
nullptr,
0,
&data->overlapped,
&OnRecv);
}
}
Key points:
wait in main loop (WaitForSingleObjectEx, WaitForMultipleObjectsEx etc.) must be alertable;
most data processing done in OnSend/OnRecv;
all processing must be done without blocking APIs in OnSend/OnRecv;
for event-based processing events must be waited in main loop.
OnRecv will be called for each processed incoming packet. OnSend will be called for each processed outgoing packet. Keep in mind: how many data you asked to send/recv is not the same as what actually processed in packet.
I am trying to read data off an Openssl linked socket using SSL_read. I perform Openssl operations in client mode that sends command and receives data from a real-world server. I used two threads where one thread handles all Openssl operations like connect, write and close. I perform the SSL_read in a separate thread. I am able to read data properly when I issue SSL_read once.
But I ran into problems when I tried to perform multiple connect, write, close sequences. Ideally I should terminate the thread performing the SSL_read in response to close. This is because for the next connect we would get a new ssl pointer and so we do not want to perform read on old ssl pointer. But problem is when I do SSL_read, I am stuck until there is data available in SSL buffer. It gets blocked on the SSL pointer, even when I have closed the SSL connection in the other thread.
while(1) {
memset(sbuf, 0, sizeof(uint8_t) * TLS_READ_RCVBUF_MAX_LEN);
read_data_len = SSL_read(con, sbuf, TLS_READ_RCVBUF_MAX_LEN);
switch (SSL_get_error(con, read)) {
case SSL_ERROR_NONE:
.
.
.
}
I tried all possible solutions to the problem but non works. Mostly I tried indication for letting me know there might be data in SSL buffer, but none of it returns proper indication.
I tried:
- Doing SSL_pending first to know if there is data in SSL buffer. But this always returns zero
- Doing select on the Openssl socket to see if it returns value bigger than zero. But it always returns zero.
- Making the socket as non-blocking and trying the select, but it doesnt seem to work. I am not sure if I got the code properly.
An example of where I used select for blocking socket is as follows. But select always returns zero.
while(1) {
// The use of Select here is to timeout
// while waiting for data to read on SSL.
// The timeout is set to 1 second
i = select(width, &readfds, NULL,
NULL, &tv);
if (i < 0) {
// Select Error. Take appropriate action for this error
}
// Check if there is data to be read
if (i > 0) {
if (FD_ISSET(SSL_get_fd(con), &readfds)) {
// TODO: We have data in the SSL buffer. But are we
// sure that the data is from read buffer? If not,
// SSL_read can be stuck indefinitely.
// Maybe we can do SSL_read(con, sbuf, 0) followed
// by SSL_pending to find out?
memset(sbuf, 0, sizeof(uint8_t) * TLS_READ_RCVBUF_MAX_LEN);
read_data_len = SSL_read(con, sbuf, TLS_READ_RCVBUF_MAX_LEN);
error = SSL_get_error(con, read_data_len);
switch (error) {
.
.
}
So as you can see I have tried number of ways to get the thread performing SSL_read to terminate in response to close, but I didnt get it to work as I expected. Did anybody get to make SSL_read work properly? Is non-blocking socket only solution to my problem? For blocking socket how do you solve the problem of quitting from SSL_read if you never get a response for command? Can you give an example of working solution for non blocking socket with read?
I can point you to a working example of non-blocking client socket with SSL ... https://github.com/darrenjs/openssl_examples
It uses non-blocking sockets with standard linux IO (based on poll event loop). Raw data is read from the socket and then fed into SSL memory BIO's, which then perform the decryption.
The approach I used was single threaded. A single thread performs the connect, write, and read. This means there cannot be any problems associated with one thread closing a socket, while another thread is trying to use that socket. Also, as noted by the SSL FAQ, "an SSL connection cannot be used concurrently by multiple threads" (https://www.openssl.org/docs/faq.html#PROG1), so single threaded approach avoids problems with concurrent SSL write & read.
The challenge with single threaded approach is that you then need to create some kind of synchronized queue & signalling mechanism for submitting and holding data pending for outbound (eg, the commands that you want to send from client to server), and get the socket event loop to detect when there is data pending for write and pull it from the queue etc. For that I would would look at standard std::list, std::mutex etc, and either pipe2 or eventfd for signalling the event loop.
OpenSSL calls recv() which in turn obeys the SOCKET's timeout, which by default is infinite. You can change the timeout thusly:
void socket_timeout_receive_set(SOCKET handle, dword milliseconds)
{
if(handle==SOCKET_HANDLE_NULL)
return;
struct timeval tv = { long(milliseconds / 1000), (milliseconds % 1000) * 1000 };
setsockopt(handle, SOL_SOCKET, SO_RCVTIMEO, (char *)&tv, sizeof(tv));
}
Unfortunately, ssl_error_get() returns SSL_ERROR_SYSCALL which it returns in other situations too, so it's not easy to determine that it timed out. But this function will help you determine if the connection is lost:
bool socket_dropped(SOCKET handle)
{
// Special thanks: "Detecting and terminating aborted TCP/IP connections" by Vinayak Gadkari
if(handle==SOCKET_HANDLE_NULL)
return true;
// create a socket set containing just this socket
fd_set socket_set;
FD_ZERO(&socket_set);
FD_SET(handle, &socket_set);
// if the connection is unreadable, it is not dropped (strange but true)
static struct timeval timeout = { 0, 0 };
int count = select(0, &socket_set, NULL, NULL, &timeout);
if(count <= 0) {
// problem: count==0 on a connection that was cut off ungracefully, presumably by a busy router
// for connections that are open for a long time but may not talk much, call keepalive_set()
return false;
}
if(!FD_ISSET(handle, &socket_set)) // creates a dependency on __WSAFDIsSet()
return false;
// peek at the next character
// recv() returns 0 if the connection was dropped
char dummy;
count = recv(handle, &dummy, 1, MSG_PEEK);
if(count > 0)
return false;
if(count==0)
return true;
return sec==WSAECONNRESET || sec==WSAECONNABORTED || sec==WSAENETRESET || sec==WSAEINVAL;
}
I wrote simple TCP/IP multi-thread ANSI C server (client is C sharp), everything works fine except when the server doesnt receive proper signal from client it wont end the thread and close its socket (for example when client crash). Eventually it could become problem if those threads accumulate.
I got threads stored in Linked List - iterating through them isnt a problem. However they are all blocked by recv() by default and since dead client wont send anything they become stuck in memory.
What is the proper way of maintaining list of online clients? (or how to detect threads with broken connection).
struct tListItem {
pthread_t thisThread;
char* name;
int c_sockfd;
int run;
tListItem* next;
tListItem* prev;};
struct tList{
tListItem* head;
int count;};
code of thread:
while(param->run)
{
bzero(&buf, sizeof(buf));
if ((readLen = recv(param->c_sockfd, buf, BUFFSIZE, 0)) == -1)
{
perror("Read error");
param->run = 0;
}
else if (readLen > 0) {
printf("%s: %s \n", param->name, buf);
parseIncoming(param->c_sockfd, param, buf);}}
and here is my attempt to detect broken connection, but this causes the server to end with no message:
void* maintenance() {
tListItem *item;
char buf[4] = "PNG";
while(1)
{
usleep(2000000);
item= threadList->head;
while(item != 0)
{
if ((send(item->c_sockfd, buf, 3, NULL)) == -1)
{
perror("Write error");
item->run = 0;
}
item = item->next;
}
}
}
There's a few common ways this is dealt with:
Implement a heartbeat/ping-pong in your protocol on top of TCP. That is, periodically the client and/or server
sends a heartbeat message to the other end. If the server has not received any data or heartbeat messages within a period of time, e.g. two times the heartbeat period, or if sending the heartbeat message from the server fails, then consider the connection to be dead and close it.
Implement an overall data timeout. Each time the server receives data, you read time current time. Periodically you check the connection for when you last received data, and time out/close connections that haven't received data in a while.
Enable TCP keepalive. This is basically a last resort if you cannot do either 1. or 2.. It'll help you detect dead peers, as the TCP keepalives will break the connection if the peer cannot be reached. (Though it will not help you detect idle clients). Note that the default for keepalives is in the order of hours.
In all cases you should always to be read()/recv() or otherwise monitoring the socket for read events so you can learn as quick as possible if the connection actively breaks.
It's also quite hard to implement this if you're doing blocking read()/recv() calls, you would normally need to set a timeout on the read() so you can wake up periodically and send a heartbeat message or check if the client has been idle for too long - this is best done by using select()/poll() or the like so you can get a timeout instead of doing a block read() that might never return.
I am writing a simple web server and client using UDP and so far: the programs can connect to each other, the client can send a request, the server can read the request, the server can recognize the client's IP address and client's port, and the server can send a message back to the client
My problem is that my client code gets stuck waiting in the rcvfrom function, even after the server has sent a response.
Here is the function that is supposed to pick up the server message and return the number of bytes read by the socket:
ssize_t receive_from_server(rdp_socket *rsocket, char *buffer, size_t buf_len){
socklen_t sendsize = sizeof(rsocket->server_addr);
bzero(&(rsocket->server_addr), sendsize);
//STUCK HERE:
return recvfrom(rsocket->sockfd, buffer, buf_len, 0,
(struct sockaddr*)&(rsocket->server_addr), &sendsize);
}
I set the sockopts for both SO_SNDTIMEO and SO_RCVTIMEO to timeout after a few seconds.
Question:
In the short term future I will be adding acknowledgements (ACKs) for reliable data transfer. I imagine that missing ACKs could be the issue but I'm just wondering if, to the trained eye, it looks like a different problem.
Are ACKs necessary for a timeout to work?
How can I synchronize my client and server so that they can actually communicate with each other?
Since UDP does not provide reliability, you will need to implement retransmission of missing data. Since it looks like this is a client request server response model, the easiest retransmission implementation for you may be to resend the request when you time out waiting for the response, and wait for the response again. You may want to implement a retry counter and give up after a certain number of retries.
If the SO_RCVTIMEO and SO_SNDTIMEO socket options do not seem to be taking effect, it may be those options are not implemented for that type of socket. Check the return value of the setsockopt()call to make sure they succeeded.
As a workaround, you can change your receive_from_server() function to use poll() or select() to wait for a readable event for some amount of time, instead of blocking in recvfrom().
ssize_t receive_from_server(rdp_socket *rsocket, char *buffer, size_t buf_len){
struct pollfd pfd = { rsocket->sockfd, POLLIN };
int pollresult = poll(&pfd, 1, RECV_TIMEOUT_SECONDS * 1000);
if (pollresult > 0) {
socklen_t sendsize = sizeof(rsocket->server_addr);
bzero(&(rsocket->server_addr), sendsize);
return recvfrom(rsocket->sockfd, buffer, buf_len, MSG_DONTWAIT,
(struct sockaddr*)&(rsocket->server_addr), &sendsize);
}
if (pollresult == 0) {
errno = ETIME;
}
return -1;
}
I have written a forward proxy. I gonna use it for both windows and linux. I do have required changes as per the OS. However, I keep seeing some raise conditions. Mostly I believe they are due to my misunderstanding in guessing which is the last packet (FIN sigal). Currently I do select on set of sockets. Whichever socket gets signalled, I do read() on it. If read returns 0 then I assume it is a FIN packet and I close that socket. Can it happen that my read() gives non zero value. But that packet does contain FIN (I think it can happen). So, I do not close some sockets though they have got closed.
I am not sure how proxies detect which socket has closed? Or which is a last packet on the established connection.
My code looks like follow:
I have 100 fds which I have accepted from client. I store them an array sock_array[total_size].
select(copy_of_sock_array,timeout)
for(int cnt=0;cnt<total_size;cnt++)
{
if(FD_ISSET(sock_array[cnt],sock_array))
{
ret = recv(sock_array[cnt],buffer,len);
if(ret<=0){
/*This must be a FIN packet */
/* Close corresponding socket which is opened with outer world */
close(/*corresponding socket*/);
}
}
}
Does this look ok?
Thanks
You need to do a non-blocking read, and keep reading from the socket until you get a return value that indicates you should stop reading.
ssize_t r = 0;
for (;;) {
r = recv(sock, buf, bufsz, MSG_DONTWAIT);
if (r <= 0) {
if (r < 0 && errno == EINTR) {
continue;
}
break;
}
/* ... handle data in buf .. */
}
if (r < 0) {
if (errno == EAGAIN) {
/* ... wait in select again ... */
} else {
/* ... handle error ... */
}
} else {
/* got FIN */
}
Note that just because FIN is received does not necessarily mean the connection should be closed. The FIN merely indicates that no more data will be sent, but the peer may still be willing to accept more data. This can happen in HTTP where the client only wants a single response, so it delivers a FIN after its request. It still expects to receive the response though.
Your proxy likely has two sockets, say sock1 and sock2. So receipt of the FIN on sock1 should mean that this indication be forwarded onto sock2 after any data that has been queued on it has been delivered (and the mirror is true as well). You can forward the FIN by using shutdown.
shutdown(sock2, SHUT_WR);
When FIN has been received from both sock1 and sock2, you can call close on both sockets.
So addressing your questions.
Can it happen that my read() gives non zero value. But that packet does contain FIN (I think it can happen).
Yes, this may happen. This is why you continue reading until you get an indication to stop. Well, technically, you don't have to. You can defer that until you have processed some other connection if you have per connection fairness issues. But, you need to come back to it and finish reading before you enter your select wait.
So, I do not close some sockets though they have got closed. I am not sure how proxies detect which socket has closed? Or which is a last packet on the established connection.
As I described, as a (transparent) proxy, the socket can be safely closed once you have forwarded a FIN on it and a FIN has been received on it. If you are not a transparent proxy, you play by a different set of rules, since you really are the server for the client in that case. So, you can close the socket whenever the application protocol you are implementing permits you to do so.
Sockets have a well defined behavior. If you receive data and the connection is closed after that, you'll need two read()s. The first will return the data and the second one will return 0, to signal the end of connection.
You always have to read until the syscall returns 0.
And you don't need a non-blocking read to detect this!