I try to make a client/server system in C on a Linux platform. I want to listen on four different ports. Therefore I create four file descriptors, one for each port. Additionally the process binds the fd to the port and starts listen to it. This works fine.
Further I use select() to listen for connections and there I get a problem. In the first run the program waits at select until a client is connected. After I send with telnet a string to the server it continues. But in the second run of the loop the program stops again at select and waits as long as a new client connects. Even if I send a new string via telnet it waits and handles this after a new connection has been established. The example I used is similar to this link.
Therefore I do not use a timeout.
Why does it wait for a new connection at select? How can I handle this issue?
My code:
FD_ZERO(&read_sock);
FD_SET(fd[0], &read_sock);
FD_SET(fd[1], &read_sock);
FD_SET(fd[2], &read_sock);
FD_SET(fd[3], &read_sock);
while(TRUE){
fprintf(stderr,"Waiting for incoming connections...\n");
status = select(maxfd+1,&read_sock,NULL,NULL,NULL);
fprintf(stderr,"Number of fd: %d\n",status);
if(status>0){
for(int i=0; i< FD_SET_SIZE; i++){
if(FD_ISSET(fd[i], &read_sock)){
fd_accept=accept(fd[i],(struct sockaddr*)&client_address[i], &len);
if(client_sock[i] < 0) {
client_sock[i] = fd_accept;
}
int lenght = recv(client_sock[i],data,BUFFER-1,0);
if(lenght>0){
data[lenght] = '\0';
fprintf (stderr,"Received: %s\n", data);
}else if(lenght==0){
getpeername(fd[i],(struct sockaddr*)&client_address[i], &len);
close(fd[i]);
client_sock[i] = -1;
}else{
fprintf(stderr,"Error: %d\n",errno);
}
char string[] = "Test"; //sends a char Test to the client
write(client_sock[i],&string,sizeof(string));
}
}
}
}
select is a blocking operations. From man page:
select() and pselect() allow a program to monitor multiple file descriptors,
waiting until one or more of the file descriptors become "ready" for some class
of I/O operation (e.g., input possible). A file descriptor is considered ready
if it is possible to perform the corresponding I/O operation (e.g., read(2))
without blocking.
Since the descriptors you give it are listener sockets, it will block until there is a new socket on either one.
If you want to listen for data on one of the accepted sockets you need to select on the socket descriptor as well, which you store in the client_address vector.
Your code effectively listens for new connections, accepts it, reads data, writes something in return and then throws the socket away without closing it.
Related
Before I Start
Please don't mark this question as a duplicate. I have already seen the numerous posts on SO about handling multiple clients with socket programming. Most people recommend just multi-threading, but I am trying to avoid that path because I have read it has a few problems:
Bad Scalability
Large Overhead/Inefficient/Memory Hungry
Difficult to Debug
Any posts that I have read that specifically talk about using a single thread either have bad/no answers or have unclear explanations, like people saying "Just use select()!"
The Problem
I am writing code for a server to handle multiple (~1000) clients, and I'm having trouble figuring out how to create an efficient solution. Right now I already have the code for my server that is able to handle 1 client at a time. Both are written in C; the server is on Windows using WinSock and the client is on Linux.
The server and client send several communications back and forth, using send() and blocking recv() calls. Writing this code was pretty simple, and I won't post it here because it is pretty long and I doubt anyone will actually read through all of it. Also the exact implementation is not important, I just want to talk about high level pseudocode. The real difficulty is changing the server to handle multiple clients.
What's Already Out There
I have found a nice PDF tutorial about how to create a WinSock server that handles multiple clients and it can be found here: WinSock Multiple Client Support. It's in C++ but it's easily transferable to C.
From what I understand the server operates something like this:
while (running) {
Sleep(1000);
/* Accept all incoming clients and add to clientArray. */
for (client in clientArray) {
/* Interact with client */
if (recv(...) == "disconnect") {
/* Disconnect from client */
}
}
}
/* Close all connections. */
The problem that I see with using this approach is that you essentially only handle one client at a time (which is obvious because you aren't multithreading), but what if the interaction with each client only needs to happen once? Meaning, what if I just want to send some data back and forth and close the connection? This operation could take anywhere from 5 seconds to 5 minutes depending on the speed of the clients connection, so other clients would be blocking on a connect() call to the server while the server handles a client for 5 minutes. It doesn't seem very efficient, but maybe the best way would be to implement a waiting queue, where clients are connected and told to wait for a while? I'm not sure, but it makes me curious about how large servers send out update downloads concurrently to thousands of clients, and if I should operate the same way.
Also, is there a reason for adding a Sleep(1000) call in the main server loop, if the send() and recv() between the server and client take a while (~1 minute)?
What I'm Asking For
What I want is a solution to handling multiple clients on a single threaded server that is efficient enough for ~1000 clients. If you tell me that the solution in the PDF is fine, that's good enough for me (maybe I'm just too preoccupied with efficiency.)
Please give answers that include a verbal explanation of the implementation, server/client pseudocode, or even a small sample code for the server, if you're feeling sadistic.)
Thanks in advance.
I have written single thread socket pool handling. Im using non-blocking sockets and select call to handle all send, receive and errors.
My class keep all sockets in array, and build 3 fd set's for select call. When something happens it check read or write or error list and handle those events.
For example, non-blocking client socket during connection can trigger write or error event. If error event happens then connection failed. If write happens, connection is established.
All sockets is in read fd set. If you create server socket (with bind and listen) new connection will trigger read event. Then check if socket is server socket then call accept for new connection. If read operation is triggered by regular socket then there is some bytes to read.. just call recv with buffer arge enough to suck all data from that socket.
SOCKET maxset=0;
fd_set rset, wset, eset;
FD_ZERO(&rset);
FD_ZERO(&wset);
FD_ZERO(&eset);
for (size_t i=0; i<readsockets.size(); i++)
{
SOCKET s = readsockets[i]->s->GetSocket();
FD_SET(s, &rset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<writesockets.size(); i++)
{
SOCKET s = writesockets[i]->s->GetSocket();
FD_SET(s, &wset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<errorsockets.size(); i++)
{
SOCKET s = errorsockets[i]->s->GetSocket();
FD_SET(s, &eset);
if (s > maxset) maxset = s;
}
int ret = 0;
if (bBlocking)
ret = select(maxset + 1, &rset, &wset, &eset, NULL/*&tv*/);
else
{
timeval tv= {0, timeout*1000};
ret = select(maxset + 1, &rset, &wset, &eset, &tv);
}
if (ret < 0)
{
//int err = errno;
NetworkCheckError();
return false;
}
if (ret > 0)
{
// loop through eset and check each with FD_ISSET. if you find some socket it means connect failed
// loop through wset and check each with FD_ISSET. If you find some socket check is there any pending connectin on that socket. If there is pending connection then that socket just got connected. Otherwise select just reported that some data has been sent and you can send more.
// finally, loop through rset and check each with FD_ISSET. If you find some socket then check is this socket your server socket (bind and listen). If its server socket then this is signal new client want to connect.. just call accept and new connection is established. If this is not server socket, then just do recv on that socket to collect new data.
}
There is few more things to handle... All sockets must be in non-blocking mode. Each send or recv calls will return -1 (error) but error code is EWOULDBLOCK. Thats normal and ignore error. If recv returns 0 then this connection is dropped. If send return 0 bytes sent then internal buffer is full.
You need to write additional code to serialize and parse data. For example, after recv, message may not be complete (depending on message size) so it may take more than one recv calls to receive complete message. Sometimes if messages is short recv call can deliver several messages in buffer. So, you need to write good parser or design good protocol, easy to parse.
First, regarding single-thread approach: I'd say it's bad idea because your server processing power is limited by performance of single processor core. But other than that it'll work to some extent.
Now about multiclient problem. I'd suggest using WSASend and WSARecv with their compilation routines. It also can be scaled to multiple threads if necessary.
Server core will look something like this:
struct SocketData {
::SOCKET socket;
::WSAOVERLAPPED overlapped;
::WSABUF bufferRef;
char buf [1024];
// other client-related data
SocketData (void) {
overlapped->hEvent = (HANDLE) this;
bufferRef->buf = buf;
bufferRef->len = sizeof (buf);
// ...
}
};
void OnRecv (
DWORD dwError,
DWORD cbTransferred,
LPWSAOVERLAPPED lpOverlapped,
DWORD dwFlags) {
auto data = (SocketData*) lpOverlapped->hEvent;
if (dwError || !cbTransferred) {
::closesocket (data->socket);
delete data;
return;
}
// process received data
// ...
}
// same for OnSend
void main (void) {
// init and start async listener
::SOCKET serverSocket = ::socket (...);
HANDLE hAccept = ::CreateEvent (nullptr, 0, 0, nullptr);
::WSAEventSelect (serverSocket, FD_ACCEPT, hAccept);
::bind (serverSocket, ...);
::listen (serverSocket, ...);
// main loop
for (;;) {
int r = ::WaitForSingleObjectEx (hAccept, INFINITE, 1);
if (r == WAIT_IO_COMPLETION)
continue;
// accept processing
auto data = new SocketData ();
data->socket = ::accept (serverSocket, ...);
// detach new socket from hAccept event
::WSAEventSelect (data->socket, 0, nullptr);
// recv first data from client
::WSARecv (
data->socket,
&data->bufferRef,
1,
nullptr,
0,
&data->overlapped,
&OnRecv);
}
}
Key points:
wait in main loop (WaitForSingleObjectEx, WaitForMultipleObjectsEx etc.) must be alertable;
most data processing done in OnSend/OnRecv;
all processing must be done without blocking APIs in OnSend/OnRecv;
for event-based processing events must be waited in main loop.
OnRecv will be called for each processed incoming packet. OnSend will be called for each processed outgoing packet. Keep in mind: how many data you asked to send/recv is not the same as what actually processed in packet.
I am trying to read data off an Openssl linked socket using SSL_read. I perform Openssl operations in client mode that sends command and receives data from a real-world server. I used two threads where one thread handles all Openssl operations like connect, write and close. I perform the SSL_read in a separate thread. I am able to read data properly when I issue SSL_read once.
But I ran into problems when I tried to perform multiple connect, write, close sequences. Ideally I should terminate the thread performing the SSL_read in response to close. This is because for the next connect we would get a new ssl pointer and so we do not want to perform read on old ssl pointer. But problem is when I do SSL_read, I am stuck until there is data available in SSL buffer. It gets blocked on the SSL pointer, even when I have closed the SSL connection in the other thread.
while(1) {
memset(sbuf, 0, sizeof(uint8_t) * TLS_READ_RCVBUF_MAX_LEN);
read_data_len = SSL_read(con, sbuf, TLS_READ_RCVBUF_MAX_LEN);
switch (SSL_get_error(con, read)) {
case SSL_ERROR_NONE:
.
.
.
}
I tried all possible solutions to the problem but non works. Mostly I tried indication for letting me know there might be data in SSL buffer, but none of it returns proper indication.
I tried:
- Doing SSL_pending first to know if there is data in SSL buffer. But this always returns zero
- Doing select on the Openssl socket to see if it returns value bigger than zero. But it always returns zero.
- Making the socket as non-blocking and trying the select, but it doesnt seem to work. I am not sure if I got the code properly.
An example of where I used select for blocking socket is as follows. But select always returns zero.
while(1) {
// The use of Select here is to timeout
// while waiting for data to read on SSL.
// The timeout is set to 1 second
i = select(width, &readfds, NULL,
NULL, &tv);
if (i < 0) {
// Select Error. Take appropriate action for this error
}
// Check if there is data to be read
if (i > 0) {
if (FD_ISSET(SSL_get_fd(con), &readfds)) {
// TODO: We have data in the SSL buffer. But are we
// sure that the data is from read buffer? If not,
// SSL_read can be stuck indefinitely.
// Maybe we can do SSL_read(con, sbuf, 0) followed
// by SSL_pending to find out?
memset(sbuf, 0, sizeof(uint8_t) * TLS_READ_RCVBUF_MAX_LEN);
read_data_len = SSL_read(con, sbuf, TLS_READ_RCVBUF_MAX_LEN);
error = SSL_get_error(con, read_data_len);
switch (error) {
.
.
}
So as you can see I have tried number of ways to get the thread performing SSL_read to terminate in response to close, but I didnt get it to work as I expected. Did anybody get to make SSL_read work properly? Is non-blocking socket only solution to my problem? For blocking socket how do you solve the problem of quitting from SSL_read if you never get a response for command? Can you give an example of working solution for non blocking socket with read?
I can point you to a working example of non-blocking client socket with SSL ... https://github.com/darrenjs/openssl_examples
It uses non-blocking sockets with standard linux IO (based on poll event loop). Raw data is read from the socket and then fed into SSL memory BIO's, which then perform the decryption.
The approach I used was single threaded. A single thread performs the connect, write, and read. This means there cannot be any problems associated with one thread closing a socket, while another thread is trying to use that socket. Also, as noted by the SSL FAQ, "an SSL connection cannot be used concurrently by multiple threads" (https://www.openssl.org/docs/faq.html#PROG1), so single threaded approach avoids problems with concurrent SSL write & read.
The challenge with single threaded approach is that you then need to create some kind of synchronized queue & signalling mechanism for submitting and holding data pending for outbound (eg, the commands that you want to send from client to server), and get the socket event loop to detect when there is data pending for write and pull it from the queue etc. For that I would would look at standard std::list, std::mutex etc, and either pipe2 or eventfd for signalling the event loop.
OpenSSL calls recv() which in turn obeys the SOCKET's timeout, which by default is infinite. You can change the timeout thusly:
void socket_timeout_receive_set(SOCKET handle, dword milliseconds)
{
if(handle==SOCKET_HANDLE_NULL)
return;
struct timeval tv = { long(milliseconds / 1000), (milliseconds % 1000) * 1000 };
setsockopt(handle, SOL_SOCKET, SO_RCVTIMEO, (char *)&tv, sizeof(tv));
}
Unfortunately, ssl_error_get() returns SSL_ERROR_SYSCALL which it returns in other situations too, so it's not easy to determine that it timed out. But this function will help you determine if the connection is lost:
bool socket_dropped(SOCKET handle)
{
// Special thanks: "Detecting and terminating aborted TCP/IP connections" by Vinayak Gadkari
if(handle==SOCKET_HANDLE_NULL)
return true;
// create a socket set containing just this socket
fd_set socket_set;
FD_ZERO(&socket_set);
FD_SET(handle, &socket_set);
// if the connection is unreadable, it is not dropped (strange but true)
static struct timeval timeout = { 0, 0 };
int count = select(0, &socket_set, NULL, NULL, &timeout);
if(count <= 0) {
// problem: count==0 on a connection that was cut off ungracefully, presumably by a busy router
// for connections that are open for a long time but may not talk much, call keepalive_set()
return false;
}
if(!FD_ISSET(handle, &socket_set)) // creates a dependency on __WSAFDIsSet()
return false;
// peek at the next character
// recv() returns 0 if the connection was dropped
char dummy;
count = recv(handle, &dummy, 1, MSG_PEEK);
if(count > 0)
return false;
if(count==0)
return true;
return sec==WSAECONNRESET || sec==WSAECONNABORTED || sec==WSAENETRESET || sec==WSAEINVAL;
}
I am using the preforking concept.
When I establish a socket in the server, the file descriptor value returned is 7. I know that it will allocate a available number for the file descriptor.
When I made the same child process to accept for new connections on the same socket, It wont accept() the connection.
But when I reset the value of file descriptor to 7, then It starts accepting the connections.
I am not finding the reason behind it. Can anyone through some light on this.
My code looks similar to this
for (;;)
{
int session_fd=accept(server_fd,0,0);
if (session_fd==-1)
{
if (errno==EINTR) continue;
die("failed to accept connection (errno=%d)",errno);
}
handle_session(session_fd);
close(session_fd);
server_fd = 7;
}
When I do some read and write operations, the value of file descriptor goes on increasing.
This doesn't make sense, reading or writing should be performed on the same socket for which the system returned a handle.
So I am just curious to know about the significance of value of socket file descriptor.
There is no significance, it's a process specific handle which is represented by an integer, usually it increases by 1 each time you open and/or create a new socket, etc.
I'm going to go out on a limb and assume your code looks something like this, where you bind server_fd and then fork off a child to deal with each connection.
If so, the value 7 doesn't mean anything. It just so happens that at that stage in your program, the lower numbers are already in use. If you opened some other files or sockets before calling listen, server_fd would have a different number.
A socket is just its number, so when you reset the value of the file descriptor, you're reading from your original, bound socket, not the new one that results from the accept call. And I suspect you don't want to be listening on the same socket twice, though I've never tried.
if (listen(server_fd,SOMAXCONN)) {
die("failed to listen for connections (errno=%d)",errno);
}
for (;;) {
int session_fd=accept(server_fd,0,0);
if (session_fd==-1) {
if (errno==EINTR) continue;
die("failed to accept connection (errno=%d)",errno);
}
pid_t pid=fork();
if (pid==-1) {
die("failed to create child process (errno=%d)",errno);
} else if (pid==0) {
close(server_fd);
handle_session(session_fd);
close(session_fd);
_exit(0);
} else {
close(session_fd);
}
}
Sorry for bothering you all. I found the reason behind the issue with change of session_fd value.
session_fd is a global variable. Since it's a huge legacy code, I was not aware that the value of session_fd was changed by some other socket.
Thanks all for providing your inputs :)
I wrote simple TCP/IP multi-thread ANSI C server (client is C sharp), everything works fine except when the server doesnt receive proper signal from client it wont end the thread and close its socket (for example when client crash). Eventually it could become problem if those threads accumulate.
I got threads stored in Linked List - iterating through them isnt a problem. However they are all blocked by recv() by default and since dead client wont send anything they become stuck in memory.
What is the proper way of maintaining list of online clients? (or how to detect threads with broken connection).
struct tListItem {
pthread_t thisThread;
char* name;
int c_sockfd;
int run;
tListItem* next;
tListItem* prev;};
struct tList{
tListItem* head;
int count;};
code of thread:
while(param->run)
{
bzero(&buf, sizeof(buf));
if ((readLen = recv(param->c_sockfd, buf, BUFFSIZE, 0)) == -1)
{
perror("Read error");
param->run = 0;
}
else if (readLen > 0) {
printf("%s: %s \n", param->name, buf);
parseIncoming(param->c_sockfd, param, buf);}}
and here is my attempt to detect broken connection, but this causes the server to end with no message:
void* maintenance() {
tListItem *item;
char buf[4] = "PNG";
while(1)
{
usleep(2000000);
item= threadList->head;
while(item != 0)
{
if ((send(item->c_sockfd, buf, 3, NULL)) == -1)
{
perror("Write error");
item->run = 0;
}
item = item->next;
}
}
}
There's a few common ways this is dealt with:
Implement a heartbeat/ping-pong in your protocol on top of TCP. That is, periodically the client and/or server
sends a heartbeat message to the other end. If the server has not received any data or heartbeat messages within a period of time, e.g. two times the heartbeat period, or if sending the heartbeat message from the server fails, then consider the connection to be dead and close it.
Implement an overall data timeout. Each time the server receives data, you read time current time. Periodically you check the connection for when you last received data, and time out/close connections that haven't received data in a while.
Enable TCP keepalive. This is basically a last resort if you cannot do either 1. or 2.. It'll help you detect dead peers, as the TCP keepalives will break the connection if the peer cannot be reached. (Though it will not help you detect idle clients). Note that the default for keepalives is in the order of hours.
In all cases you should always to be read()/recv() or otherwise monitoring the socket for read events so you can learn as quick as possible if the connection actively breaks.
It's also quite hard to implement this if you're doing blocking read()/recv() calls, you would normally need to set a timeout on the read() so you can wake up periodically and send a heartbeat message or check if the client has been idle for too long - this is best done by using select()/poll() or the like so you can get a timeout instead of doing a block read() that might never return.
I have the following situation. My server receives data from remote server (fd_server) and forwards it to the client (fd_client). I'm using edge triggered epoll so I can handle multiple clients and multiple server conncetions.
Procedure:
client connects to the server.
my server connects to the remote server and requests data.
remote server responds and my server forwards data to the client.
Details:
After my server connects to the remote server the fd_server is added to epoll control with EPOLLIN flag. Server waits for events.
When epoll_wait return the fd_server as readable I go in the following loop displayed bellow.
After some read/writes my sctp_sendmsg return EAGAIN, which means sctp send buffer is full. How should I handle this situation without loosing the data I have already read from the fd_server socket?
IS there a way of knowing before hand, how much data can I send, so I only read the right amount?
while(1){
N = recv(fd_server,buf, sizeof buf,0);
if (N == -1){
/* If errno == EAGAIN, that means we have read all
data. So go back to the main loop. */
if (errno != EAGAIN){
perror ("tcp_recv error");
}
break;
}
if(N == 0){
/* End of file. The remote has closed the
connection. */
close(fd_server);
break;
}
pos = 0;
while(pos < N){
got = sctp_sendmsg(fd_client, &buf[pos], N-pos, to, tolen, 0, 0, stream, 0, 0);
if(got<=0){
if (errno == EAGAIN){
//what to do?
}else{
perror("tcp to sctp send error");
}
}
else{
pos += got;}
}
}
After some read/writes my sctp_sendmsg return EAGAIN, which means sctp send buffer is full. How should I handle this situation without losing the data I have already read from the fd_server socket?
You need to keep some sort of "context" (data structure) for each fd_client socket. For each new client socket that gets connected to your server, create an instance of a "connection state" struct and store it in a hash table. This will be something like the following:
struct ConnectionState
{
int fd_client; // socket
uint8_t buffer[MAX_CHUNK_SIZE]; // protocol buffer for this connection
int buffer_length; // how many bytes received into this buffer
int pos; // how many bytes transmitted back out on fd_client from "buffer"
int has_data; // boolean to indicate protocol state (1 if there's still data in buffer to send)
};
If you can't send everything at once, toggle the fd_client socket from EPOLLIN to EPOLLOUT in your epoll mode. Change "has_data" to true in the ConnectionState structure. Then go back to waiting for socket events. When you are able to send again, you look at your ConnectionState struct for that socket to decide if you still need to keep sending or receive a new buffer.
Be careful with edge triggered sockets. When you do transition from EPOLLOUT back to EPOLLIN, you need to go ahead and recv() again just to make sure you don't lose any data. (Similarly for entering the send state, try an initial send).