This time I code a chat server and client. The idea is this.
The server uses the select method for with a readfd(FD_SET) to seek which of the clients connected on it have something to send. If it founds something it send it to the rest of clients.
Here is the select function on server and a part of server's code.
SelectResults = select(maxDescriptor+1,&BackUpfdread,NULL,NULL,&time);
I use the select function in client too, to make it seek for incoming and outcoming messages.(these that server sends from other clients and these that this client want to send).
Well, the client in the select function has a fdread and fdwrite (FD_SET).
Here is the part of the code that client has for the connection and for the chat.
The problem is that if i connect two clients on the server message transfer isn't concurrent and that means that clients cannot chat correctly.
Finally i thought to use threads in server. One thread for waiting to receive and one for sending to the clients, but i want to hear and your opinion.
In the client you don't really need that loop from 0 to maxDescriptor. Just check if ConnectSocket is set. Something like this:
// Main loop starts here
for(; ;)
{
memset(SentBuff, 0, sizeof(SentBuff));
printf("Write: ");
gets_s(SentBuff, sizeof(SentBuff));
// Copy the fdread into BackUpfdread and fdwrite to BackUpfdwrite.
BackUpfdread = fdread;
BackUpfdwrite = fdwrite;
SelectResults = select(maxDescriptor+1,&BackUpfdread,&BackUpfdwrite,NULL,&timer);
if(SelectResults == -1)
{
perror("Client-select() error!\n");
exit(1);
}
if (FD_ISSET(ConnectSocket, &BackUpfdread))
{
RecvBytes = recv(ConnectSocket, RecvBuff, sizeof(RecvBuff), 0);
if(RecvBytes > 0)
{
printf("%s\n",RecvBuff);
// Cleaning the Receive Buffer
memset(RecvBuff,0,sizeof(RecvBuff));
}
}
if (FD_ISSET(ConnectSocket, &BackUpfdwrite))
{
SentBytes = send(ConnectSocket, SentBuff,sizeof(SentBuff),0);
// Cleaning the Sent Buffer
memset(SentBuff,0,sizeof(SentBuff));
}
} // Main loop ends here
Also don't forget to check for errors from send and recv. Especially recv is important, as it's the call that will tell you the server has disconnected.
Edit: Another important thing to note, is that the socket may be writeable always, so add a check if there is something to write before you check if the socket is writeable.
Related
Before I Start
Please don't mark this question as a duplicate. I have already seen the numerous posts on SO about handling multiple clients with socket programming. Most people recommend just multi-threading, but I am trying to avoid that path because I have read it has a few problems:
Bad Scalability
Large Overhead/Inefficient/Memory Hungry
Difficult to Debug
Any posts that I have read that specifically talk about using a single thread either have bad/no answers or have unclear explanations, like people saying "Just use select()!"
The Problem
I am writing code for a server to handle multiple (~1000) clients, and I'm having trouble figuring out how to create an efficient solution. Right now I already have the code for my server that is able to handle 1 client at a time. Both are written in C; the server is on Windows using WinSock and the client is on Linux.
The server and client send several communications back and forth, using send() and blocking recv() calls. Writing this code was pretty simple, and I won't post it here because it is pretty long and I doubt anyone will actually read through all of it. Also the exact implementation is not important, I just want to talk about high level pseudocode. The real difficulty is changing the server to handle multiple clients.
What's Already Out There
I have found a nice PDF tutorial about how to create a WinSock server that handles multiple clients and it can be found here: WinSock Multiple Client Support. It's in C++ but it's easily transferable to C.
From what I understand the server operates something like this:
while (running) {
Sleep(1000);
/* Accept all incoming clients and add to clientArray. */
for (client in clientArray) {
/* Interact with client */
if (recv(...) == "disconnect") {
/* Disconnect from client */
}
}
}
/* Close all connections. */
The problem that I see with using this approach is that you essentially only handle one client at a time (which is obvious because you aren't multithreading), but what if the interaction with each client only needs to happen once? Meaning, what if I just want to send some data back and forth and close the connection? This operation could take anywhere from 5 seconds to 5 minutes depending on the speed of the clients connection, so other clients would be blocking on a connect() call to the server while the server handles a client for 5 minutes. It doesn't seem very efficient, but maybe the best way would be to implement a waiting queue, where clients are connected and told to wait for a while? I'm not sure, but it makes me curious about how large servers send out update downloads concurrently to thousands of clients, and if I should operate the same way.
Also, is there a reason for adding a Sleep(1000) call in the main server loop, if the send() and recv() between the server and client take a while (~1 minute)?
What I'm Asking For
What I want is a solution to handling multiple clients on a single threaded server that is efficient enough for ~1000 clients. If you tell me that the solution in the PDF is fine, that's good enough for me (maybe I'm just too preoccupied with efficiency.)
Please give answers that include a verbal explanation of the implementation, server/client pseudocode, or even a small sample code for the server, if you're feeling sadistic.)
Thanks in advance.
I have written single thread socket pool handling. Im using non-blocking sockets and select call to handle all send, receive and errors.
My class keep all sockets in array, and build 3 fd set's for select call. When something happens it check read or write or error list and handle those events.
For example, non-blocking client socket during connection can trigger write or error event. If error event happens then connection failed. If write happens, connection is established.
All sockets is in read fd set. If you create server socket (with bind and listen) new connection will trigger read event. Then check if socket is server socket then call accept for new connection. If read operation is triggered by regular socket then there is some bytes to read.. just call recv with buffer arge enough to suck all data from that socket.
SOCKET maxset=0;
fd_set rset, wset, eset;
FD_ZERO(&rset);
FD_ZERO(&wset);
FD_ZERO(&eset);
for (size_t i=0; i<readsockets.size(); i++)
{
SOCKET s = readsockets[i]->s->GetSocket();
FD_SET(s, &rset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<writesockets.size(); i++)
{
SOCKET s = writesockets[i]->s->GetSocket();
FD_SET(s, &wset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<errorsockets.size(); i++)
{
SOCKET s = errorsockets[i]->s->GetSocket();
FD_SET(s, &eset);
if (s > maxset) maxset = s;
}
int ret = 0;
if (bBlocking)
ret = select(maxset + 1, &rset, &wset, &eset, NULL/*&tv*/);
else
{
timeval tv= {0, timeout*1000};
ret = select(maxset + 1, &rset, &wset, &eset, &tv);
}
if (ret < 0)
{
//int err = errno;
NetworkCheckError();
return false;
}
if (ret > 0)
{
// loop through eset and check each with FD_ISSET. if you find some socket it means connect failed
// loop through wset and check each with FD_ISSET. If you find some socket check is there any pending connectin on that socket. If there is pending connection then that socket just got connected. Otherwise select just reported that some data has been sent and you can send more.
// finally, loop through rset and check each with FD_ISSET. If you find some socket then check is this socket your server socket (bind and listen). If its server socket then this is signal new client want to connect.. just call accept and new connection is established. If this is not server socket, then just do recv on that socket to collect new data.
}
There is few more things to handle... All sockets must be in non-blocking mode. Each send or recv calls will return -1 (error) but error code is EWOULDBLOCK. Thats normal and ignore error. If recv returns 0 then this connection is dropped. If send return 0 bytes sent then internal buffer is full.
You need to write additional code to serialize and parse data. For example, after recv, message may not be complete (depending on message size) so it may take more than one recv calls to receive complete message. Sometimes if messages is short recv call can deliver several messages in buffer. So, you need to write good parser or design good protocol, easy to parse.
First, regarding single-thread approach: I'd say it's bad idea because your server processing power is limited by performance of single processor core. But other than that it'll work to some extent.
Now about multiclient problem. I'd suggest using WSASend and WSARecv with their compilation routines. It also can be scaled to multiple threads if necessary.
Server core will look something like this:
struct SocketData {
::SOCKET socket;
::WSAOVERLAPPED overlapped;
::WSABUF bufferRef;
char buf [1024];
// other client-related data
SocketData (void) {
overlapped->hEvent = (HANDLE) this;
bufferRef->buf = buf;
bufferRef->len = sizeof (buf);
// ...
}
};
void OnRecv (
DWORD dwError,
DWORD cbTransferred,
LPWSAOVERLAPPED lpOverlapped,
DWORD dwFlags) {
auto data = (SocketData*) lpOverlapped->hEvent;
if (dwError || !cbTransferred) {
::closesocket (data->socket);
delete data;
return;
}
// process received data
// ...
}
// same for OnSend
void main (void) {
// init and start async listener
::SOCKET serverSocket = ::socket (...);
HANDLE hAccept = ::CreateEvent (nullptr, 0, 0, nullptr);
::WSAEventSelect (serverSocket, FD_ACCEPT, hAccept);
::bind (serverSocket, ...);
::listen (serverSocket, ...);
// main loop
for (;;) {
int r = ::WaitForSingleObjectEx (hAccept, INFINITE, 1);
if (r == WAIT_IO_COMPLETION)
continue;
// accept processing
auto data = new SocketData ();
data->socket = ::accept (serverSocket, ...);
// detach new socket from hAccept event
::WSAEventSelect (data->socket, 0, nullptr);
// recv first data from client
::WSARecv (
data->socket,
&data->bufferRef,
1,
nullptr,
0,
&data->overlapped,
&OnRecv);
}
}
Key points:
wait in main loop (WaitForSingleObjectEx, WaitForMultipleObjectsEx etc.) must be alertable;
most data processing done in OnSend/OnRecv;
all processing must be done without blocking APIs in OnSend/OnRecv;
for event-based processing events must be waited in main loop.
OnRecv will be called for each processed incoming packet. OnSend will be called for each processed outgoing packet. Keep in mind: how many data you asked to send/recv is not the same as what actually processed in packet.
I wrote simple TCP/IP multi-thread ANSI C server (client is C sharp), everything works fine except when the server doesnt receive proper signal from client it wont end the thread and close its socket (for example when client crash). Eventually it could become problem if those threads accumulate.
I got threads stored in Linked List - iterating through them isnt a problem. However they are all blocked by recv() by default and since dead client wont send anything they become stuck in memory.
What is the proper way of maintaining list of online clients? (or how to detect threads with broken connection).
struct tListItem {
pthread_t thisThread;
char* name;
int c_sockfd;
int run;
tListItem* next;
tListItem* prev;};
struct tList{
tListItem* head;
int count;};
code of thread:
while(param->run)
{
bzero(&buf, sizeof(buf));
if ((readLen = recv(param->c_sockfd, buf, BUFFSIZE, 0)) == -1)
{
perror("Read error");
param->run = 0;
}
else if (readLen > 0) {
printf("%s: %s \n", param->name, buf);
parseIncoming(param->c_sockfd, param, buf);}}
and here is my attempt to detect broken connection, but this causes the server to end with no message:
void* maintenance() {
tListItem *item;
char buf[4] = "PNG";
while(1)
{
usleep(2000000);
item= threadList->head;
while(item != 0)
{
if ((send(item->c_sockfd, buf, 3, NULL)) == -1)
{
perror("Write error");
item->run = 0;
}
item = item->next;
}
}
}
There's a few common ways this is dealt with:
Implement a heartbeat/ping-pong in your protocol on top of TCP. That is, periodically the client and/or server
sends a heartbeat message to the other end. If the server has not received any data or heartbeat messages within a period of time, e.g. two times the heartbeat period, or if sending the heartbeat message from the server fails, then consider the connection to be dead and close it.
Implement an overall data timeout. Each time the server receives data, you read time current time. Periodically you check the connection for when you last received data, and time out/close connections that haven't received data in a while.
Enable TCP keepalive. This is basically a last resort if you cannot do either 1. or 2.. It'll help you detect dead peers, as the TCP keepalives will break the connection if the peer cannot be reached. (Though it will not help you detect idle clients). Note that the default for keepalives is in the order of hours.
In all cases you should always to be read()/recv() or otherwise monitoring the socket for read events so you can learn as quick as possible if the connection actively breaks.
It's also quite hard to implement this if you're doing blocking read()/recv() calls, you would normally need to set a timeout on the read() so you can wake up periodically and send a heartbeat message or check if the client has been idle for too long - this is best done by using select()/poll() or the like so you can get a timeout instead of doing a block read() that might never return.
I am writing a simple web server and client using UDP and so far: the programs can connect to each other, the client can send a request, the server can read the request, the server can recognize the client's IP address and client's port, and the server can send a message back to the client
My problem is that my client code gets stuck waiting in the rcvfrom function, even after the server has sent a response.
Here is the function that is supposed to pick up the server message and return the number of bytes read by the socket:
ssize_t receive_from_server(rdp_socket *rsocket, char *buffer, size_t buf_len){
socklen_t sendsize = sizeof(rsocket->server_addr);
bzero(&(rsocket->server_addr), sendsize);
//STUCK HERE:
return recvfrom(rsocket->sockfd, buffer, buf_len, 0,
(struct sockaddr*)&(rsocket->server_addr), &sendsize);
}
I set the sockopts for both SO_SNDTIMEO and SO_RCVTIMEO to timeout after a few seconds.
Question:
In the short term future I will be adding acknowledgements (ACKs) for reliable data transfer. I imagine that missing ACKs could be the issue but I'm just wondering if, to the trained eye, it looks like a different problem.
Are ACKs necessary for a timeout to work?
How can I synchronize my client and server so that they can actually communicate with each other?
Since UDP does not provide reliability, you will need to implement retransmission of missing data. Since it looks like this is a client request server response model, the easiest retransmission implementation for you may be to resend the request when you time out waiting for the response, and wait for the response again. You may want to implement a retry counter and give up after a certain number of retries.
If the SO_RCVTIMEO and SO_SNDTIMEO socket options do not seem to be taking effect, it may be those options are not implemented for that type of socket. Check the return value of the setsockopt()call to make sure they succeeded.
As a workaround, you can change your receive_from_server() function to use poll() or select() to wait for a readable event for some amount of time, instead of blocking in recvfrom().
ssize_t receive_from_server(rdp_socket *rsocket, char *buffer, size_t buf_len){
struct pollfd pfd = { rsocket->sockfd, POLLIN };
int pollresult = poll(&pfd, 1, RECV_TIMEOUT_SECONDS * 1000);
if (pollresult > 0) {
socklen_t sendsize = sizeof(rsocket->server_addr);
bzero(&(rsocket->server_addr), sendsize);
return recvfrom(rsocket->sockfd, buffer, buf_len, MSG_DONTWAIT,
(struct sockaddr*)&(rsocket->server_addr), &sendsize);
}
if (pollresult == 0) {
errno = ETIME;
}
return -1;
}
I have the following situation. My server receives data from remote server (fd_server) and forwards it to the client (fd_client). I'm using edge triggered epoll so I can handle multiple clients and multiple server conncetions.
Procedure:
client connects to the server.
my server connects to the remote server and requests data.
remote server responds and my server forwards data to the client.
Details:
After my server connects to the remote server the fd_server is added to epoll control with EPOLLIN flag. Server waits for events.
When epoll_wait return the fd_server as readable I go in the following loop displayed bellow.
After some read/writes my sctp_sendmsg return EAGAIN, which means sctp send buffer is full. How should I handle this situation without loosing the data I have already read from the fd_server socket?
IS there a way of knowing before hand, how much data can I send, so I only read the right amount?
while(1){
N = recv(fd_server,buf, sizeof buf,0);
if (N == -1){
/* If errno == EAGAIN, that means we have read all
data. So go back to the main loop. */
if (errno != EAGAIN){
perror ("tcp_recv error");
}
break;
}
if(N == 0){
/* End of file. The remote has closed the
connection. */
close(fd_server);
break;
}
pos = 0;
while(pos < N){
got = sctp_sendmsg(fd_client, &buf[pos], N-pos, to, tolen, 0, 0, stream, 0, 0);
if(got<=0){
if (errno == EAGAIN){
//what to do?
}else{
perror("tcp to sctp send error");
}
}
else{
pos += got;}
}
}
After some read/writes my sctp_sendmsg return EAGAIN, which means sctp send buffer is full. How should I handle this situation without losing the data I have already read from the fd_server socket?
You need to keep some sort of "context" (data structure) for each fd_client socket. For each new client socket that gets connected to your server, create an instance of a "connection state" struct and store it in a hash table. This will be something like the following:
struct ConnectionState
{
int fd_client; // socket
uint8_t buffer[MAX_CHUNK_SIZE]; // protocol buffer for this connection
int buffer_length; // how many bytes received into this buffer
int pos; // how many bytes transmitted back out on fd_client from "buffer"
int has_data; // boolean to indicate protocol state (1 if there's still data in buffer to send)
};
If you can't send everything at once, toggle the fd_client socket from EPOLLIN to EPOLLOUT in your epoll mode. Change "has_data" to true in the ConnectionState structure. Then go back to waiting for socket events. When you are able to send again, you look at your ConnectionState struct for that socket to decide if you still need to keep sending or receive a new buffer.
Be careful with edge triggered sockets. When you do transition from EPOLLOUT back to EPOLLIN, you need to go ahead and recv() again just to make sure you don't lose any data. (Similarly for entering the send state, try an initial send).
The situation: I am creating a server daemon in c that accepts numerous simultaneous connections, and the clients will be sending data to the server. I currently have each client connection being spawned into a new thread.
The problem: if a client sends numerous lines of content very quickly (eg, 10 lines of data in less than a second), the server will see the first two lines, but not the rest.
The question: How can I "queue" the data coming in from the clients (the recv command in c)? Is this something that select or poll would be needed for? Basically, I want to make sure any client can send large amounts of data very quickly without having to worry about any content being dropped. How can this be achieved?
Sample Code: (note: the below code has obviously been heavily modified, esp. by removing error checking. I tried to modify my code so as to make the problem/solution clear without getting bogged down in semantics of irrelevant parts. Please don't get caught up with any non-standard or missing elements here)
//this function handles the threads
void *ThreadedFunction(void *arg) {
// do some stuff, like: pull vars out of mystruct
int nbytes;
char buf[256];
while(1) {
if((nbytes=recv(conid, buf, sizeof buf, 0)) <= 0) {
//handle break in connection
} else {
//for this example, just print out data from client to make my point
buf[nbytes] = 0;
printf("%s\n",buf);
}
}
}
//main just sets up the connections and creates threads
int main(int argc. char *argv[])
{
// bind(), listen(), etc... blah blah blah
while(1) {
conid = accept(...); //get a connection
// ... build mystruct to pass vars to threaded function ...
pthread_t p;
pthread_create(&p,NULL,ThreadedFunction,&mystruct); //create new thread
}
}
You don't need to "queue" the data coming in from the clients.
Because TCP do that for you. Flow control of TCP even slows down clients, if the server is too slow to make space to TCP receiving buffer.
So, probably there is bug in the code of server or client. Maybe client sends '\0' in the end of each line. In that case, the following code would not print all lines:
if((nbytes=recv(conid, buf, sizeof buf, 0)) <= 0) {
//handle break in connection
} else {
//for this example, just print out data from client to make my point
buf[nbytes] = 0;
printf("%s\n",buf);
}
It is even expected that the 2nd line is the last line what you see, if client sends '\0' at the end of each line.
For example:
If client sends the following lines:
"abc\n\0"
"def\n\0"
"ghi\n\0"
TCP will usually send those by using two packets, that contains following:
"abc\n\0"
"def\n\0ghi\n\0"
Server usually needs 2 recv calls to receive the incoming data.
So your server will use 2 print calls:
printf("%s\n", "abc\n\0\0");
printf("%s\n", "def\n\0ghi\n\0\0");
And the result output is:
abc
def