The situation: I am creating a server daemon in c that accepts numerous simultaneous connections, and the clients will be sending data to the server. I currently have each client connection being spawned into a new thread.
The problem: if a client sends numerous lines of content very quickly (eg, 10 lines of data in less than a second), the server will see the first two lines, but not the rest.
The question: How can I "queue" the data coming in from the clients (the recv command in c)? Is this something that select or poll would be needed for? Basically, I want to make sure any client can send large amounts of data very quickly without having to worry about any content being dropped. How can this be achieved?
Sample Code: (note: the below code has obviously been heavily modified, esp. by removing error checking. I tried to modify my code so as to make the problem/solution clear without getting bogged down in semantics of irrelevant parts. Please don't get caught up with any non-standard or missing elements here)
//this function handles the threads
void *ThreadedFunction(void *arg) {
// do some stuff, like: pull vars out of mystruct
int nbytes;
char buf[256];
while(1) {
if((nbytes=recv(conid, buf, sizeof buf, 0)) <= 0) {
//handle break in connection
} else {
//for this example, just print out data from client to make my point
buf[nbytes] = 0;
printf("%s\n",buf);
}
}
}
//main just sets up the connections and creates threads
int main(int argc. char *argv[])
{
// bind(), listen(), etc... blah blah blah
while(1) {
conid = accept(...); //get a connection
// ... build mystruct to pass vars to threaded function ...
pthread_t p;
pthread_create(&p,NULL,ThreadedFunction,&mystruct); //create new thread
}
}
You don't need to "queue" the data coming in from the clients.
Because TCP do that for you. Flow control of TCP even slows down clients, if the server is too slow to make space to TCP receiving buffer.
So, probably there is bug in the code of server or client. Maybe client sends '\0' in the end of each line. In that case, the following code would not print all lines:
if((nbytes=recv(conid, buf, sizeof buf, 0)) <= 0) {
//handle break in connection
} else {
//for this example, just print out data from client to make my point
buf[nbytes] = 0;
printf("%s\n",buf);
}
It is even expected that the 2nd line is the last line what you see, if client sends '\0' at the end of each line.
For example:
If client sends the following lines:
"abc\n\0"
"def\n\0"
"ghi\n\0"
TCP will usually send those by using two packets, that contains following:
"abc\n\0"
"def\n\0ghi\n\0"
Server usually needs 2 recv calls to receive the incoming data.
So your server will use 2 print calls:
printf("%s\n", "abc\n\0\0");
printf("%s\n", "def\n\0ghi\n\0\0");
And the result output is:
abc
def
Related
I'm building a multi-client<->server messaging application over TCP.
I created a non blocking server using epoll to multiplex linux file descriptors.
When a fd receives data, I read() /or/ recv() into buf.
I know that I need to either specify a data length* at the start of the transmission, or use a delimiter** at the end of the transmission to segregate the messages.
*using a data length:
char *buffer_ptr = buffer;
do {
switch (recvd_bytes = recv(new_socket, buffer_ptr, rem_bytes, 0)) {
case -1: return SOCKET_ERR;
case 0: return CLOSE_SOCKET;
default: break;
}
buffer_ptr += recvd_bytes;
rem_bytes -= recvd_bytes;
} while (rem_bytes != 0);
**using a delimiter:
void get_all_buf(int sock, std::string & inStr)
{
int n = 1, total = 0, found = 0;
char c;
char temp[1024*1024];
// Keep reading up to a '\n'
while (!found) {
n = recv(sock, &temp[total], sizeof(temp) - total - 1, 0);
if (n == -1) {
/* Error, check 'errno' for more details */
break;
}
total += n;
temp[total] = '\0';
found = (strchr(temp, '\n') != 0);
}
inStr = temp;
}
My question: Is it OK to loop over recv() until one of those conditions is met? What if a client sends a bogus message length or no delimiter or there is packet loss? Wont I be stuck looping recv() in my program forever?
Is it OK to loop over recv() until one of those conditions is met?
Probably not, at least not for production-quality code. As you suggested, the problem with looping until you get the full message is that it leaves your thread at the mercy of the client -- if a client decides to only send part of the message and then wait for a long time (or even forever) without sending the last part, then your thread will be blocked (or looping) indefinitely and unable to serve any other purpose -- usually not what you want.
What if a client sends a bogus message length
Then you're in trouble (although if you've chosen a maximum-message-size you can detect obviously bogus message-lengths that are larger than that size, and defend yourself by e.g. forcibly closing the connection)
or there is packet loss?
If there is a reasonably small amount of packet loss, the TCP layer will automatically retransmit the data, so your program won't notice the difference (other than the message officially "arriving" a bit later than it otherwise would have). If there is really bad packet loss (e.g. someone pulled the Ethernet cable out of the wall for 5 minutes), then the rest of the message might be delayed for several minutes or more (until connectivity recovers, or the TCP layer gives up and closes the TCP connection), trapping your thread in the loop.
So what is the industrial-grade, evil-client-and-awful-network-proof solution to this dilemma, so that your server can remain responsive to other clients even when a particular client is not behaving itself?
The answer is this: don't depend on receiving the entire message all at once. Instead, you need to set up a simple state-machine for each client, such that you can recv() as many (or as few) bytes from that client's TCP socket as it cares to send to you at any particular time, and save those bytes to a local (per-client) buffer that is associated with that client, and then go back to your normal event loop even though you haven't received the entire message yet. Keep careful track of how many valid received-bytes-of-data you currently have on-hand from each client, and after each recv() call has returned, check to see if the associated per-client incoming-data-buffer contains an entire message yet, or not -- if it does, parse the message, act on it, then remove it from the buffer. Lather, rinse, and repeat.
Before I Start
Please don't mark this question as a duplicate. I have already seen the numerous posts on SO about handling multiple clients with socket programming. Most people recommend just multi-threading, but I am trying to avoid that path because I have read it has a few problems:
Bad Scalability
Large Overhead/Inefficient/Memory Hungry
Difficult to Debug
Any posts that I have read that specifically talk about using a single thread either have bad/no answers or have unclear explanations, like people saying "Just use select()!"
The Problem
I am writing code for a server to handle multiple (~1000) clients, and I'm having trouble figuring out how to create an efficient solution. Right now I already have the code for my server that is able to handle 1 client at a time. Both are written in C; the server is on Windows using WinSock and the client is on Linux.
The server and client send several communications back and forth, using send() and blocking recv() calls. Writing this code was pretty simple, and I won't post it here because it is pretty long and I doubt anyone will actually read through all of it. Also the exact implementation is not important, I just want to talk about high level pseudocode. The real difficulty is changing the server to handle multiple clients.
What's Already Out There
I have found a nice PDF tutorial about how to create a WinSock server that handles multiple clients and it can be found here: WinSock Multiple Client Support. It's in C++ but it's easily transferable to C.
From what I understand the server operates something like this:
while (running) {
Sleep(1000);
/* Accept all incoming clients and add to clientArray. */
for (client in clientArray) {
/* Interact with client */
if (recv(...) == "disconnect") {
/* Disconnect from client */
}
}
}
/* Close all connections. */
The problem that I see with using this approach is that you essentially only handle one client at a time (which is obvious because you aren't multithreading), but what if the interaction with each client only needs to happen once? Meaning, what if I just want to send some data back and forth and close the connection? This operation could take anywhere from 5 seconds to 5 minutes depending on the speed of the clients connection, so other clients would be blocking on a connect() call to the server while the server handles a client for 5 minutes. It doesn't seem very efficient, but maybe the best way would be to implement a waiting queue, where clients are connected and told to wait for a while? I'm not sure, but it makes me curious about how large servers send out update downloads concurrently to thousands of clients, and if I should operate the same way.
Also, is there a reason for adding a Sleep(1000) call in the main server loop, if the send() and recv() between the server and client take a while (~1 minute)?
What I'm Asking For
What I want is a solution to handling multiple clients on a single threaded server that is efficient enough for ~1000 clients. If you tell me that the solution in the PDF is fine, that's good enough for me (maybe I'm just too preoccupied with efficiency.)
Please give answers that include a verbal explanation of the implementation, server/client pseudocode, or even a small sample code for the server, if you're feeling sadistic.)
Thanks in advance.
I have written single thread socket pool handling. Im using non-blocking sockets and select call to handle all send, receive and errors.
My class keep all sockets in array, and build 3 fd set's for select call. When something happens it check read or write or error list and handle those events.
For example, non-blocking client socket during connection can trigger write or error event. If error event happens then connection failed. If write happens, connection is established.
All sockets is in read fd set. If you create server socket (with bind and listen) new connection will trigger read event. Then check if socket is server socket then call accept for new connection. If read operation is triggered by regular socket then there is some bytes to read.. just call recv with buffer arge enough to suck all data from that socket.
SOCKET maxset=0;
fd_set rset, wset, eset;
FD_ZERO(&rset);
FD_ZERO(&wset);
FD_ZERO(&eset);
for (size_t i=0; i<readsockets.size(); i++)
{
SOCKET s = readsockets[i]->s->GetSocket();
FD_SET(s, &rset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<writesockets.size(); i++)
{
SOCKET s = writesockets[i]->s->GetSocket();
FD_SET(s, &wset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<errorsockets.size(); i++)
{
SOCKET s = errorsockets[i]->s->GetSocket();
FD_SET(s, &eset);
if (s > maxset) maxset = s;
}
int ret = 0;
if (bBlocking)
ret = select(maxset + 1, &rset, &wset, &eset, NULL/*&tv*/);
else
{
timeval tv= {0, timeout*1000};
ret = select(maxset + 1, &rset, &wset, &eset, &tv);
}
if (ret < 0)
{
//int err = errno;
NetworkCheckError();
return false;
}
if (ret > 0)
{
// loop through eset and check each with FD_ISSET. if you find some socket it means connect failed
// loop through wset and check each with FD_ISSET. If you find some socket check is there any pending connectin on that socket. If there is pending connection then that socket just got connected. Otherwise select just reported that some data has been sent and you can send more.
// finally, loop through rset and check each with FD_ISSET. If you find some socket then check is this socket your server socket (bind and listen). If its server socket then this is signal new client want to connect.. just call accept and new connection is established. If this is not server socket, then just do recv on that socket to collect new data.
}
There is few more things to handle... All sockets must be in non-blocking mode. Each send or recv calls will return -1 (error) but error code is EWOULDBLOCK. Thats normal and ignore error. If recv returns 0 then this connection is dropped. If send return 0 bytes sent then internal buffer is full.
You need to write additional code to serialize and parse data. For example, after recv, message may not be complete (depending on message size) so it may take more than one recv calls to receive complete message. Sometimes if messages is short recv call can deliver several messages in buffer. So, you need to write good parser or design good protocol, easy to parse.
First, regarding single-thread approach: I'd say it's bad idea because your server processing power is limited by performance of single processor core. But other than that it'll work to some extent.
Now about multiclient problem. I'd suggest using WSASend and WSARecv with their compilation routines. It also can be scaled to multiple threads if necessary.
Server core will look something like this:
struct SocketData {
::SOCKET socket;
::WSAOVERLAPPED overlapped;
::WSABUF bufferRef;
char buf [1024];
// other client-related data
SocketData (void) {
overlapped->hEvent = (HANDLE) this;
bufferRef->buf = buf;
bufferRef->len = sizeof (buf);
// ...
}
};
void OnRecv (
DWORD dwError,
DWORD cbTransferred,
LPWSAOVERLAPPED lpOverlapped,
DWORD dwFlags) {
auto data = (SocketData*) lpOverlapped->hEvent;
if (dwError || !cbTransferred) {
::closesocket (data->socket);
delete data;
return;
}
// process received data
// ...
}
// same for OnSend
void main (void) {
// init and start async listener
::SOCKET serverSocket = ::socket (...);
HANDLE hAccept = ::CreateEvent (nullptr, 0, 0, nullptr);
::WSAEventSelect (serverSocket, FD_ACCEPT, hAccept);
::bind (serverSocket, ...);
::listen (serverSocket, ...);
// main loop
for (;;) {
int r = ::WaitForSingleObjectEx (hAccept, INFINITE, 1);
if (r == WAIT_IO_COMPLETION)
continue;
// accept processing
auto data = new SocketData ();
data->socket = ::accept (serverSocket, ...);
// detach new socket from hAccept event
::WSAEventSelect (data->socket, 0, nullptr);
// recv first data from client
::WSARecv (
data->socket,
&data->bufferRef,
1,
nullptr,
0,
&data->overlapped,
&OnRecv);
}
}
Key points:
wait in main loop (WaitForSingleObjectEx, WaitForMultipleObjectsEx etc.) must be alertable;
most data processing done in OnSend/OnRecv;
all processing must be done without blocking APIs in OnSend/OnRecv;
for event-based processing events must be waited in main loop.
OnRecv will be called for each processed incoming packet. OnSend will be called for each processed outgoing packet. Keep in mind: how many data you asked to send/recv is not the same as what actually processed in packet.
I am writing a client-server program in c where I have to send multiple image files from the server to the client. Is there any way for the server to know when the client has finished reading from socket, creating the image file locally, and successfully written to it? P.S. I already tried sending a message to the socket and when I try to read the socket from the server, the program hangs. Any help will be much appreciated.
Here is a function from the Server code which sends the file to socket:
while(1)
{
unsigned char buf[256] = {0};
int n = fread(buf,1,256,fp);
if(n>0) { send(sockfd,buf,n,0); }
if(n<256) {
if(feof(fp))
printf("Sent to socket\n");
break;
}
}
fclose(fp);
}
char buf[5]
read(sockfd,buf,5);
if(strcmp(buf,"ready")==0) //send more files
And here is a function from the client to write to the file:
FILE* fp;
fp = fopen(file_path,"ab");
char buf[256];
int num;
int total=0;
while(( num = recv(sockfd,buf,256,0))>0)
{
total+=num;
fwrite(buf,1,num,fp);
}
fclose(fp);
write(sockfd,"ready",5);
}
When I do a read on the server after one file transfer, the program hangs.
You problem is here in the client:
while(( num = recv(sockfd,buf,256,0))>0)
recv() will only return 0 at end-of-file - ie when the server shuts down the sending side of the socket. However your server isn't doing that - it's waiting for a response from the client. This means you deadlock.
If you want to send multiple images in the one connection, you need to send your client some information to allow it to tell when one image ends. One way to do this is to first send your client the size of the file it should expect, then the file data. The client can then keep track of how many bytes it's recieved, and send the "ready" reponse after that.
You need a kind of protocol to allow :
the server to tell the client that all has been written
the client to tell the server that all has been read
If you only send one file, the simplest way is to use shutdown(sockfd, SHUT_WR) server side after all data has beed sent. That way the client will get a 0 as return from recv signaling end of data and will be able to send its acknowledgement. But you can no longer write on the socket server side.
If you want to be able to send more than one file, you will have to imagine a cleverer protocol. A common one would be to send blocs of data preceded by their size
short sz = htons(n); /* deals with possible endianness problems */
send(sockfd, &sz, sizeof(short), 0)
send(sockfd, buf, n);
And a sz == 0 (with no buf ...) would signal end of data
This would still be a simple protocol with no error recovery, but at least it can work when no incident happens.
I wrote simple TCP/IP multi-thread ANSI C server (client is C sharp), everything works fine except when the server doesnt receive proper signal from client it wont end the thread and close its socket (for example when client crash). Eventually it could become problem if those threads accumulate.
I got threads stored in Linked List - iterating through them isnt a problem. However they are all blocked by recv() by default and since dead client wont send anything they become stuck in memory.
What is the proper way of maintaining list of online clients? (or how to detect threads with broken connection).
struct tListItem {
pthread_t thisThread;
char* name;
int c_sockfd;
int run;
tListItem* next;
tListItem* prev;};
struct tList{
tListItem* head;
int count;};
code of thread:
while(param->run)
{
bzero(&buf, sizeof(buf));
if ((readLen = recv(param->c_sockfd, buf, BUFFSIZE, 0)) == -1)
{
perror("Read error");
param->run = 0;
}
else if (readLen > 0) {
printf("%s: %s \n", param->name, buf);
parseIncoming(param->c_sockfd, param, buf);}}
and here is my attempt to detect broken connection, but this causes the server to end with no message:
void* maintenance() {
tListItem *item;
char buf[4] = "PNG";
while(1)
{
usleep(2000000);
item= threadList->head;
while(item != 0)
{
if ((send(item->c_sockfd, buf, 3, NULL)) == -1)
{
perror("Write error");
item->run = 0;
}
item = item->next;
}
}
}
There's a few common ways this is dealt with:
Implement a heartbeat/ping-pong in your protocol on top of TCP. That is, periodically the client and/or server
sends a heartbeat message to the other end. If the server has not received any data or heartbeat messages within a period of time, e.g. two times the heartbeat period, or if sending the heartbeat message from the server fails, then consider the connection to be dead and close it.
Implement an overall data timeout. Each time the server receives data, you read time current time. Periodically you check the connection for when you last received data, and time out/close connections that haven't received data in a while.
Enable TCP keepalive. This is basically a last resort if you cannot do either 1. or 2.. It'll help you detect dead peers, as the TCP keepalives will break the connection if the peer cannot be reached. (Though it will not help you detect idle clients). Note that the default for keepalives is in the order of hours.
In all cases you should always to be read()/recv() or otherwise monitoring the socket for read events so you can learn as quick as possible if the connection actively breaks.
It's also quite hard to implement this if you're doing blocking read()/recv() calls, you would normally need to set a timeout on the read() so you can wake up periodically and send a heartbeat message or check if the client has been idle for too long - this is best done by using select()/poll() or the like so you can get a timeout instead of doing a block read() that might never return.
This time I code a chat server and client. The idea is this.
The server uses the select method for with a readfd(FD_SET) to seek which of the clients connected on it have something to send. If it founds something it send it to the rest of clients.
Here is the select function on server and a part of server's code.
SelectResults = select(maxDescriptor+1,&BackUpfdread,NULL,NULL,&time);
I use the select function in client too, to make it seek for incoming and outcoming messages.(these that server sends from other clients and these that this client want to send).
Well, the client in the select function has a fdread and fdwrite (FD_SET).
Here is the part of the code that client has for the connection and for the chat.
The problem is that if i connect two clients on the server message transfer isn't concurrent and that means that clients cannot chat correctly.
Finally i thought to use threads in server. One thread for waiting to receive and one for sending to the clients, but i want to hear and your opinion.
In the client you don't really need that loop from 0 to maxDescriptor. Just check if ConnectSocket is set. Something like this:
// Main loop starts here
for(; ;)
{
memset(SentBuff, 0, sizeof(SentBuff));
printf("Write: ");
gets_s(SentBuff, sizeof(SentBuff));
// Copy the fdread into BackUpfdread and fdwrite to BackUpfdwrite.
BackUpfdread = fdread;
BackUpfdwrite = fdwrite;
SelectResults = select(maxDescriptor+1,&BackUpfdread,&BackUpfdwrite,NULL,&timer);
if(SelectResults == -1)
{
perror("Client-select() error!\n");
exit(1);
}
if (FD_ISSET(ConnectSocket, &BackUpfdread))
{
RecvBytes = recv(ConnectSocket, RecvBuff, sizeof(RecvBuff), 0);
if(RecvBytes > 0)
{
printf("%s\n",RecvBuff);
// Cleaning the Receive Buffer
memset(RecvBuff,0,sizeof(RecvBuff));
}
}
if (FD_ISSET(ConnectSocket, &BackUpfdwrite))
{
SentBytes = send(ConnectSocket, SentBuff,sizeof(SentBuff),0);
// Cleaning the Sent Buffer
memset(SentBuff,0,sizeof(SentBuff));
}
} // Main loop ends here
Also don't forget to check for errors from send and recv. Especially recv is important, as it's the call that will tell you the server has disconnected.
Edit: Another important thing to note, is that the socket may be writeable always, so add a check if there is something to write before you check if the socket is writeable.