I am writing a simple web server and client using UDP and so far: the programs can connect to each other, the client can send a request, the server can read the request, the server can recognize the client's IP address and client's port, and the server can send a message back to the client
My problem is that my client code gets stuck waiting in the rcvfrom function, even after the server has sent a response.
Here is the function that is supposed to pick up the server message and return the number of bytes read by the socket:
ssize_t receive_from_server(rdp_socket *rsocket, char *buffer, size_t buf_len){
socklen_t sendsize = sizeof(rsocket->server_addr);
bzero(&(rsocket->server_addr), sendsize);
//STUCK HERE:
return recvfrom(rsocket->sockfd, buffer, buf_len, 0,
(struct sockaddr*)&(rsocket->server_addr), &sendsize);
}
I set the sockopts for both SO_SNDTIMEO and SO_RCVTIMEO to timeout after a few seconds.
Question:
In the short term future I will be adding acknowledgements (ACKs) for reliable data transfer. I imagine that missing ACKs could be the issue but I'm just wondering if, to the trained eye, it looks like a different problem.
Are ACKs necessary for a timeout to work?
How can I synchronize my client and server so that they can actually communicate with each other?
Since UDP does not provide reliability, you will need to implement retransmission of missing data. Since it looks like this is a client request server response model, the easiest retransmission implementation for you may be to resend the request when you time out waiting for the response, and wait for the response again. You may want to implement a retry counter and give up after a certain number of retries.
If the SO_RCVTIMEO and SO_SNDTIMEO socket options do not seem to be taking effect, it may be those options are not implemented for that type of socket. Check the return value of the setsockopt()call to make sure they succeeded.
As a workaround, you can change your receive_from_server() function to use poll() or select() to wait for a readable event for some amount of time, instead of blocking in recvfrom().
ssize_t receive_from_server(rdp_socket *rsocket, char *buffer, size_t buf_len){
struct pollfd pfd = { rsocket->sockfd, POLLIN };
int pollresult = poll(&pfd, 1, RECV_TIMEOUT_SECONDS * 1000);
if (pollresult > 0) {
socklen_t sendsize = sizeof(rsocket->server_addr);
bzero(&(rsocket->server_addr), sendsize);
return recvfrom(rsocket->sockfd, buffer, buf_len, MSG_DONTWAIT,
(struct sockaddr*)&(rsocket->server_addr), &sendsize);
}
if (pollresult == 0) {
errno = ETIME;
}
return -1;
}
Related
Before I Start
Please don't mark this question as a duplicate. I have already seen the numerous posts on SO about handling multiple clients with socket programming. Most people recommend just multi-threading, but I am trying to avoid that path because I have read it has a few problems:
Bad Scalability
Large Overhead/Inefficient/Memory Hungry
Difficult to Debug
Any posts that I have read that specifically talk about using a single thread either have bad/no answers or have unclear explanations, like people saying "Just use select()!"
The Problem
I am writing code for a server to handle multiple (~1000) clients, and I'm having trouble figuring out how to create an efficient solution. Right now I already have the code for my server that is able to handle 1 client at a time. Both are written in C; the server is on Windows using WinSock and the client is on Linux.
The server and client send several communications back and forth, using send() and blocking recv() calls. Writing this code was pretty simple, and I won't post it here because it is pretty long and I doubt anyone will actually read through all of it. Also the exact implementation is not important, I just want to talk about high level pseudocode. The real difficulty is changing the server to handle multiple clients.
What's Already Out There
I have found a nice PDF tutorial about how to create a WinSock server that handles multiple clients and it can be found here: WinSock Multiple Client Support. It's in C++ but it's easily transferable to C.
From what I understand the server operates something like this:
while (running) {
Sleep(1000);
/* Accept all incoming clients and add to clientArray. */
for (client in clientArray) {
/* Interact with client */
if (recv(...) == "disconnect") {
/* Disconnect from client */
}
}
}
/* Close all connections. */
The problem that I see with using this approach is that you essentially only handle one client at a time (which is obvious because you aren't multithreading), but what if the interaction with each client only needs to happen once? Meaning, what if I just want to send some data back and forth and close the connection? This operation could take anywhere from 5 seconds to 5 minutes depending on the speed of the clients connection, so other clients would be blocking on a connect() call to the server while the server handles a client for 5 minutes. It doesn't seem very efficient, but maybe the best way would be to implement a waiting queue, where clients are connected and told to wait for a while? I'm not sure, but it makes me curious about how large servers send out update downloads concurrently to thousands of clients, and if I should operate the same way.
Also, is there a reason for adding a Sleep(1000) call in the main server loop, if the send() and recv() between the server and client take a while (~1 minute)?
What I'm Asking For
What I want is a solution to handling multiple clients on a single threaded server that is efficient enough for ~1000 clients. If you tell me that the solution in the PDF is fine, that's good enough for me (maybe I'm just too preoccupied with efficiency.)
Please give answers that include a verbal explanation of the implementation, server/client pseudocode, or even a small sample code for the server, if you're feeling sadistic.)
Thanks in advance.
I have written single thread socket pool handling. Im using non-blocking sockets and select call to handle all send, receive and errors.
My class keep all sockets in array, and build 3 fd set's for select call. When something happens it check read or write or error list and handle those events.
For example, non-blocking client socket during connection can trigger write or error event. If error event happens then connection failed. If write happens, connection is established.
All sockets is in read fd set. If you create server socket (with bind and listen) new connection will trigger read event. Then check if socket is server socket then call accept for new connection. If read operation is triggered by regular socket then there is some bytes to read.. just call recv with buffer arge enough to suck all data from that socket.
SOCKET maxset=0;
fd_set rset, wset, eset;
FD_ZERO(&rset);
FD_ZERO(&wset);
FD_ZERO(&eset);
for (size_t i=0; i<readsockets.size(); i++)
{
SOCKET s = readsockets[i]->s->GetSocket();
FD_SET(s, &rset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<writesockets.size(); i++)
{
SOCKET s = writesockets[i]->s->GetSocket();
FD_SET(s, &wset);
if (s > maxset) maxset = s;
}
for (size_t i=0; i<errorsockets.size(); i++)
{
SOCKET s = errorsockets[i]->s->GetSocket();
FD_SET(s, &eset);
if (s > maxset) maxset = s;
}
int ret = 0;
if (bBlocking)
ret = select(maxset + 1, &rset, &wset, &eset, NULL/*&tv*/);
else
{
timeval tv= {0, timeout*1000};
ret = select(maxset + 1, &rset, &wset, &eset, &tv);
}
if (ret < 0)
{
//int err = errno;
NetworkCheckError();
return false;
}
if (ret > 0)
{
// loop through eset and check each with FD_ISSET. if you find some socket it means connect failed
// loop through wset and check each with FD_ISSET. If you find some socket check is there any pending connectin on that socket. If there is pending connection then that socket just got connected. Otherwise select just reported that some data has been sent and you can send more.
// finally, loop through rset and check each with FD_ISSET. If you find some socket then check is this socket your server socket (bind and listen). If its server socket then this is signal new client want to connect.. just call accept and new connection is established. If this is not server socket, then just do recv on that socket to collect new data.
}
There is few more things to handle... All sockets must be in non-blocking mode. Each send or recv calls will return -1 (error) but error code is EWOULDBLOCK. Thats normal and ignore error. If recv returns 0 then this connection is dropped. If send return 0 bytes sent then internal buffer is full.
You need to write additional code to serialize and parse data. For example, after recv, message may not be complete (depending on message size) so it may take more than one recv calls to receive complete message. Sometimes if messages is short recv call can deliver several messages in buffer. So, you need to write good parser or design good protocol, easy to parse.
First, regarding single-thread approach: I'd say it's bad idea because your server processing power is limited by performance of single processor core. But other than that it'll work to some extent.
Now about multiclient problem. I'd suggest using WSASend and WSARecv with their compilation routines. It also can be scaled to multiple threads if necessary.
Server core will look something like this:
struct SocketData {
::SOCKET socket;
::WSAOVERLAPPED overlapped;
::WSABUF bufferRef;
char buf [1024];
// other client-related data
SocketData (void) {
overlapped->hEvent = (HANDLE) this;
bufferRef->buf = buf;
bufferRef->len = sizeof (buf);
// ...
}
};
void OnRecv (
DWORD dwError,
DWORD cbTransferred,
LPWSAOVERLAPPED lpOverlapped,
DWORD dwFlags) {
auto data = (SocketData*) lpOverlapped->hEvent;
if (dwError || !cbTransferred) {
::closesocket (data->socket);
delete data;
return;
}
// process received data
// ...
}
// same for OnSend
void main (void) {
// init and start async listener
::SOCKET serverSocket = ::socket (...);
HANDLE hAccept = ::CreateEvent (nullptr, 0, 0, nullptr);
::WSAEventSelect (serverSocket, FD_ACCEPT, hAccept);
::bind (serverSocket, ...);
::listen (serverSocket, ...);
// main loop
for (;;) {
int r = ::WaitForSingleObjectEx (hAccept, INFINITE, 1);
if (r == WAIT_IO_COMPLETION)
continue;
// accept processing
auto data = new SocketData ();
data->socket = ::accept (serverSocket, ...);
// detach new socket from hAccept event
::WSAEventSelect (data->socket, 0, nullptr);
// recv first data from client
::WSARecv (
data->socket,
&data->bufferRef,
1,
nullptr,
0,
&data->overlapped,
&OnRecv);
}
}
Key points:
wait in main loop (WaitForSingleObjectEx, WaitForMultipleObjectsEx etc.) must be alertable;
most data processing done in OnSend/OnRecv;
all processing must be done without blocking APIs in OnSend/OnRecv;
for event-based processing events must be waited in main loop.
OnRecv will be called for each processed incoming packet. OnSend will be called for each processed outgoing packet. Keep in mind: how many data you asked to send/recv is not the same as what actually processed in packet.
I wrote simple TCP/IP multi-thread ANSI C server (client is C sharp), everything works fine except when the server doesnt receive proper signal from client it wont end the thread and close its socket (for example when client crash). Eventually it could become problem if those threads accumulate.
I got threads stored in Linked List - iterating through them isnt a problem. However they are all blocked by recv() by default and since dead client wont send anything they become stuck in memory.
What is the proper way of maintaining list of online clients? (or how to detect threads with broken connection).
struct tListItem {
pthread_t thisThread;
char* name;
int c_sockfd;
int run;
tListItem* next;
tListItem* prev;};
struct tList{
tListItem* head;
int count;};
code of thread:
while(param->run)
{
bzero(&buf, sizeof(buf));
if ((readLen = recv(param->c_sockfd, buf, BUFFSIZE, 0)) == -1)
{
perror("Read error");
param->run = 0;
}
else if (readLen > 0) {
printf("%s: %s \n", param->name, buf);
parseIncoming(param->c_sockfd, param, buf);}}
and here is my attempt to detect broken connection, but this causes the server to end with no message:
void* maintenance() {
tListItem *item;
char buf[4] = "PNG";
while(1)
{
usleep(2000000);
item= threadList->head;
while(item != 0)
{
if ((send(item->c_sockfd, buf, 3, NULL)) == -1)
{
perror("Write error");
item->run = 0;
}
item = item->next;
}
}
}
There's a few common ways this is dealt with:
Implement a heartbeat/ping-pong in your protocol on top of TCP. That is, periodically the client and/or server
sends a heartbeat message to the other end. If the server has not received any data or heartbeat messages within a period of time, e.g. two times the heartbeat period, or if sending the heartbeat message from the server fails, then consider the connection to be dead and close it.
Implement an overall data timeout. Each time the server receives data, you read time current time. Periodically you check the connection for when you last received data, and time out/close connections that haven't received data in a while.
Enable TCP keepalive. This is basically a last resort if you cannot do either 1. or 2.. It'll help you detect dead peers, as the TCP keepalives will break the connection if the peer cannot be reached. (Though it will not help you detect idle clients). Note that the default for keepalives is in the order of hours.
In all cases you should always to be read()/recv() or otherwise monitoring the socket for read events so you can learn as quick as possible if the connection actively breaks.
It's also quite hard to implement this if you're doing blocking read()/recv() calls, you would normally need to set a timeout on the read() so you can wake up periodically and send a heartbeat message or check if the client has been idle for too long - this is best done by using select()/poll() or the like so you can get a timeout instead of doing a block read() that might never return.
accept() is defined to always create another file descriptor to accept new connections from the client, but if it is known beforehand that we are only going to be accepting one client and one connection, why bother with creating a new file descriptor? Are there any descriptions of why this is the case in any defined standards?
When designing APIs I think there is value in being generic. Why have 2 APIs, one for accepting potentially multiple connections and one for using fewer file descriptors? The latter case doesn't seem high priority enough to justify an entirely new syscall when the API we have today will do and you can use it to implement the behavior you want just fine.
On the other hand, Windows has AcceptEx which lets you re-use previous socket handles that previously represented otherwise unrelated, previously connected sockets. I believe this is to avoid the performance hit of entering the kernel again to close sockets after they are disconnected. Not exactly what you are describing but vaguely similar. (Though meant to scale up rather than scale down.)
Update: One month later I think it's a little strange that you created a bounty on this. I think the answer is clear - the current interfaces can do what you ask for just fine and there's really no motivation to add, let alone standardize, a new interface for your fringe case. With the current interfaces you can close the original socket after accept succeeds and it won't harm anyone.
The TCP protocol described in RFC 793 describes the terms socket and connection. A socket is an IP address and port number pair. A connection is a pair of sockets. In this sense, the same socket can be used for multiple connections. It is in this sense that the socket being passed to accept() is being used. Since a socket can be used for multiple connections, and the socket passed to accept() represents that socket, the API creates a new socket to represent the connection.
If you just want an easy way to make sure the one socket that accept() creates for you is the same socket you used to do the accept() call on, then use a wrapper FTW:
int accept_one (int accept_sock, struct sockaddr *addr, socklen_t *addrlen) {
int sock = accept(accept_sock, addr, addrlen);
if (sock >= 0) {
dup2(sock, accept_sock);
close(sock);
sock = accept_sock;
}
return sock;
}
If you are wanting a way for a client and server to connect to each other, without creating any more than just one socket on each side, such an API does exist. The API is connect(), and it succeeds when you achieve a simultaneous open.
static struct sockaddr_in server_addr;
static struct sockaddr_in client_addr;
void init_addr (struct sockaddr_in *addr, short port) {
struct sockaddr_in tmp = {
.sin_family = AF_INET, .sin_port = htons(port),
.sin_addr = { htonl(INADDR_LOOPBACK) } };
*addr = tmp;
}
void connect_accept (int sock,
struct sockaddr_in *from, struct sockaddr_in *to) {
const int one = 1;
int r;
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
bind(sock, (struct sockaddr *)from, sizeof(*from));
do r = connect(sock, (struct sockaddr *)to, sizeof(*to)); while (r != 0);
}
void do_peer (char *who, const char *msg, size_t len,
struct sockaddr_in *from, struct sockaddr_in *to) {
int sock = socket(PF_INET, SOCK_STREAM, 0);
connect_accept(sock, from, to);
write(sock, msg, len-1);
shutdown(sock, SHUT_WR);
char buf[256];
int r = read(sock, buf, sizeof(buf));
close(sock);
if (r > 0) printf("%s received: %.*s%s", who, r, buf,
buf[r-1] == '\n' ? "" : "...\n");
else if (r < 0) perror("read");
}
void do_client () {
const char msg[] = "client says hi\n";
do_peer("client", msg, sizeof(msg), &client_addr, &server_addr);
}
void do_server () {
const char msg[] = "server says hi\n";
do_peer("server", msg, sizeof(msg), &server_addr, &client_addr);
}
int main () {
init_addr(&server_addr, 4321);
init_addr(&client_addr, 4322);
pid_t p = fork();
switch (p) {
case 0: do_client(); break;
case -1: perror("fork"); exit(EXIT_FAILURE);
default: do_server(); waitpid(p, 0, 0);
}
return 0;
}
If instead you are worried about performance issues, I believe those worries are misguided. Using the TCP protocol, you already have to wait at least one full round trip on the network between the client and the server, so the extra overhead of dealing with another socket is negligible. A possible case where you might care about that overhead is if the client and server are on the same machine, but even then, it is only an issue if the connections are very short lived. If the connections are so short lived, then it would probably be better to redesign your solution to either use a cheaper communication medium (e.g., shared memory), or apply framing on your data and use a persistent connection.
Because it isn't required. If you only have one client, you only do the operation once; you have plenty of file descriptors to spare; and compared to network overheads the 'overhead' is vanishingly small. The case that you would want to 'optimize' as an API designer is when you have thousands of clients.
The only thing that changes between the socket returned by listen and the socket descriptor returned by accept, is that the new socket is in the ESTABILISHED state instead of the LISTEN state.So you can re-use the socket created after invoking the listen functions to accept other connections.
As accept() is designed to accept new client .
it required three things, general socket descriptor which must bind to a specific port number for serving at that port number and a structure to store the client information and another int value to store size of client .
it return a new_socket_descriptor for serving the particular client which is accepted by server.
the first parameter is a socket descriptor used to accept client.And for concurrence server, it is always use for accepting client connection .So it should not modify by any accept() call.
so new socket descriptor returned by accept() to serve new connected client.
the server socket descriptor(1st parameter) bind to server property.server property always designed to a fixed type that is its port number ,type of connection,protocol family all are fixed.So same file descriptor is used again and again.
Another point is that these property are used to filter client connection which are made for that particular server.
For clients,information for each client different minimum ip address used by every client unique and these property are bind to new file descriptor so always a new file descriptor returned by accept() function success.
NOTE:-
that is you require one file descriptor must for client accepting and depending upon maximum number of client you want to accept/serve use that much file descriptor for serving clients.
The answer is that your specific example of exactly one connection is handled in the current API and was designed into the API's use cases from the start. The explanation for how the single socket case is handled lies in the way socket programs were designed to work when the BSD socket interface was first invented.
The socket API was designed to always be able to accept connections. The fundamental principle is that when a connection arrives, the program should have the final decision as to whether the connection is accepted or not. However, the application must also never miss a connection while making this decision. Thus, the API was designed only to be parallel and accept() was specified to return a different socket from listen(), so that listen() could continue listening for further connection requests while the application made its decision about the connection request just received. This was a fundamental design decision and is not documented anywhere; it was just assumed that socket programs would have to work that way in order to be useful.
In the old days before threads were invented, the parallelism required to implement socket servers on Unix-like systems relied on fork(). A new connection was accepted, the program would split itself into two identical copies using fork(), and then one copy would handle the new connection while the original copy continued listening for incoming connection attempts. In the fork() model, even though accept() returns a new file handle, the use case of handling exactly one connection was supported and was achieved by just letting the "listening" copy of the program exit while the second "accept" copy handles the single connection.
The following pseudo code shows this:
fd = socket();
listen(fd, 1); /* allow 1 unanswered connection in the backlog */
switch (fork())
{
case 0: break; /* child process; handle connection */
case -1: exit (1); /* error. exit anyway. */
default: exit (0); /* parent process; exit as only one connection needed */
}
/* if we get here our single connection can be accepted and handled.
*/
accept_fd = accept(fd);
This programming paradigm meant that whether servers accepted a single connection, or stayed in loops handling multiple connections, the code was virtually identical in both cases. Nowadays we have threads instead of fork(). However, as the paradigm still remains to this today, it has never been necessary to change or upgrade the socket API.
I am using blocking TCP sockets for my client and server. Whenever I read, I first check whether data is available on the stream using select. I always read and write 40 bytes at a time. While most reads take few milliseconds or less, some just take more than half a second. That after I know that there is data available on the socket.
I am also using TCP_NODELAY
What could be causing it ?
EDIT 2
I analyzed the timestamp for each packet sent and received and saw that this delay happens only when client tries to read the object before the next object is written by the server. For instance, the server wrote object number x and after that the client tried to read object x, before the server was able to begin writing object number x+1. This makes me suspect that some kind of coalescing is taking place on the server side.
EDIT
The server is listening on 3 different ports. The client connects one by one to each of these ports.
There are three connections : One that sends some data frequently from the server to the client. A second one that only sends data from the client to the server. And a third one that is used very rarely to send single byte of data. I am facing the problem with the first connection. I am checking using select() that data is available on that connection and then when I timestamp the 40 byte read, I find that about half a second was taken for that read.
Any pointers as to how to profile this would be very helpful
using gcc on linux.
rdrr_server_start(void)
{
int rr_sd;
int input_sd;
int ack_sd;
int fp_sd;
startTcpServer(&rr_sd, remote_rr_port);
startTcpServer(&input_sd, remote_input_port);
startTcpServer(&ack_sd, remote_ack_port);
startTcpServer(&fp_sd, remote_fp_port);
connFD_rr = getTcpConnection(rr_sd);
connFD_input = getTcpConnection(input_sd);
connFD_ack= getTcpConnection(ack_sd);
connFD_fp=getTcpConnection(fp_sd);
}
static int getTcpConnection(int sd)
{
socklen_t l en;
struct sockaddr_in clientAddress;
len = sizeof(clientAddress);
int connFD = accept(sd, (struct sockaddr*) &clientAddress, &len);
nodelay(connFD);
fflush(stdout);
return connFD;
}
static void
startTcpServer(int *sd, const int port)
{
*sd= socket(AF_INET, SOCK_STREAM, 0);
ASSERT(*sd>0);
// Set socket option so that port can be reused
int enable = 1;
setsockopt(*sd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(int));
struct sockaddr_in a;
memset(&a,0,sizeof(a));
a.sin_family = AF_INET;
a.sin_port = port;
a.sin_addr.s_addr = INADDR_ANY;
int bindResult = bind(*sd, (struct sockaddr *) &a, sizeof(a));
ASSERT(bindResult ==0);
listen(*sd,2);
}
static void nodelay(int fd) {
int flag=1;
ASSERT(setsockopt(fd, SOL_TCP, TCP_NODELAY, &flag, sizeof flag)==0);
}
startTcpClient() {
connFD_rr = socket(AF_INET, SOCK_STREAM, 0);
connFD_input = socket(AF_INET, SOCK_STREAM, 0);
connFD_ack = socket(AF_INET, SOCK_STREAM, 0);
connFD_fp= socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in a;
memset(&a,0,sizeof(a));
a.sin_family = AF_INET;
a.sin_port = remote_rr_port;
a.sin_addr.s_addr = inet_addr(remote_server_ip);
int CONNECT_TO_SERVER= connect(connFD_rr, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
a.sin_port = remote_input_port;
CONNECT_TO_SERVER= connect(connFD_input, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
a.sin_port = remote_ack_port;
CONNECT_TO_SERVER= connect(connFD_ack, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
a.sin_port = remote_fp_port;
CONNECT_TO_SERVER= connect(connFD_fp, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
nodelay(connFD_rr);
nodelay(connFD_input);
nodelay(connFD_ack);
nodelay(connFD_fp);
}
I would be suspicious of the this line of code:
ASSERT(setsockopt(fd, SOL_TCP, TCP_NODELAY, &flag, sizeof flag)==0);
If you are running a release build, then ASSERT is mostly likely defined to nothing, so the call would not actually be made. The setsockopt call should not be in the ASSERT statement. Instead, the return value (in a variable) should be verified in the assert statement. Asserts with side effects are generally a bad thing. So even if this is not the problem, it should probably be changed.
One client and multiple connections?
some of socket functions might be blocking your execution (i.e. waiting for result of functions). I would suggest opening a new thread (on server side) for each connection so they won't interfere with each other...
but I'm shooting in the dark; you'll need to send some additional info...
Your statement is still confusing i.e. "multiple tcp connections with only one client". Obviously you have a single server listening on one port. Now if you have multiple connections this means there is more than one client connecting to the server each connected on a different tcp client port. Now server runs select and responds to whichever client has data (meaning client sent some data on his socket). Now if two clients send data simultaneously, server can only process them sequentially. So second client won't get processed until server is done processing with first.
Select only allows server to monitor more than one descriptors (sockets) and process which ever has data available. It is not like that it does processing in parallel. You need multiple threads or processes for that.
Maybe it is something related to the timeout argument.
What do you set for timeout argument of select call?
Try to change the timeout argument to a bigger one and observe the latency. Sometimes too small timeout and very often system calls can actually kill throughput . Maybe you can achieve better results if you assume a little bigger latency, that is realizable.
I suspect timeout or some code bug.
You may try using TCP_CORK (CORK'ed mode) with kernel extensions GRO, GSO and TSO disabled by ethtool:
sending inside TCP_CORK flagged session will ensure that the data will not be sent in partial segment
disabling generic-segmentation-offload, generic-receive-offload and tcp-segmentation-offload will ensure that kernel will not introduce artificial delays to collect additional tcp segments before moving data to/from userspace
I am currently using this function in a C client program. Everything seems to work fine but when the server to which this client is connected is shut down, write_all() returns 4 (that's len) instead of the expected -1.
int write_all(int sock, const void *buf, size_t len)
{
int buf_size = len;
while(len > 0)
{
int result = write(sock, buf, len);
if(result < 0)
{
if(errno == EINTR)
continue;
return result;
}
buf += result;
len -= result;
}
return buf_size;
}
Is there anything I am missing in this function? Is there any other function I can call beforehand to make sure the server is still up?
Thanks
You say "shut down", do you mean that you switch the power off, without gracefull TCP closing?
In that case write call returns with success. Data is in TCP sending buffer, and TCP stack does not yet know that peer is down. Program will get EPIPE or other error during later calls.
TCP stack will try retransmission a while, before making decision of connection failure.
To me this looks like you won't get around implementing some sort of hand shake.
As if it's not enough for your sender to know the data it send had been fully received (what I assume is the case), but it also needs to know if any kind on processing had been done on it by the receiver, you expect more from the socket's mechanics than they can provide ...
The sockets are just the transmitter.
Note: I'm assuming TCP here.
From the return value, I gather that the client managed to write 4 bytes to the send buffer before learning that the server closed its end or otherwise disappeared. If it disappeared without proper closing, the only way to know would be a timed-out send. The next write, shutdown or close after that will get the error.
If you want to get prompt notification of disappearing endpoints without having to constantly send data, you can activate the socket keepalive option. In Linux, that would be a setsockopt(..., SOL_SOCKET, SO_KEEPALIVE, ...), and TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT at the SOL_TCP level.