What could be the cause of very slow socket reads? - c

Question

I am using blocking TCP sockets for my client and server. Whenever I read, I first check whether data is available on the stream using select. I always read and write 40 bytes at a time. While most reads take few milliseconds or less, some just take more than half a second. That after I know that there is data available on the socket.
I am also using TCP_NODELAY
What could be causing it ?
EDIT 2
I analyzed the timestamp for each packet sent and received and saw that this delay happens only when client tries to read the object before the next object is written by the server. For instance, the server wrote object number x and after that the client tried to read object x, before the server was able to begin writing object number x+1. This makes me suspect that some kind of coalescing is taking place on the server side.
EDIT
The server is listening on 3 different ports. The client connects one by one to each of these ports.
There are three connections : One that sends some data frequently from the server to the client. A second one that only sends data from the client to the server. And a third one that is used very rarely to send single byte of data. I am facing the problem with the first connection. I am checking using select() that data is available on that connection and then when I timestamp the 40 byte read, I find that about half a second was taken for that read.
Any pointers as to how to profile this would be very helpful
using gcc on linux.
rdrr_server_start(void)
{
int rr_sd;
int input_sd;
int ack_sd;
int fp_sd;
startTcpServer(&rr_sd, remote_rr_port);
startTcpServer(&input_sd, remote_input_port);
startTcpServer(&ack_sd, remote_ack_port);
startTcpServer(&fp_sd, remote_fp_port);
connFD_rr = getTcpConnection(rr_sd);
connFD_input = getTcpConnection(input_sd);
connFD_ack= getTcpConnection(ack_sd);
connFD_fp=getTcpConnection(fp_sd);
}
static int getTcpConnection(int sd)
{
socklen_t l en;
struct sockaddr_in clientAddress;
len = sizeof(clientAddress);
int connFD = accept(sd, (struct sockaddr*) &clientAddress, &len);
nodelay(connFD);
fflush(stdout);
return connFD;
}
static void
startTcpServer(int *sd, const int port)
{
*sd= socket(AF_INET, SOCK_STREAM, 0);
ASSERT(*sd>0);
// Set socket option so that port can be reused
int enable = 1;
setsockopt(*sd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(int));
struct sockaddr_in a;
memset(&a,0,sizeof(a));
a.sin_family = AF_INET;
a.sin_port = port;
a.sin_addr.s_addr = INADDR_ANY;
int bindResult = bind(*sd, (struct sockaddr *) &a, sizeof(a));
ASSERT(bindResult ==0);
listen(*sd,2);
}
static void nodelay(int fd) {
int flag=1;
ASSERT(setsockopt(fd, SOL_TCP, TCP_NODELAY, &flag, sizeof flag)==0);
}
startTcpClient() {
connFD_rr = socket(AF_INET, SOCK_STREAM, 0);
connFD_input = socket(AF_INET, SOCK_STREAM, 0);
connFD_ack = socket(AF_INET, SOCK_STREAM, 0);
connFD_fp= socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in a;
memset(&a,0,sizeof(a));
a.sin_family = AF_INET;
a.sin_port = remote_rr_port;
a.sin_addr.s_addr = inet_addr(remote_server_ip);
int CONNECT_TO_SERVER= connect(connFD_rr, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
a.sin_port = remote_input_port;
CONNECT_TO_SERVER= connect(connFD_input, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
a.sin_port = remote_ack_port;
CONNECT_TO_SERVER= connect(connFD_ack, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
a.sin_port = remote_fp_port;
CONNECT_TO_SERVER= connect(connFD_fp, &a, sizeof(a));
ASSERT(CONNECT_TO_SERVER==0) ;
nodelay(connFD_rr);
nodelay(connFD_input);
nodelay(connFD_ack);
nodelay(connFD_fp);
}

Answer 1

I would be suspicious of the this line of code:
ASSERT(setsockopt(fd, SOL_TCP, TCP_NODELAY, &flag, sizeof flag)==0);
If you are running a release build, then ASSERT is mostly likely defined to nothing, so the call would not actually be made. The setsockopt call should not be in the ASSERT statement. Instead, the return value (in a variable) should be verified in the assert statement. Asserts with side effects are generally a bad thing. So even if this is not the problem, it should probably be changed.

Answer 2

One client and multiple connections?
some of socket functions might be blocking your execution (i.e. waiting for result of functions). I would suggest opening a new thread (on server side) for each connection so they won't interfere with each other...
but I'm shooting in the dark; you'll need to send some additional info...

Answer 3

Your statement is still confusing i.e. "multiple tcp connections with only one client". Obviously you have a single server listening on one port. Now if you have multiple connections this means there is more than one client connecting to the server each connected on a different tcp client port. Now server runs select and responds to whichever client has data (meaning client sent some data on his socket). Now if two clients send data simultaneously, server can only process them sequentially. So second client won't get processed until server is done processing with first.
Select only allows server to monitor more than one descriptors (sockets) and process which ever has data available. It is not like that it does processing in parallel. You need multiple threads or processes for that.

Answer 4

Maybe it is something related to the timeout argument.
What do you set for timeout argument of select call?
Try to change the timeout argument to a bigger one and observe the latency. Sometimes too small timeout and very often system calls can actually kill throughput . Maybe you can achieve better results if you assume a little bigger latency, that is realizable.
I suspect timeout or some code bug.

Answer 5

You may try using TCP_CORK (CORK'ed mode) with kernel extensions GRO, GSO and TSO disabled by ethtool:
sending inside TCP_CORK flagged session will ensure that the data will not be sent in partial segment
disabling generic-segmentation-offload, generic-receive-offload and tcp-segmentation-offload will ensure that kernel will not introduce artificial delays to collect additional tcp segments before moving data to/from userspace