Ubuntu Socket programming : Packets are repackaged between TX and RX - c

Question

I have 2 Ubuntu 14.04 PCs. One is used as a server and the other one is used as a client. The client setup a TCP connection to the server which sends some packets back. Here's the code on the server:
send(sd, pkt, pkt_len, MSG_NOSIGNAL);
The code on the client side is also very simple:
read(sd, buf, buf_size);
If the transmissions on the server is spaced out, I don't see any issue. However, if server is doing rapid transmissions, then thing looks ugly. Here's an example when the server is sending 8 packets back-to-back.
The server code shows the size of these 8 packets are: 752 (bytes), 713, 713, 713, 396, 398, 396, 396
tcpdump on the server captures 4 TX packets: 752 (bytes), 1398, 1398, 929
tcpdump on the client captures 3 RX packets: 752 (bytes), 2796, 929
The client code shows it receives only 2 packets with 3548 bytes and 929 bytes, respectively.
So you can see all the bytes sent by the server are received by the client. However, packets are combined at various points in the transmission path. I guess this is due to TSO, GSO, GRO, etc. However, shouldn't these optimizations re-assemble the packets back to the correct form when the packets are delivered to the receiving application?
How do I get around this issue?

Answer 1

TCP is carefully designed to not only permit but implement exactly what you're seeing. It is a byte-stream protocol. If you want messages you have to implement them yourself via a superimposed application protocol.

Answer 2

How do I get around this issue?
So you're using TCP (a byte-stream-oriented transport mechanism) but you'd like it to have message-oriented behavior. You can't change the way TCP works (it is, by design, allowed to transport bytes in whatever-sized groups it chooses to, as long as the bytes are all received and they are received in the same order). But you can add a layer on top of TCP to simulate packet-oriented behavior.
For example, say you wanted to simulate the transmission of a 1000-byte "packet". Your sending program could first send out a fixed-size (let's say, 4-byte) header that would tell the receiver how many bytes the "packet" will contain:
size_t myPacketSize = 1000; // or whatever the size of your packet is
uint32_t bePacketSize = htonl(myPacketSize); // convert native-endian to big-endian for cross-platform compatibility
if (send(sd, &bePacketSize, sizeof(bePacketSize), 0) != sizeof(bePacketSize))
{
perror("send(header)");
}
.... then right after that you'd send out the packet's payload data:
if (send(sd, packetDataPtr, myPacketSize, 0) != myPacketSize)
{
perror("send(body)");
}
The receiver would need to receive the header/size value, then allocate an array of that size and receive the payload data into it. Since this code has to handle the incoming data correctly no matter how many bytes are returned by each recv() call, it's a little more complex than the sending code:
void HandleReceivedPseudoPacket(const char * packetBytes, uint32_t packetSizeBytes)
{
// Your received-packet-handling code goes here
}
// Parses an incoming TCP stream of header+body data back into pseudo-packets for handling
void ReadPseudoPacketsFromTCPStreamForever(int sd)
{
uint32_t headerBuf; // we'll read each 4-byte header's bytes into here
uint32_t numValidHeaderBytes = 0; // how many bytes in (headerBuf) are currently valid
char * bodyBuf = NULL; // will be allocated as soon as we know how many bytes to allocate
uint32_t bodySize; // How many bytes (bodyBuf) points to
uint32_t numValidBodyBytes = 0; // how many bytes in (bodyBuf) are currently valid
while(1)
{
if (bodyBuf == NULL)
{
// We don't know the bodySize yet, so read in header bytes to find out
int32_t numBytesRead = recv(sd, ((char *)&headerBuf)+numValidHeaderBytes, sizeof(headerBuf)-numValidHeaderBytes, 0);
if (numBytesRead > 0)
{
numValidHeaderBytes += numBytesRead;
if (numValidHeaderBytes == sizeof(headerBuf))
{
// We've read the entire 4-byte header, so now we can allocate the body buffer
numValidBodyBytes = 0;
bodySize = ntohl(headerBuf); // convert from big-endian to the CPU's native-endian
bodyBuf = (char *) malloc(bodySize);
if (bodyBuf == NULL)
{
perror("malloc");
break;
}
}
}
else if (numBytesRead < 0)
{
perror("recv(header)");
break;
}
else
{
printf("TCP connection was closed while reading header bytes!\n");
break;
}
}
else
{
// If we got here, then we know the bodySize and now we need to read in the body bytes
int32_t numBytesRead = recv(sd, &bodyBuf[numValidBodyBytes], bodySize-numValidBodyBytes, 0);
if (numBytesRead > 0)
{
numValidBodyBytes += numBytesRead;
if (numValidBodyBytes == bodySize)
{
// At this point the pseudo-packet is fully received and ready to be handled
HandleReceivedPseudoPacket(bodyBuf, bodySize);
// Reset our state variables so we'll be ready to receive the next header
free(bodyBuf);
bodyBuf = NULL;
numValidHeaderBytes = 0;
}
}
else if (numBytesRead < 0)
{
perror("recv(body)");
break;
}
else
{
printf("TCP connection was closed while reading body bytes!\n");
break;
}
}
}
// Avoid memory leak if we exited the while loop in the middle of reading a psuedo-packet's body
if (bodyBuf) free(bodyBuf);
}