Passing multiple buffers with iovec in C Linux sockets - c

I'm writing a linux C client-server programs that communicates with each other with unix domain sockets and passes couple of buffers each time.
I'm using ioverctors but for some reasons the server program only receives the first io vector.
Any idea ?
I attached the relevant code snippets.
Client code:
struct iovec iov[2];
struct msghdr mh;
int rc;
char str1[] = "abc";
char str2[] = "1234";
iov[0].iov_base = (caddr_t)str1;
iov[0].iov_len = sizeof(str1);
iov[1].iov_base = (caddr_t)str2;
iov[1].iov_len = sizeof(str2);
memset(&mh, 0, sizeof(mh));
mh.msg_iov = iov;
mh.msg_iovlen = 2;
n = sendmsg(sockfd, &mh, 0); /* no flags used*/
if (n > 0) {
printf("Sendmsg successfully executed\n");
}
}
Server code:
{
struct sockaddr_un *client_sockaddr = (sockaddr_un *)opq;
struct msghdr msg;
struct iovec io[2];
char buf[16];
char buf2[16];
io[0].iov_base = buf;
io[0].iov_len = sizeof(buf);
io[1].iov_base = buf2;
io[1].iov_len = sizeof(buf2);
msg.msg_iov = io;
msg.msg_iovlen = 2;
int len = recvmsg(sock, &msg, 0);
if (len > 0) {
printf("recv: %s %d %s %d\n", msg.msg_iov[0].iov_base, msg.msg_iov[0].iov_len, msg.msg_iov[1].iov_base, msg.msg_iov[1].iov_len);
}
return 0;
}
The output i'm getting from the server:
recv: abc 16 16

sendmsg(), writev(), pwritev(), and pwritev2() do not operate on multiple buffers, but one discontiguous buffer. They operate exactly as if you'd allocate a large enough temporary buffer, gather the data there, and then do the corresponding syscall on the single temporary buffer.
Their counterparts recvmsg(), readv(), preadv(), and preadv2() similarly do not operate on multiple buffers, only on one discontiguous buffer. They operate exactly as if you'd allocate a large enough temporary buffer, receive data into that buffer, then scatter the data from that buffer to the discontiguous buffer parts.
Unix domain datagram (SOCK_DGRAM) and seqpacket (SOCK_SEQPACKET) sockets preserve message boundaries, but stream sockets (SOCK_STREAM) do not. That is, using a datagram or seqpacket socket you receive each message as it was sent. With a stream socket, message boundaries are lost: two consecutively sent messages can be received as a single message, and you can (at least in theory) receive a partial message now and the rest later.
You can use the Linux-specific sendmmsg() function to send several messages in one call (using the same socket). If you use an Unix domain datagram or seqpacket socket, these will then retain their message boundaries.
Each message is described using a struct mmsghdr. It contains struct msghdr msg_hdr; and unsigned int msg_len;. msg_hdr is the same as you use when sending a single message using e.g. sendmsg(); you can use more than one iovec for each message, but the recipient will receive them concatenated into a single buffer (but can scatter that buffer using e.g. recvmsg()). msg_len will be filled in by the sendmmsg() call: the number of bytes sent for that particular message, similar to the return value of e.g. sendmsg() call when no errors occur.
The return value from the sendmmsg() call is the number of messages sent successfully (which may be fewer than requested!), or -1 if an error occurs (with errno indicating the error as usual). Thus, you'll want to write a helper function or a loop around sendmmsg() to make sure you send all the messages. For portability, I recommend a helper function, because you can then provide another based on a loop around sendmsg() for use when sendmmsg() is not available.
The only real benefit of sendmmsg() is that you need fewer syscalls to send a large number of messages: it boosts efficiency in certain situations, that's all.

Related

Best practice of sending data of different sizes over the network

I want to send data in varying sizes over UDP. The size of data to be sent is not fixed. I have the following scenario:
unsigned char buffer[BUFFERSIZE];
int bytes = fill_buffer(buffer, sizeof(buffer)): // Returns number of filled bytes.
sendto(socket, buffer, bytes, 0, (struct sockaddr *)&server, sizeof(server))
In the example above, receiving side does not know how many bytes to receive. I also thought of first sending the number of bytes to receive and then sending the data. But in that case, I don't know what would happen if the packets arrive out-of-order.
Sender side would be
sendto(socket, &bytes, sizeof(bytes), 0, (struct sockaddr *)&server, sizeof(server))
sendto(socket, buffer, bytes, 0, (struct sockaddr *)&server, sizeof(server))
Receving side would be
recvfrom(socket, &bytes, sizeof(bytes), 0, NULL, NULL)
recvfrom(socket, buffer, bytes, 0, NULL, NULL)
But could it be that sent data comes out-of-order?
I think you can send both in a single datagram if you add a message header.
The sender only sends the amount of payload data it has.
The receiver always requests the maximum payload size but examines the header and the return from recvfrom to determine the actual length.
Here's some rough code that illustrates what I'm thinking of:
struct header {
u32 magic_number;
u32 seq_no;
u32 msg_type;
u32 payload_length;
} __attribute__((__packed__));
#define MAXPAYLOAD 1024
struct message {
struct header info;
unsigned char payload[MAXPAYLOAD];
} __attribute__((__packed__));
void
sendone(int sockfd,const void *buf,size_t buflen)
{
struct message msg;
static u32 seqno = 0;
memcpy(&msg.payload[0],buf,buflen);
msg.info.magic_number = 0xDEADADDE;
msg.info.seq_no = seqno++;
msg.info.payload_length = buflen;
sendto(sockfd,&msg,sizeof(struct header) + buflen,...);
}
ssize_t
getone(int sockfd,void *buf,size_t buflen)
{
struct message msg;
ssize_t rawlen;
ssize_t paylen;
static u32 seqno = 0;
rawlen = recvfrom(sockfd,&msg,sizeof(struct header) + MAXPAYLOAD,...);
paylen = msg.info.payload_length;
if (rawlen != (sizeof(struct header) + paylen))
// error ...
memcpy(buf,&msg.payload[0],paylen);
return paylen;
}
The receiver can check the magic number and sequence number to look for corruption or missing/dropped packets, etc.
In fact, you can probably get more efficiency by using sendmsg and recvmsg since they allow you to send a single message using a scatter/gather list. (i.e.) The data would not have to be copied in/out using memcpy from the message struct [you'd only need struct header], so closer to zero copy buffering.
Another option may be to use the MSG_PEEK flag with the recvfrom/recvmsg. I've never used this myself, but it would be something like:
Do recvmsg with length of sizeof(struct header) with MSG_PEEK flag
Do second recvmsg with length of sizeof(struct header) + msg.info.payload_length
This is just a nicety of not having to always provide a maximum sized buffer. Since it involves two syscalls, it may be a bit slower. But, it might allow allow some tricks with selecting a payload buffer from a pool, based on the type of message and/or length
Unlike TCP which is a stream-based protocol, meaning that calls to recv don't exactly correspond to a call to send, UDP is packet based meaning that each recvfrom matches with exactly one sendto. This also means you need to take care of how large each message you send is.
If you send a UDP datagram that is larger that what can be contained in a IP packet, the UDP message will be fragmented across multiple UDP packets, increasing the chance of data loss. That's something you want to avoid. Also, if you're using IPv6, you'll get an error when you attempt to send because IPv6 doesn't support fragmentation.
What does this mean in relation to what you're doing? It means that, roughly speaking, your messages shouldn't be any larger than about 1450 bytes, so you can use that value as the size of your input buffer. Then you can use the return value of recvfrom to see how many bytes were actually read. If your messages are larger than that, you should break them up into multiple messages.
As with any UDP based protocol, you need to account for the case where messages get lost and they need to be retransmitted, or if messages come out of order.
Actually answer to this question was quite simple.
Given:
unsigned char buffer[BUFFERSIZE];
int bytes = fill_buffer(buffer, sizeof(buffer)): // Returns number of filled bytes.
sendto(socket, buffer, bytes, 0, (struct sockaddr *)&server, sizeof(server))
The return value of recvfrom tells us how many bytes are received, although we make a full read,
int bytesReceived = recvfrom(socket, buffer, sizeof(buffer), 0, NULL, NULL);
// Process bytesReceived number of bytes in the buffer

Maximum data size that can be sent and received using sockets, at once?(TCP socket)

I am designing a game which has master and multiple players. They send and receive data using TCP sockets.
Players transfer character strings between themselves via TCP sockets.The programs are being executed in red hat linux 6 os .
The character string transferred between players is of the type
char chain[2*hops+10];
The player code on sender side is
len = send(to,chain,sizeof(chain),0);
if (len != sizeof(chain)) {
perror("send");
exit(1);}
The code where player receives the data is like this :
char chain[2*hops+10];
len = recv(current,chain,sizeof(chain),0);
The value of hops is same for both the players.
For hops value till around 8000 it is working fine, but once the hops value crosses some point, the same program is not working. I believe data is not transferred in one go.
Is there a maximum buffer size for send and recv buffer?
Note: The sockets between them are opened using this code:
s = socket(AF_INET, SOCK_STREAM, 0);
and then the usual connect and bind sockets on both sides.
TCP is a stream-oriented protocol (as implied by SOCK_STREAM). Data that an application sends or receives (in [maximum-sized] chunks) is not received or sent in same-sized chunks. Thus one should read from a socket until enough data to be processed have been received, then attempt to process said data, and repeat:
while (true) {
unsigned char buffer [4096] = {};
for (size_t nbuffer = 0; nbuffer < sizeof buffer
; nbuffer = MAX(nbuffer, sizeof buffer)) { /* Watch out for buffer overflow */
int len = recv (sockd, buffer, sizeof buffer, 0);
/* FIXME: Error checking */
nbuffer += len;
}
/* We have a whole chunk, process it: */
;
}
You can also handle partial sends on the other side as described here, much better than I ever would.

(C Socket Programming) Seperate send() calls from server ending up in same client recv() buffer

I was wondering if anyone could shed any light as to why two seperate send() calls would end up in the same recv() buffer using the loopback address for testing yet once switched to two remote machines they would require two recv() calls instead? I have been looking at the wireshark captures yet cant seem to make any sense as to why this would be occuring. Perhaps someone could critique my code and tell me where im going wrong. The two incoming messages from the server is of an undetermined length to the client. By the way i'm using BSD sockets using C in Ubuntu.
In the example shown below im parsing the entire buffer to extract the two seperate messages from it which i'll admit isn't an ideal approach.
-------SERVER SIDE--------
// Send greeting string and receive again until end of stream
ssize_t numBytesSent = send(clntSocket, greeting, greetingStringLen, 0);
if (numBytesSent < 0)
DieWithSystemMessage("send() failed");
//-----------------------------Generate "RANDOM" Message -----------------------
srand(time(NULL)); //seed random number from system clock
size_t randomStringLen = rand() % (RANDOMMSGSIZE-3); //generates random num
// betweeen 0 and 296
char randomMsg [RANDOMMSGSIZE] = "";
// declare and initialize allowable characteer set for the
const char charSet[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
if (randomStringLen) {
--randomStringLen;
for (size_t i = 0; i < randomStringLen; i++) {
int p = rand() % (int) (sizeof charSet - 1);
randomMsg[i] = charSet[p];
}
randomStringLen = strlen(randomMsg);
printf("Random String Size Before newline: %d\n", (int)randomStringLen);
strcat(randomMsg,"\r\n");
}
randomStringLen = strlen(randomMsg);
printf("Random String: %s\n", randomMsg);
//-----------------------------Send "RANDOM" Message ---------------------------
// Send greeting string and receive again until end of stream
numBytesSent = send(clntSocket, randomMsg, randomStringLen, 0);
if (numBytesSent < 0)
DieWithSystemMessage("send() failed");
//------------------------------------------------------------------------------
------CLIENT SIDE-------
//----------------------------- Receive Server Greeting ---------------------------
char buffer[BUFSIZE] = ""; // I/O buffer
// Receive up to the buffer size (minus 1 to leave space for
// a null terminator) bytes from the sender
ssize_t numBytesRcvd = recv(sock, buffer, BUFSIZE - 1, 0);
if (numBytesRcvd < 0)
DieWithSystemMessage("recv() failed");
buffer[numBytesRcvd] = '\0'; //terminate the string after calling recv()
printf("Buffer contains: %s\n",buffer); // Print the buffer
//printf("numBytesRecv: %d\n",(int)numBytesRcvd); // Print the buffer
//------------------------ Extracts the random message from buffer ---------------------------
char *randomMsg = strstr(buffer, "\r\n"); // searches from first occurance of substring
char randomMessage [BUFSIZE] = "";
strcat(randomMessage, randomMsg+2);
int randomStringLen = strlen(randomMessage)-2;
printf("Random Message: %s\n",randomMessage); // Print the buffer
char byteSize [10];
sprintf(byteSize,"%d", randomStringLen);
printf("ByteSize = %s\n",byteSize);
//----------------------- Send the number for random bytes recieved -------------------------
size_t byteStringLen = strlen(byteSize); // Determine input length
numBytes = send(sock, byteSize, byteStringLen, 0);
if (numBytes < 0)
DieWithSystemMessage("send() failed");
else if (numBytes != byteStringLen)
DieWithUserMessage("send()", "sent unexpected number of bytes");
shutdown(sock,SHUT_WR); // further sends are disallowed yet recieves are still possible
//----------------------------------- Recieve Cookie ----------------------------------------
On Unix systems recv and send are just special cases of the read and write that accepts additional flags. (Windows also emulates this with Winsock).
You shouldn't assume that one recv corresponds to one send because that's generally isn't true (just like you can read a file in multiple parts, even if it was written in a single write). Instead you should start each "message" with a header that tells you how long the message is, if it's important to know what were the separate messages, or just read the stream like a normal file, if it's not important.
TCP is a byte-stream protocol, not a message protocol. There is no guarantee that what you write with a single send() will be received via a single recv(). If you need message boundaries you must implement them yourself, e.g. with a length-word prefix, a type-length-value protocol, or a self-describing protocol like XML.
You're experiencing a TCP congestion avoidance optimization commonly referred to as the Nagle algorithm (named after John Nagle, its inventor).
The purpose of this optimization is to reduce the number of small TCP segments circulating over a socket by combining them together into larger ones. When you write()/send() on a TCP socket, the kernel may not transmit your data immediately; instead it may buffer the data for a very short delay (typically a few tens of milliseconds), in case another request follows.
You may disable Nagle's algorithm on a per-socket basis, by setting the TCP_NODELAY option.
It is customary to disable Nagle in latency-sensitive applications (remote control applications, online games, etc..).

domain socket fragmentation

I am using domain sockets (AF_UNIX) to communicate between two threads for inter process communication. This is chosen to work well with libev: I use it on the recv end of the domain socket. This works very well except that the data I am sending is constant 4864 bytes. I cannot afford to get this data fragmented. I always thought domain sockets won't fragment data, but as it turns out it does. When the communication is at its peak between the threads, I observe the following
Thread 1:
SEND = 4864 actual size = 4864
Thread 2:
READ = 3328 actual size = 4864
Thread 1:
SEND = 4864 actual size = 4864
Thread 2:
READ = 1536 actual size = 4864
As you can see, thread 2 received the data in fragments (3328 + 1536). This is really bad for my application. Is there anyway we can make it not fragment it? I understand that IP_DONTFRAG can be set to only AF_INET family? Can someone suggest an alternative?
Update: sendto code
ssize_t
socket_domain_writer_dgram_send(int *domain_sd, domain_packet_t *pkt) {
struct sockaddr_un remote;
unsigned long len = 0;
ssize_t ret = 0;
memset(&remote, '\0', sizeof(struct sockaddr_un));
remote.sun_family = AF_UNIX;
strncpy(remote.sun_path, DOMAIN_SOCK_PATH, strlen(DOMAIN_SOCK_PATH));
len = strlen(remote.sun_path) + sizeof(remote.sun_family) + 1;
ret = sendto(*domain_sd, pkt, sizeof(*pkt), 0, (struct sockaddr *)&remote, sizeof(struct sockaddr_un));
if (ret == -1) {
bps_log(BPS_LOGGER_RD, ASL_LEVEL_ERR, "Domain writer could not connect send packets", errno);
}
return ret;
}
SOCK_STREAM by definition doesn't preserve message boundaries. Try again with SOCK_DGRAM or SOCK_SEQPACKET:
http://man7.org/linux/man-pages/man7/unix.7.html
On the other hand, consider that you may be passing messages larger than your architecture page size. For example, for amd64, a memory page is 4K. If that's a problem for any reason it might make sense to split the packets in 2.
Note however, that's not a real issue for the packets to arrive fragmented. It's common to have a packet assembler in the receiving end of the socket. What's wrong with implementing it ?
4864 + 3328 = 8192. My guess is that you're transmitting two 4864-byte packets back to back in some cases, and it's filling an 8 KB kernel buffer somewhere. IP_DONTFRAG isn't applicable because IP is not involved here — the "fragmentation" you're seeing is happening via a completely different mechanism.
If all the data you're transmitting consists of packets, you would do well to use a datagram socket (SOCK_DGRAM) instead of a stream. This should make the send() block when the kernel buffer doesn't have sufficient space to store an entire packet, rather than allowing a partial write through, and will make each recv() return exactly one packet, so you don't need to deal with framing.

select()ed socket fails to to recv() complete data

With the following pseudo-Python script for sending data to a local socket:
s = socket.socket(AF_UNIX, SOCK_STREAM)
s.connect("./sock.sock")
s.send("test\n")
s.send("aaa\0")
s.close()
My C program will randomly end up recving the following buffers:
test\n
test\n<random chars>
test\naaa (as expected)
The socket is being recv()'d after select() points that the socket is readable. Question is, how to avoid the first two cases?
And side question: Is it possible to send the following two messages from that script:
asd\0
dsa\0
And have select() to show the socket as readable on each of those sends, or will it only do that if I run the script again (restarting the socket client connection) and sending a message for each connect?
At a guess, the len argument to recv specifies a maximum amount of data to read, not the precise amount to be returned. recv is free to return any amount of data up to len bytes instead.
If you want to read a specific number of bytes, call recv in a loop.
int bytes = 0;
while (bytes < len) {
int remaining = len - bytes;
int read = recv(sockfd, buf+bytes, remaining, 0);
if (read < 0) {
// error
break;
}
bytes += read;
}
As noted by junix, if you'll need to send unpredictable amounts of data, consider defining a simple protocol that either starts each message with a note of its length or ends with a particular byte or sequence of bytes.

Resources