I'm sending a C struct over UDP
struct packet{
int numInt;
int* intList; //malloc'ed as (sizeof(int)*numInt)
}
It will be serialized as [numInt][intList[0]]...[intList[numInt-1]].
My understanding is that calling recvfrom on UDP will read the entire packet, even if the buffer doesn't hold that many bytes. Is using a really large buffer the only option I have?
You could pass MSG_PEEK to recvfrom to find out exactly how big the buffer needs to be. So just recvfrom a few bytes with MSG_PEEK to find numInt and then recvfrom the real thing (this time without MSG_PEEK).
The standard says something about MSG_PEEK, but kernel.org spells it better:
MSG_PEEK
This flag causes the receive operation to return data from the
beginning of the receive queue without removing that data from the
queue. Thus, a subsequent receive call will return the same data.
Obviously at some point you will start wondering if doubling the number of system calls to save memory is worth it. I think it isn't.
UDP packets are sent and received as a whole. if you receive it, the size is right. The only thing you have to do is to supply a big enough buffer on read() or recv() or recfrom(). The length field inside the payload is redundant, since the read() will tell you the correct size. It is also dangerous, since it relies on the sender and reciever having the same byte order.
You could try using a small buffer, just large enough to get numInt, with the MSG_PEEK flag set. Then you can find out the size you actually need, and receive again without MSG_PEEK to get the whole thing.
I'm pretty sure recvfrom will read up to as many bytes as is told to it by its 3rd argument, len. If there are fewer bytes available, it will return what is there. If there are more, it will return up to len bytes. You may have to make additional calls to obtain all the data your are expecting.
Related
I have written a program in Linux using C/C++ that reads multicast packets and tries to understand that a specific event occurred or not as quickly as possible. Latency is the key point here.
In the protocol, first two bytes represent the message type.
In my current implementation, I read the first two bytes and decide how many bytes I should read for the payload according to the message type. Namely, I perform 2 read operations for 1 packet. One of the read operations is for the packet length and the other is for the payload. So, there are 2 I/O operations.
Alternatively, I could do that, I read as much as I can, check the first 2 bytes, let's say it is N, go for N bytes and form the packet1 and packet2. If there are remaining bytes after reading packet1 and packet2, read more bytes and again process the byte buffer as above. In this method, I do 1 I/O but it is required to traverse in the byte buffer.
Which one is faster theoretically? I know I must implement and measure both but I just wanted to hear your suggestions.
Thanks
The fastest method I know of is:
Open a raw packet socket (AF_PACKET)
Implement a BPF filter, that filters the packets you need as specific as possible
Switch to a memory-mapped ringbuffer (PACKET_MMAP/PACKET_RX_RING)
Read the packets directly from memory instead of using recv(). This can be done using poll() or, alternatively, by busy-looping over the in-memory packet meta-data to avoid the poll() syscall.
Process the packet directly in the ring-buffer (zero-copy)
Mark the buffer as "free for reuse"
This way, no syscalls at all are necessary, the path through the kernel is short and the latency should be minimal.
For more information, see the packet mmap kernel documentation
I need to write a couple of C++ applications on Linux, one to receive data via UDP and the second TCP.
The only thing I'm unsure about is regarding the buffer.
How do I choose what size buffer?
If I make the buffer large enough, am I guaranteed to avoid scenarios where half of a packet is at the end of my buffer and I need to copy the bytes to the beginning and then receive the remaining half of the packet?
I am going to use the Linux socket API functions if it matters.
If I make the buffer large enough, am I guaranteed to avoid scenarios
where half of a packet is at the end of my buffer and I need to copy
the bytes to the beginning and then receive the remaining half of the
packet?
Based on the above paragraph, I'm going to surmise that the buffer you are referring to is the application-space buffer that you pass into your recv() calls, and not the in-kernel buffer that the networking stack maintains on your application's behalf.
For UDP, the answer is simple: Your buffer needs to be large enough to hold the largest possible datagram you expect to receive. Since UDP datagrams are typically less than 1500 bytes (to avoid fragmentation) and in all cases are <= 65507 bytes (since that is the maximum datagram size the UDP protocol supports), you can always make your receive buffer 65507 bytes long, or smaller if you want to save a bit on RAM usage.
For TCP, the protocol is stream-based, so the amount of data written in to your recv-buffer by a given recv() call is unrelated to packet sizes. Another consequence of TCP being stream-based is that it doesn't do any message-framing -- that means you will have to handle partial messages regardless of how big or small you make your buffer. The only advantage of a larger TCP buffer is that it's a bit more efficient to handle more bytes at a time instead of fewer, again at the cost of using a little more RAM.
If I make the buffer large enough, am I guaranteed to avoid scenarios where half of a packet is at the end of my buffer and I need to copy the bytes to the beginning and then receive the remaining half of the packet?
For TCP: It doesn't matter. Packets are an implementation detail. The application doesn't even have to think about them. TCP is a byte-stream protocol and all you ever get from the API is a stream of bytes. Message boundaries are never preserved.
For UDP: Packets are still an implementation detail. You send and receive datagrams. Your read function always gets an entire datagram so long as your buffer is as large as the largest datagram your application protocol supports.
I read How large should my recv buffer be when calling recv in the socket library in order to understand buffer in read. There are yet some points that i wish to know about read buffer in tcp socket connection.
My application is sending video packets. when i set buff to 80000 sender could send the packets but when i set it less for example 8000 after sending few packets it stops with RST.
a)Is this buffer, TCP receive window?
b)Is there any relation between this buffer and .net.ipv4.tcp_rmem , .net.ipv4.tcp_wmem ?if yes, Should i set read buffer based on rmem or wmem?
I would greatly appreciate any responses
a)Is this buffer, TCP receive window?
No, it is just a buffer that you provide for the TCP stack to place bytes into when you call recv().
b)Is there any relation between this buffer and .net.ipv4.tcp_rmem ,
.net.ipv4.tcp_wmem?
No.
if yes, Should i set read buffer based on rmem or women?
You can pass any size buffer you want to recv(); it is unrelated to any of the above, except that there isn't any benefit to making the buffer you pass to recv() larger than the socket's current SO_RCVBUF size, since it's unlikely that recv() would ever return more bytes at once than can be present in the socket's internal buffer.
As for how to decide what size buffer to use -- consider that a larger buffer will (of course) take up more memory, and if you are allocating that buffer on the stack, a very large buffer might cause a stack overflow. On the other hand, a smaller buffer means that you can read fewer bytes with any given call to recv(), so you may have to call recv() more times to read in the same total number of bytes.
Note that number of bytes of data returned by recv() may be any number from 1 byte up to the total size of the buffer that you passed in to recv()'s third argument, and there is no way to predict how many bytes you'll get. In particular, with TCP the number of bytes you receive from any particular call to recv() will not have any correlation to the number of bytes previously passed to any particular call to send() on the sending side. So you just need to use a "reasonably sized" array (for whatever definition of "reasonably sized" you prefer) and recv() as many bytes into it as possible, and then handle that many bytes (based on recv()'s return value).
I'm having some doubts about the number of bytes I should write/read through a socket in C on Unix. I'm used to sending 1024 bytes, but this is really too much sometimes when I send short strings.
I read a string from a file, and I don't know how many bytes this string is, it can vary every time, it can be 10, 20 or 1000. I only know for sure that it's < 1024. So, when I write the code, I don't know the size of bytes to read on the client side, (on the server I can use strlen()). So, is the only solution to always read a maximum number of bytes (1024 in this case), regardless of the length of the string I read from the file?
For instance, with this code:
read(socket,stringBuff,SIZE);
wouldn't it be better if SIZE is 10 instead of 1024 if I want to read a 10 byte string?
In the code in your question, if there are only 10 bytes to be read, then it makes no difference whether SIZE is 10 bytes, 1,024 bytes, or 1,000,024 bytes - it'll still just read 10 bytes. The only difference is how much memory you set aside for it, and if it's possible for you to receive a string up to 1,024 bytes, then you're going to have to set aside that much memory anyway.
However, regardless of how many bytes you are trying to read in, you always have to be prepared for the possibility that read() will actually read a different number of them. Particularly on a network, when you can get delays in transmission, even if your server is sending a 1,024 byte string, less than that number of bytes may have arrived by the time your client calls read(), in which case you'll read less than 1,024.
So, you always have to be prepared for the need to get your input in more than one read() call. This means you need to be able to tell when you're done reading input - you can't rely alone on the fact that read() has returned to tell you that you're done. If your server might send more than one message before you've read the first one, then you obviously can't hope to rely on this.
You have three main options:
Always send messages which are the same size, perhaps padding smaller strings with zeros if necessary. This is usually suboptimal for a TCP stream. Just read until you've received exactly this number of bytes.
Have some kind of sentinel mechanism for telling you when a message is over. This might be a newline character, a CRLF, a blank line, or a single dot on a line followed by a blank line, or whatever works for your protocol. Keep reading until you have received this sentinel. To avoid making inefficient system calls of one character at a time, you need to implement some kind of buffering mechanism to make this work well. If you can be sure that your server is sending you lines terminated with a single '\n' character, then using fdopen() and the standard C I/O library may be an option.
Have your server tell you how big the message is (either in an initial fixed length field, or using the same kind of sentinel mechanism from point 2), and then keep reading until you've got that number of bytes.
The read() system call blocks until it can read one or more bytes, or until an error occurs.
It DOESN'T guarantee that it will read the number of bytes you request! With TCP sockets, it's very common that read() returns less than you request, because it can't return bytes that are still propagating through the network.
So, you'll have to check the return value of read() and call it again to get more data if you didn't get everything you wanted, and again, and again, until you have everything.
I'm working on a client-server app. My app is working with variable size packets, each packet has a header and a payload of variable length.
My dilemma is what is the best approach for handing the packets when doing recv.
Most of the tutorials I've came across suggest using a ring buffer but as far as I can tell it's more efficient to use a buffer whose size is twice the size of the biggest packet you can handle.
If I use a ring buffer I need an additional buffer for recv and then I need to copy the buffer in the ring buffer which means I need to do one or two memcpys to insert the buffer in the ring buffer
If I use the single buffer approach I only need one buffer which I can pass to recv call and a memmove call to move data to the start of the buffer when I got a full packet and there still is data belonging to another packet in the buffer.
Am I getting something wrong ?
PS. If you can point me to any source code/example where variable length packets are handled that would be helpful.
If I use a ring buffer I need an additional buffer for recv and then I need to copy the buffer in the ring buffer which means I need to do one or two memcpys to insert the buffer in the ring buffer
Yeah, two reads and writes, no big deal (*). But you don't need an additional buffer. For read just maximize read to the space left till the end of the ring buffer.
(*): If you are worried about the cost of additional syscall, for scatter/gather read/write there is recvmsg and sendmsg.