I'm using libevent for creating a small p2p network.
Let's say that I want to send 10,000 bytes buffer from one node to another, due to libevent the max size that can be send in one time is 4,096 bytes (so 4,096 + 4,096 + 1,808).
Is it technically possible (in TCP protocol) for the node that receives the data, to know the total number of bytes that will be sent when receives the first 4,096 bytes ?
No. It is, however, common for the application protocol to tell the receiver how many bytes to expect. With HTTP 1.1, it's the Content-Length header. You can also use length prefix along the lines of <len><data>. PostgresSQL uses that protocol between the front and backend.
Related
I'm trying to count total send bytes from my program, but I can't get accurate value.
All my functions call a single function that send data to my server using send() function.
In this function, i get return of send() and sum into global counter. This is working fine.
But when I compare to 'iftop' utility (sudo iftop -f 'port 33755'), I'm getting more data on iftop then in my app....and my guess if because of tcp headers/protocol data. I really don't know how to calculate this. I'm sending packets using send() and variable data length, so I'm not sure if is possible to detect/calculate TCP packet size from there. I know that each TCP packet send TCP header, but I'm not sure how many packets is sent.
May I assume that every call to send(), if data length is less than 1518 (TCP packet size limite?), than it's only one TCP packet and I need to sum TCP Header length? Even if I sent one byte? If so, how much is these extra-bytes from TCP structure?!
For information: I'm using GCC on linux as compiler.
Tks!
How to count bytes using send() in C, including protocol size?
There is no reliable way to do so from within your program. You can compute a minimum total number of bytes required to transmit data of the total payload size you count, subject to a few assumptions, but you would need to monitor from the kernel side to determine the exact number of bytes.
May I assume that every call to send(), if data length is less than
1518 (TCP packet size limite?), than it's only one TCP packet and I
need to sum TCP Header length?
No, that would not be a safe assumption. The main problem is that the kernel does not necessarily match the data transferred by each send() call to its own sequence of packets. It may combine data from multiple send()s into a smaller number of packets. Additionally, however, it may use either a smaller MTU or a larger one than Ethernet's default of 1500 bytes, depending on various factors, and, furthermore, you need to fit packet headers into the chosen MTU, so the payload carried by one packet is smaller than that.
I suspect you're making this too hard. If this is a task that has been assigned to you -- a homework problem, for example -- then my first guess would be that it is intended that you count only the total payload size, not the protocol overhead. Alternatively, if you do need to account for the overhead, then my guess would be that you are supposed to estimate, based on measured or assumed characteristics of the network. If you've set this problem for yourself, then I can only say that people generally make one of the two computations I just described, not the one you asked about.
I have a socket server and client, both written in C.
These clients (multiple clients and one server) send information to the server, but these pieces of information must be as small as possible.
Currently, every 'packet' from the client is about 40-60 bytes and my client sends about 10 packets/second.
My protocol sends 2 bytes with packet size + payload + 2 bytes for 'end of TX', then the server compares the size + 2 bytes of EOT.
The size and two bytes of EOT are necessary because I can have those two bytes in the payload too, so size is considered too. This is working without any problem.
Now, I need to 'protect' this payload and I don't know how.
OpenSSL is not an option since I've made some tests and, using a 2048 byte certificate, the smallest 'encrypted' packet size is 256 bytes, even using 50 bytes as original data.
I don't need such strong encryption... in fact, something simple will be even better, since my client could run on embedded systems with low processing power. Probably something with one fixed 'key' on both ends would be perfect, but I don't have any idea how to do this or if there is already some standard method.
I think that any solution will increase my packet size... this is fine, but not from 50 bytes to 256 bytes (like using openSSL).
How can I protect the data with at most a moderate increase in the packet size?
I am implementing a simple network stack using UDP sockets and I wish to send about 1 MB of string data from the client to the server. However, I am not aware if there is a limit to the size of the length in the UDP sendto() API in C. If there is a limit and sendto() won't handle packetization beyond a limit then I would have to manually split the string into smaller blocks and then send them.
Is there a limit on the length of the buffer? Or does the sendto() API handle packetization by itself.
Any insight is appreciated.
There's no API limit on sendto -- it can handle any size the underlying protocol can.
There IS a packet limit for UDP -- 64K; if you exceed this, the sendto call will fail with an EMSGSIZE error code. There are also packet size limits for IP which differ between IPv4 and IPv6. Finally, the low level transport has an MTU size which may or may not be an issue. IP packets can be fragemented into multiple lower level packets and automatically reassembled, unless you've used an IP_OPTIONS setsockopt call to disable fragmentation.
The easiest way to deal with all this complexity is to make your code flexible -- detect EMSGSIZE errors from sendto and switch to using smaller messages if you get it. The also works well if you want to do path MTU discovery, which will generally accept larger messages at first, but will cut down the maximum message size when you send one that ends up exceeding the path MTU.
If you just want to avoid worrying about it, a send of 1452 bytes or less is likely to always be fine (that's the 1500 byte normal ethernet payload max minus 40 for a normal IPv6 header and 8 for a UDP header), unless you're using a VPN (in which case you need to worry about encapsulation overhead).
I'm implementing a small application using UDP (in C). A server sends to a client the data from a given file in chunks of given amount (ex. 100 bytes / call). The client downloads the file and saves it somewhere. The catch is that the client can receive a parameter saying how many bytes to read / call.
My problem is when the server sends 100 bytes / call, and the client is set to read only 15 bytes / call. The other 85 bytes are lost, because the message is removed from the UDP queue.
Is there a way to read these messages in chunks without removing them from the queue until they're completely read?
UDP does not allow for chunked reading like TCP does. Reading a UDP message is an all-or-nothing operation, you either read the whole message in full or none of it at all. There is no in-between. Because of that, UDP-based protocols either use fixed-sized messages, or require both parties to dynamically negotiate the message sizes (like TrivialFTP does, for example).
There is no reason for a UDP protocol to require sending a byte size for each message. The message size itself implicitly dictates the size of the data inside of the message.
If you absolutely must determine the message size before actually reading the message, you could try calling recvfrom() with the MSG_PEEK flag, and give it a large buffer to copy data into (at least 64K, which a UDP message will never exceed, unless you are using IPv6 Jumbograms, but that is a separate issue). The output will tell you the actual size of the message that is still in the queue. However, if you go this route, then you may as well just drop the MSG_PEEK flag and always read using 64K buffers so there is no possibility of dropping data due to insufficient buffer sizes.
You can create a Thread to read the data from UDP Buffer infinitely And save the data to a circle-buffer. Than the client consume the data with your speed. If the Buffer is overflow,You can do nothing.Because the server's sending speed is quicker than the client's.
which is the beast approach to send packets that can be of different size using TCP sockets in C?
I wonder because we're trying to write a multiplayer games that needs a protocol which has many kinds of packets of different sizes.. according to recv documentation I can get how many bytes have been read but how should I manage to dispatch packets only when they are exaclty full?
Suppose that I have packets with a 5 bytes header that contains also the length of the payload.. should I use circular buffers or something like that to dispatch packets when ready and keep new partials?
Create a static variable which represents the packet header, this variable will be five bytes long. Create an associated integer which counts how many of those five bytes have yet been read. Create a second integer which counts how many bytes of the "content" have been read. Zero both those integers. Create an associated char * pointer which eventually will point to the received packet content.
As data arrives (e.g., select indicates so), read the five bytes of header. You may receive these bytes gradually, thus you need the first integer count variable. Account for the header bytes you have received here.
When you are done receiving the header, sanity check it. Are the size values possible to satisfy (e.g. not greater than 2^30)? If so, malloc a buffer of that size or that size plus the header. (If you want the header contiguous, allocate sufficient space, then memcpy it into your new buffer.)
Now, as data arrives, place it in your allocated buffer. Account for the received bytes in the second integer you created. When you have received all the bytes the header called for, then repeat all the above.
you can design a custom header for your packet transmission, which specifies packet length, indexing info (if packet fragmentation is implemented) and some hashing if you need.
some rough pseudocode as follows :
recv(socket, headerBuf, headerSize, MSG_WAITALL);
nPacketSize = headerBuf[16]; //sample
nByteRead = 0;
while (nByteRead != nPacketSize)
{
nByteToRead = nPacketSize - nByteRead;
nCurRead = recv(socket, someBuf, nByteToRead, MSG_PARTIAL);
nByteRead += nCurRead;
Sleep(0); // Yield processor
}
TCP is a stream based protocol, not a datagram one. That means that there isn't necessarily a one to one correspondence between the number and size of buffers passed to send and what comes out of receive. That means that you need to implement your own "protocol" on top of TCP.
Examples of such protocols include HTTP, where HTTP messages are delineated by two consecutive carriage return, line feed pairs. \r\n\r\n. The only concern with such delineation is to make sure that the pattern can't occur in the body of the message, either that or to make sure it is escaped. Other protocols create a header which contains the information necessary to correctly identify and read the next piece of information. I can't think of an application that does this off the top of my head. You could even go for a hybrid approach that combines the two.