I must send 1 million bytes from A to B through UDP protocol.
These 1 million bytes can be an array of random characters. I understood that it's not possible to send packets larger than 65507 bytes, and UDP, as you all know, does not guarantee the arrival of messages.
I thought about sending several characters from A to B, unless B will not reach 1 million. Next, B must send a flag to A as a confirmation. But also here, there aren't any guarantees that the "ack" has reached the A end-system.
There is also an additional end-system C, where I could implement some additional code, if I need to.
I thought to create 2 processes for each end-system, one to send and receive the million bytes, and the second to send and receive the flags.
I have tried to implement it, but I am not very convinced. An important requisite is to avoid busy waiting.
I need a little help to move forward.
Related
My question is mainly educational so please don't tell me you shouldn't do that or you should do that. Thanks.
This is basically about how TCP works.
I'm implementing a HTTP client in C that basically just sends a HTTP GET and reads the response from ths server. Now I want to separate the headers from the body. The question is, is it possible to read from the socket byte-by-byte:
while(recv(sockfd, buffer, 1, 0))
{
// do whatever with buffer[0]
}
or is it so that: once the server wrote to the socket, say, 1000 bytes, so once the client reads even 1 bytes than all this message is "wasted" and cannot be read anymore?
Because I remember that when dealing with sockets sometime in the past that was how I understood that it is working.
Yes, in TCP that is possible. TCP does not have messages, only bytes, like a file. You can send 1000 bytes and then receive 1 byte 1000 times.
In fact that is a convenient way to start out, because if you try to receive more than 1 byte, you might not get the same number of bytes you asked for, you could get any number down to 1 byte. If you only ask to receive 1 byte, then you always get exactly 1 byte (unless the connection is closed).
It is inefficient, however, because the receive function uses a certain amount of CPU time just to process the fact that you want to receive - i.e. the function call itself costs a bit of CPU time. If you ask for, say, 100000 bytes at once, then you don't have to call the receive function 100000 times to get it. It's probably fast enough for you, just not as fast as it could be.
Although it is not good practice, it isn't "wasted". Most socket implementations on OS's do use a socket buffer, ranging from 4K till 6MB. So yes you can, as long as you read it fast engough.
But still, it is saver to just copy the stuf to your own memory-managed buffer..
I'm trying to do a multiplayer game in c, but when I send multiple package like "ARV 2\n\0" and "POS 2 0 0\n\0" from the server to the client (with send()), when I try to read them with recv(), he only found 1 package that appear to be the 2 package in 1..
So I'm asking, is that normal ? And if yes, how could I force my client to read 1 by 1 the packages ? (or my server to send them 1 by 1 if the problem come from the call send)
Thanks !
Short answer: Yes, this is normal. You are using TCP/IP, I assume. It is a byte stream protocol, there are no "packets". Network and OS on either end may combine and split the data you send in any way that fits in some buffers, or parts of network. Only thing guaranteed is, that you get the same bytes in same order.
You need to use your own packet framing. For text protocol, separate packets with, for example, '\0' bytes or newlines. Also note that network or OS may give you partial packets per single "read", so you need to handle that in your code as well. This is easiest if packet separator is single byte.
Especially for a binary protocol where there are no "unused" byte values to mark packet boundaries, you could write length of packet as binary data, then that many data bytes, then again length, data, and so on. Note that the data stream may get split to different "read" calls even in the middle of the length info as well (unless length is single byte), so you may need a few lines more of code to handle receiving split packets.
Another option would be to use UDP protocol, which indeed sends packets. But UDP packets may get lost or delivered in wrong order (and have a few other problems), so you need to handle that somehow, and this often results in you re-inventing TCP, poorly. So unless you notice TCP/IP just won't cut it, stick with that.
I'm trying to count total send bytes from my program, but I can't get accurate value.
All my functions call a single function that send data to my server using send() function.
In this function, i get return of send() and sum into global counter. This is working fine.
But when I compare to 'iftop' utility (sudo iftop -f 'port 33755'), I'm getting more data on iftop then in my app....and my guess if because of tcp headers/protocol data. I really don't know how to calculate this. I'm sending packets using send() and variable data length, so I'm not sure if is possible to detect/calculate TCP packet size from there. I know that each TCP packet send TCP header, but I'm not sure how many packets is sent.
May I assume that every call to send(), if data length is less than 1518 (TCP packet size limite?), than it's only one TCP packet and I need to sum TCP Header length? Even if I sent one byte? If so, how much is these extra-bytes from TCP structure?!
For information: I'm using GCC on linux as compiler.
Tks!
How to count bytes using send() in C, including protocol size?
There is no reliable way to do so from within your program. You can compute a minimum total number of bytes required to transmit data of the total payload size you count, subject to a few assumptions, but you would need to monitor from the kernel side to determine the exact number of bytes.
May I assume that every call to send(), if data length is less than
1518 (TCP packet size limite?), than it's only one TCP packet and I
need to sum TCP Header length?
No, that would not be a safe assumption. The main problem is that the kernel does not necessarily match the data transferred by each send() call to its own sequence of packets. It may combine data from multiple send()s into a smaller number of packets. Additionally, however, it may use either a smaller MTU or a larger one than Ethernet's default of 1500 bytes, depending on various factors, and, furthermore, you need to fit packet headers into the chosen MTU, so the payload carried by one packet is smaller than that.
I suspect you're making this too hard. If this is a task that has been assigned to you -- a homework problem, for example -- then my first guess would be that it is intended that you count only the total payload size, not the protocol overhead. Alternatively, if you do need to account for the overhead, then my guess would be that you are supposed to estimate, based on measured or assumed characteristics of the network. If you've set this problem for yourself, then I can only say that people generally make one of the two computations I just described, not the one you asked about.
i have a client which sends data to a server with 2 consecutive send calls:
send(_sockfd,msg,150,0);
send(_sockfd,msg,150,0);
and the server is receiving when the first send call was sent (let's say i'm using select):
recv(_sockfd,buf,700,0);
note that the buffer i'm receiving is much bigger.
my question is: is there any chance that buf will contain both msgs? of do i need 2 recv() calls to get both msgs?
thank you!
TCP is a stream oriented protocol. Not message / record / chunk oriented. That is, all that is guaranteed is that if you send a stream, the bytes will get to the other side in the order you sent them. There is no provision made by RFC 793 or any other document about the number of segments / packets involved.
This is in stark contrast with UDP. As #R.. correctly said, in UDP an entire message is sent in one operation (notice the change in terminology: message). Try to send a giant message (several times larger than the MTU) with TCP ? It's okay, it will split it for you.
When running on local networks or on localhost you will certainly notice that (generally) one send == one recv. Don't assume that. There are factors that change it dramatically. Among these
Nagle
Underlying MTU
Memory usage (possibly)
Timers
Many others
Of course, not having a correspondence between an a send and a recv is a nuisance and you can't rely on UDP. That is one of the reasons for SCTP. SCTP is a really really interesting protocol and it is message-oriented.
Back to TCP, this is a common nuisance. An equally common solution is this:
Establish that all packets begin with a fixed-length sequence (say 32 bytes)
These 32 bytes contain (possibly among other things) the size of the message that follows
When you read any amount of data from the socket, add the data to a buffer specific for that connection. When 32 bytes are reached, read the length you still need to read until you get the message.
It is really important to notice how there are really no messages on the wire, only bytes. Once you understand it you will have made a giant leap towards writing network applications.
The answer depends on the socket type, but in general, yes it's possible. For TCP it's the norm. For UDP I believe it cannot happen, but I'm not an expert on network protocols/programming.
Yes, it can and often does. There is no way of matching up sends and receive calls when using TCP/IP. Your program logic should test the return values of both send and recv calls in a loop, which terminates when everything has been sent or recieved.
which is the beast approach to send packets that can be of different size using TCP sockets in C?
I wonder because we're trying to write a multiplayer games that needs a protocol which has many kinds of packets of different sizes.. according to recv documentation I can get how many bytes have been read but how should I manage to dispatch packets only when they are exaclty full?
Suppose that I have packets with a 5 bytes header that contains also the length of the payload.. should I use circular buffers or something like that to dispatch packets when ready and keep new partials?
Create a static variable which represents the packet header, this variable will be five bytes long. Create an associated integer which counts how many of those five bytes have yet been read. Create a second integer which counts how many bytes of the "content" have been read. Zero both those integers. Create an associated char * pointer which eventually will point to the received packet content.
As data arrives (e.g., select indicates so), read the five bytes of header. You may receive these bytes gradually, thus you need the first integer count variable. Account for the header bytes you have received here.
When you are done receiving the header, sanity check it. Are the size values possible to satisfy (e.g. not greater than 2^30)? If so, malloc a buffer of that size or that size plus the header. (If you want the header contiguous, allocate sufficient space, then memcpy it into your new buffer.)
Now, as data arrives, place it in your allocated buffer. Account for the received bytes in the second integer you created. When you have received all the bytes the header called for, then repeat all the above.
you can design a custom header for your packet transmission, which specifies packet length, indexing info (if packet fragmentation is implemented) and some hashing if you need.
some rough pseudocode as follows :
recv(socket, headerBuf, headerSize, MSG_WAITALL);
nPacketSize = headerBuf[16]; //sample
nByteRead = 0;
while (nByteRead != nPacketSize)
{
nByteToRead = nPacketSize - nByteRead;
nCurRead = recv(socket, someBuf, nByteToRead, MSG_PARTIAL);
nByteRead += nCurRead;
Sleep(0); // Yield processor
}
TCP is a stream based protocol, not a datagram one. That means that there isn't necessarily a one to one correspondence between the number and size of buffers passed to send and what comes out of receive. That means that you need to implement your own "protocol" on top of TCP.
Examples of such protocols include HTTP, where HTTP messages are delineated by two consecutive carriage return, line feed pairs. \r\n\r\n. The only concern with such delineation is to make sure that the pattern can't occur in the body of the message, either that or to make sure it is escaped. Other protocols create a header which contains the information necessary to correctly identify and read the next piece of information. I can't think of an application that does this off the top of my head. You could even go for a hybrid approach that combines the two.