I wrote a server that should wait for messages from a client after opening a connection to it:
while(1){
if(recv(mySocket, buffer, 1000, 0) < 1){
continue;
}
printf("Message received: %s", buffer);
}
I checked with wireshark which packets were sent to this server, but for every packet sent there were 2 printf outputs.
My question is now where did I get this additional message from.
(The additional message are some random bytes. But every time the same.)
Your apparent expectations for the behavior of recv() are not justified. As #KarolyHorvath observed in comments, stream sockets (among which TCP-based sockets fall) have no sense whatever of "messages". In particular, network packets do not correspond to messages on a stream socket. POSIX has this to say about the behavior of recv(), in fact:
For stream-based sockets, [...] message boundaries shall be ignored.
Although that's more likely to have the effect of combining multiple "messages", it can also mean that a single message (as dispatched by a single send() call) is split over multiple recv() calls. It certainly will mean that if the buffer length you specify to recv() is less than the number of bytes actually received on the socket, but there are other circumstances in which that result could be obtained, too.
On success, recv() returns the number of bytes copied into the receive buffer. If you are genuinely trying to implement some sort of "message" exchange, then you can use that to help you split incoming data on message boundaries. Do recognize, however, that that constitutes implementing a message-passing protocol on top of a stream, so sender and receiver need to cooperate, at least implicitly, for it to work.
John Bollinger's answer is accurate and provides insight into what you should do to create a reliable client / server application.
Regarding your question, There is another problem that explains the actual output you see. the packet is most probably sent and received in a single chunk, as you observe with wireshark. The bug is in your server: You receive the data in a char array and print it directly as a string with printf. I suspect the packet does not contain the terminating '\0' to make the buffer a proper string for "%s". printf will output the packet contents plus whatever buffer contents is there until it reaches a '\0' byte, possibly invoking undefined behaviour. If the packet is split in several chunks, you may see the same contents several times, and random characters too.
Here is how you should fix your code:
char buffer[2000];
...
for (;;) {
ssize_t count = recv(mySocket, buffer, 1999, 0);
if (count >= 1) {
buffer[count] = '\0';
printf("Message received: |%s|", buffer);
}
}
Note that the buffer must be at least 1 byte longer than the maximum packet size, and this tracing method cannot handle embedded '\0' bytes in the packets.
Of course the packets can be sliced and diced on the way between the client and the server, so you must deal with this appropriately to implement a proper protocol.
Related
What are the best practices when sending (or writing) and recving (or reading) to/from a TCP socket ?
Assume usual blocking I/O on sockets. From what I understand :
writing (sending) should be fine without a loop, because it will block if the write buffer of the socket is full, so something like
if ((nbytes_w = write(sock, buf, nb)) < nb)
/* something bad happened : error or interrupted by signal */
should always be correct ?
on the other hand, there is no guaranty that one will read a full message, so one should read with
while ((nbytes_r = read(sock, buf, MAX)) > 0) {
/* do something with these bytes */
/* break if encounter specific application protocol end of message flag
or total number of bytes was known from previous message
and/or application protocol header */
}
Am I correct ? Or is there some "small message size" or other conditions allowing to read safely outside a loop ?
I am confused because I have seen examples of "naked reads", for instance in Tanenbaum-Wetherall:
read(sa, buf, BUF_SIZE); /* read file name in socket */
Yes you must loop on the receive
Once a week I answer a question where someones TCP app stops working for this very reason. The real killer is that they developped the client and server on the same machine, so they get loopback connection. Almost all the time a loopback will receive the send messages in the same blocks as they were sent. This makes it look like the code is correct.
The really big challenge is that this means you need to know before the loop how big the message is that you are going to receive. Possibilities
send a fixed length length (ie you know its , say, 4 bytes) first.
have a recognizable end sequence (like the double crlf at the end of an HTTP request.
Have a fixed size message
I would always have a 'pull the next n bytes' function.
Writing should loop too, but that easy, its just a matter of looping.
I am having trouble ending tcp stream. I am writing a simple server and client where the client connects to the server and the server displays a welcome message asking the client for a username.
The problem is, when the server writes the message, the client's read() gets blocked. It only gets unblocked when I call shutdown().
Server:
if (FD_ISSET(tcp_listenfd, &rset)) {
len = sizeof(cliaddr);
if ((new_confd = accept(tcp_listenfd, (struct sockaddr *) &cliaddr, &len)) < 0) {
perror("accept");
exit(1);
}
/* Send connection message asking for handle */
writen(new_confd, handle_msg, strlen(handle_msg));
/* Fork here or shutdown fd is inherited */
shutdown(new_confd, SHUT_WR);
Clients:
if ((connect(sock, (struct sockaddr *) server, sizeof(struct sockaddr_in))) < 0) {
perror("inet_wstream:connect");
exit(1);
}
s_welcome_msg[19] = '\0';
readn(sock, s_welcome_msg, 20); //Blocks here if shutdown() is not called in server
The readn() and writen() functions are adapted from "The Socket Networking API" by Stevens found here: http://www.informit.com/articles/article.aspx?p=169505&seqNum=9
How do I write a welcome message from the server without calling shutdown() and not having the client block? If more context is needed, I will post more code.
Note that readn() is designed to read() in a loop until either 20 bytes are read or there's EOF or an error on the socket. If the message the server sends is less than 20 bytes long, the client will block waiting for more data.
To prevent it from blocking, you could do a normal read() (or recv()) on the socket instead. In this case, that is likely to do what you want.
In general, you can't rely on being able to pair up write()s and read()s for TCP connections though. A single write() of a string "bar" could split the data up arbitrarily. As an extreme example, three successive read()s might return "b", "a", and "r". That particular example is unlikely, but for larger write()s and read()s you have to take this into account (and for smaller transmissions too, if you want to be perfectly safe).
To work around this issue, you will have to do your own buffering on the receiving end. The simplest solution in this case is to read() one character at a time (or to use readn() with exactly the amount of data you expect, if it is known). A more general solution is to read() as much data as is currently available (make sure to check the return value of read() to see how much data you get back!) into a buffer and only acting on the data whenever you've collected enough of it. A plain read() will not block as long as there's some data available to be read, but you might get back less data than you requested.
"Enough of it" would usually be a full "message" in your protocol. You will need some way to determine message boundaries. Two alternatives are length fields (usually the best solution in my experience) or message terminators. Both would be sent along with the rest of the data.
Update:
You have a bug in your null-termination logic by the way. Reading twenty bytes into s_welcome_msg will set s_welcome_msg[19] to the last byte read, overwriting your null terminator. If you want to read a 20-byte non-null-terminated string into s_welcome_msg and null-terminate it, s_welcome_msg will need to be 21 bytes long, and you will need to do s_welcome_msg[20] = '\0'.
read(n) will block until it receives the requested number of bytes
(and the receiving field is only 19 bytes, so it it did read 20 bytes
that would be a buffer overflow which is undefined behaviour and can/will result in a seg fault event)
I suggest, as one possible fix, a loop with a select() statement with a timeout
and when select() indicates some data available,
read only one byte
append that byte to the s_welcome_msg[] buffer
(while always checking that the buffer is not overflowed
which, generally, would mean only read a max of 18 bytes
so the read value would be a valid string)
Your code should make the read() be non-blocking
so it will not hang.
After reading the byte,
if input buffer not full (18 bytes read)
then loop back to the select() statement
If the select() timeout occurs,
then assume all the data has been read
and proceed to the next code statements after the select/read loop
Also remember to always 'refresh' the timeout value
on the select() statement parameter before
executing the select()
I am programming TCP server client. I sending the three string seperately using seperate send system call.
But receiving end i getting only single string that is first string which i send. remaining two string missed.
Below i given the part of my server client program.
client.c
char *info = "infolog";
char *size = "filesize";
char *end = "fileend";
send(client, info, strlen(info)+1, 0);
send(client, size, strlen(size)+1, 0);
send(client, end, strlen(end)+1, 0);
server.c
while ((read_size = recv(client, msg, sizeof(msg), 0))) {
printf("Data: %s\n", msg);
memset(msg, 0, sizeof(msg));
}
Actual output:
Data: infolog
Expected output
Data: infolog
Data: filesize
Data: fileend
Thanks.
Try printing out read_size. You probably have received all the messages already.
Due to Nagle's Algorithm, the sender probably batched up your three send() calls and sent a single packet to the server. While you can disable Nagle's algorithm, I don't think it's a good idea in this case. Your server needs to be able to handle receiving of partial data, and handle receiving more data than it expects.
You might want to look into using an upper-layer protocol for your messages, such as Google Protocol Buffers. Take a look at the techniques page, where they describe how they might do it: build up a protocol buffer, and write its length to the stream before writing the buffer itself. That way the receive side can read the length and then determine how many bytes it needs to read before it has a complete message.
TCP is not a message protocol but a byte stream protocol.
The three send-s could be recv-ed as a single input (or something else, e.g. in two or five recv etc....)
The application should analyze the input and buffer it to be able to splice it in meaningful messages.
the transmission may split or merge the messages, e.g. intermediate routers can and will split or merge the "packets".
In practice you'll better have some good conventions about your messages. Either decide that each message is e.g. newline terminated, or decide that it starts with some header giving its size.
Look at HTTP, SMTP, IMAP, SCGI or ONC/XDR (documented in RFC5531) as concrete examples. And document quite well your protocol (a minima, in long descriptive comments for some homework toy project, and more seriously, in a separate public document).
So in the case of TCP streams, one needs to keep track of how much is read of the message by each recv call. Partial reads can then be pieced together and ultimately the message can be received ok.
But for UDP messages, how should partial reads be handled (assuming that all UDP messages are small enough to avoid fragmentation)? As remaining data of partial messages seem to be discarded, is it just a matter of making sure that recvfrom returns the same size as the buffer that's been sent? If there is a difference it means it's a partial and defective message and it should be skipped.
So conceptually, while the TCP example needs a loop, the UDP example just needs an if statement.
Is this correct?
It is not possible to do partial reads in UDP. UDP guarantees the datagram received is as was sent, fragmented or not, so must have received the entire datagram before you can read it.
See: http://en.wikipedia.org/wiki/User_Datagram_Protocol, you can follow the references in there for more official sources.
As for reading into a buffer you will need a buffer at least the size of any possible datagram. Or another common way is to include the size on the datagram near the beginning of the datagram so you can read only those bytes to get that number then use a buffer the appropriate size.
Incorrect. If recv() returns the same length it was given, the message was either that length or greater. There is no way of telling which. The correct technique is to use a buffer one greater than the largest possible datagram expected. Then if you get that length, it must have been an error on the sender's part.
correct. However that if condition will work only if receiver knows in advance how many bytes are being sent by the sender in advance.
As already mentioned, closely related to this question is the need for a strategy to pass a properly sized (large enough) buffer to recv/recvmsg/recvfrom when using these with datagram protocols. For UDP, a simple and 100% reliable method is to pass a buffer of at least 65507 bytes, the maximum UDP payload size.
However, a cleaner method that I prefer is to explicitly ask recv() how many bytes the buffer needs to be. This can be accomplished thusly:
int buflen = recv(sockfd, NULL, 0, MSG_PEEK | MSG_TRUNC);
if (buflen < 0) {
// handle error
return;
}
uint8_t buf[buflen];
rxlen = recv(sockfd, buf, buflen, 0);
if (rxlen < 0) {
// again, handle error
return;
}
// Voila! We've received our entire datagram
// without need to know the maximum datagram
// size before runtime.
In Receiver, I have
recvfd=accept(sockfd,&other_side,&len);
while(1)
{
recv(recvfd,buf,MAX_BYTES-1,0);
buf[MAX_BYTES]='\0';
printf("\n Number %d contents :%s\n",counter,buf);
counter++;
}
In Sender , I have
send(sockfd,mesg,(size_t)length,0);
send(sockfd,mesg,(size_t)length,0);
send(sockfd,mesg,(size_t)length,0);
MAX_BYTES is 1024 and length of mesg is 15. Currently, It calls recv only one time. I want recv function to be called three times for each corresponding send. How do I achieve it?
In short: yes, it is blocking. But not in the way you think.
recv() blocks until any data is readable. But you don't know the size in advance.
In your scenario, you could do the following:
call select() and put the socket where you want to read from into the READ FD set
when select() returns with a positive number, your socket has data ready to be read
then, check if you could receive length bytes from the socket:
recv(recvfd, buf, MAX_BYTES-1, MSG_PEEK), see man recv(2) for the MSG_PEEK param or look at MSDN, they have it as well
now you know how much data is available
if there's less than length available, return and do nothing
if there's at least length available, read length and return (if there's more than length available, we'll continue with step 2 since a new READ event will be signalled by select()
To send discrete messages over a byte stream protocol, you have to encode messages into some kind of framing language. The network can chop up the protocol into arbitrarily sized packets, and so the receives do not correlate with your messages in any way. The receiver has to implement a state machine which recognizes frames.
A simple framing protocol is to have some length field (say two octets: 16 bits, for a maximum frame length of 65535 bytes). The length field is followed by exactly that many bytes.
You must not even assume that the length field itself is received all at once. You might ask for two bytes, but recv could return just one. This won't happen for the very first message received from the socket, because network (or local IPC pipe, for that matter) segments are never just one byte long. But somewhere in the middle of the stream, it is possible that the fist byte of the 16 bit length field could land on the last position of one network frame.
An easy way to deal with this is to use a buffered I/O library instead of raw operating system file handles. In a POSIX environment, you can take an open socket handle, and use the fdopen function to associate it with a FILE * stream. Then you can use functions like getc and fread to simplify the input handling (somewhat).
If in-band framing is not acceptable, then you have to use a protocol which supports framing, namely datagram type sockets. The main disadvantage of this is that the principal datagram-based protocol used over IP is UDP, and UDP is unreliable. This brings in a lot of complexity in your application to deal with out of order and missing frames. The size of the frames is also restricted by the maximum IP datagram size which is about 64 kilobytes, including all the protocol headers.
Large UDP datagrams get fragmented, which, if there is unreliability in the network, adds up to greater unreliability: if any IP fragment is lost, the entire packet is lost. All of it must be retransmitted; there is no way to just get a repetition of the fragment that was lost. The TCP protocol performs "path MTU discovery" to adjust its segment size so that IP fragmentation is avoided, and TCP has selective retransmission to recover missing segments.
I bet you've created a TCP socket using SOCK_STREAM, which would cause the three messages to be read into your buffer during the first recv call. If you want to read the messages one-by-one, create a UPD socket using SOCK_DGRAM, or develop some type of message format which allows you to parse your messages when they arrive in a stream (assuming your messages will not always be fixed length).
First send the length to be received in a fixed format regarding the size of length in bytes you use to transmit this length, then make recv() loop until length bytes had been received.
Note the fact (as also already mentioned by other answers), that the size and number of chunks received do not necessarly need to be the same as sent. Only the sum of all bytes received shall be the same as the sum of all bytes sent.
Read the man pages for recvand send. Especially read the sections on what those functions RETURN.
recv will block until the entire buffer is filled, or the socket is closed.
If you want to read length bytes and return, then you must only pass to recv a buffer of size length.
You can use select to determine if
there are any bytes waiting to be read,
how many bytes are waiting to be read, then
read only those bytes
This can avoid recv from blocking.
Edit:
After re-reading the docs, the following may be true: your three "messages" may be being read all-at-once since length + length + length < MAX_BYTES - 1.
Another possibility, if recv is never returning, is that you may need to flush your socket from the sender-side. The data may be waiting in a buffer to actually be sent to the receiver.