so I have two processes, one client-process one server-process. The user can issue a command to the client, when a user enters a command the client will send the command length to the server, and after that it will send the actual command.
The server then sends back first the length of the response and then a response.
I can do 5-30 commands or so with no problem at all, but at some point it fails to read enough bytes, despite the correct response size being received.
The server sends the response in the following way:
str[0] = '\0';
unsigned long int totalSize = 0;
while ((fgets(outBuf, MAXOUTPUT, myFile)) != NULL)
{
strcat(str, outBuf);
}
uint32_t *un = 0;
totalSize = strlen(str);
*un = htonl(totalSize);
result= send(clientFD, un, sizeof(uint32_t), 0);
if(result < 1)
{
printf("Failed sending message size to client");
exit(-1);
}
while(token != NULL)
{
size_t length = strlen(token);
token[length] = '\n';
write(clientFD, token, length + 1);
token = strtok(NULL, "\n");
}
The client has received the message length correctly(verified with prints) and reads the response this way:
result = read(socketFD, recvBuf, bufferlen); //bufferlen is response size
if(result < bufferlen)
{
perror("read()");
exit(-1);
}
I have verified that the client receives the correct message length every time, including the last one where it fails to read.
So my question is: What are likely reasons that my read sometimes fail to retrieve the full response? It happens after doing about 5-30 commands or so usually, and the perror returned is Error 0 (aka no error to be found).
As an additional note, the commands tested are
ls -la, ls -l, ls.
I have not found a pattern in which commands cause the crash, but I have combined them a lot.
Also: Both the client and server are 32 bit and being run on the same machine locally.
read() (especially on sockets) returns as soon as some data is available, it may always return less bytes than you asked for. In this case, you will need to repeat the read until you have read enough data:
size_t bytes_read = 0;
while (bytes_read < bufferlen) {
result = read(socketFD, recvBuf + bytes_read, bufferlen - bytes_read);
if (result < 0) {
perror("read()");
exit(-1);
}
bytes_read += result;
}
The TCP socket works on a byte stream concept. The server is adding bytes to the byte stream, and the client is consuming them. The socket need not send all of the bytes at once; it will eventually send them and they will be read in order at the other end. Messages are not guaranteed to be kept whole. You encounter a problem when you can read the bufferlen field but the whole corresponding message has not arrived yet.
Your client needs to continue reading from the socket until bufferlen bytes have been read.
Also be aware that the act of reading the bufferlen field may also need to be completed with multiple reads.
Related
I want to implement protocol using TCP sockets in C which works kind of this way:
Client connects to server and sends filename that it wants to
download
Server reads that value and checks whether it is valid filename (does it exist on the server) + sends ACCEPT or FAILURE status to client
Client reads that status and prepares itself to download + sends READY status to server
Server sends file and closes connection
Server code:
char response[128];
int bytes_read;
while ((bytes_read = read(info.socket, response, 128)) > 0) {}
if (valid_request(files, files_count, response)) {
write(info.socket, MC_ACCEPT, 4);
} else {
write(info.socket, MC_FAILURE, 4);
}
Client code:
int w_status = write(sck, requested_file, strlen(requested_file));
if (w_status < 0) {
fprintf(stderr, "Error writing to socket. Status: %d", w_status);
exit(1);
}
char status[4];
while ((resp = read(sck, status, 4)) > 0) {}
if (strcmp(status, MC_ACCEPT) == 0) {
printf("ACCEPTED!\n");
} else if (strcmp(status, MC_ACCEPT) == 0) {
printf("FAILURE\n");
} else {
printf("DONT KNOW\n");
}
close(sck);
The problem is that server freezes itself on the read() part. It looks like client sends the filename and waits for server response (with status) but server is frozen at read().
Am I somehow blocking the TCP socket? What is wrong with my reasoning?
On the server side:
char response[128];
int bytes_read;
while ((bytes_read = read(info.socket, response, 128)) > 0) {}
You try to read those 128 chars in several read calls. But it will block forever until client closes the socket (it's TCP connected, there's always something to read unless peer closes connection).
And if the data arrives in more than 1 chunk, your code is incorrect, because the first chunk will be overwritten by the next one, and so on. You have to change the offset of your buffer, and do not attempt to read 128 bytes every time, or you'll get stuck.
int bytes_read = 0;
while (bytes_read < 128)
{
int currently_read = read(info.socket, response + currently_read, 128-bytes_read);
bytes_read += currently_read;
}
On the client side, same kind of issues too:
You seem to wait for 4 chars.
You try to read those 4 chars in the first read. But you don't check if 4 chars are actually read (return code discarded).
After that, you read using a loop until you get 0 bytes. But since connection doesn't end, you're stuck there.
What you want is to read exactly 4 bytes before doing something else.
And increase your buffer size & null-terminate your string or strcmp will fail.
char status[5];
status[4] = '\0';
int nb_read = 0;
while (nb_read < 4)
{
int currently_read = read(sck, status + nb_read, 4-nb_read);
nb_read += currently_read;
}
The problem is that you're not processing the request inside the while loop:
while ((bytes_read = read(info.socket, response, 128)) > 0) {}
This keeps looping until read() returns 0, which happens when the client closes the connection, or gets an error and returns -1. After it reads the request from the client, it goes back and calls read() again. Since the client hasn't sent anything else, this blocks.
This is why killing the client gets it unstuck. That closes the connection, so it gets EOF and read() returns 0.
You need to process the input inside the loop:
while ((bytes_read = read(info.socket, response, sizeof response -1)) > 0) {
response[bytes_read] = '\0'; // add string null terminator
if (valid_request(files, files_count, response)) {
write(info.socket, MC_ACCEPT, 4);
} else {
write(info.socket, MC_FAILURE, 4);
}
}
I was wondering if anyone could shed any light as to why two seperate send() calls would end up in the same recv() buffer using the loopback address for testing yet once switched to two remote machines they would require two recv() calls instead? I have been looking at the wireshark captures yet cant seem to make any sense as to why this would be occuring. Perhaps someone could critique my code and tell me where im going wrong. The two incoming messages from the server is of an undetermined length to the client. By the way i'm using BSD sockets using C in Ubuntu.
In the example shown below im parsing the entire buffer to extract the two seperate messages from it which i'll admit isn't an ideal approach.
-------SERVER SIDE--------
// Send greeting string and receive again until end of stream
ssize_t numBytesSent = send(clntSocket, greeting, greetingStringLen, 0);
if (numBytesSent < 0)
DieWithSystemMessage("send() failed");
//-----------------------------Generate "RANDOM" Message -----------------------
srand(time(NULL)); //seed random number from system clock
size_t randomStringLen = rand() % (RANDOMMSGSIZE-3); //generates random num
// betweeen 0 and 296
char randomMsg [RANDOMMSGSIZE] = "";
// declare and initialize allowable characteer set for the
const char charSet[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
if (randomStringLen) {
--randomStringLen;
for (size_t i = 0; i < randomStringLen; i++) {
int p = rand() % (int) (sizeof charSet - 1);
randomMsg[i] = charSet[p];
}
randomStringLen = strlen(randomMsg);
printf("Random String Size Before newline: %d\n", (int)randomStringLen);
strcat(randomMsg,"\r\n");
}
randomStringLen = strlen(randomMsg);
printf("Random String: %s\n", randomMsg);
//-----------------------------Send "RANDOM" Message ---------------------------
// Send greeting string and receive again until end of stream
numBytesSent = send(clntSocket, randomMsg, randomStringLen, 0);
if (numBytesSent < 0)
DieWithSystemMessage("send() failed");
//------------------------------------------------------------------------------
------CLIENT SIDE-------
//----------------------------- Receive Server Greeting ---------------------------
char buffer[BUFSIZE] = ""; // I/O buffer
// Receive up to the buffer size (minus 1 to leave space for
// a null terminator) bytes from the sender
ssize_t numBytesRcvd = recv(sock, buffer, BUFSIZE - 1, 0);
if (numBytesRcvd < 0)
DieWithSystemMessage("recv() failed");
buffer[numBytesRcvd] = '\0'; //terminate the string after calling recv()
printf("Buffer contains: %s\n",buffer); // Print the buffer
//printf("numBytesRecv: %d\n",(int)numBytesRcvd); // Print the buffer
//------------------------ Extracts the random message from buffer ---------------------------
char *randomMsg = strstr(buffer, "\r\n"); // searches from first occurance of substring
char randomMessage [BUFSIZE] = "";
strcat(randomMessage, randomMsg+2);
int randomStringLen = strlen(randomMessage)-2;
printf("Random Message: %s\n",randomMessage); // Print the buffer
char byteSize [10];
sprintf(byteSize,"%d", randomStringLen);
printf("ByteSize = %s\n",byteSize);
//----------------------- Send the number for random bytes recieved -------------------------
size_t byteStringLen = strlen(byteSize); // Determine input length
numBytes = send(sock, byteSize, byteStringLen, 0);
if (numBytes < 0)
DieWithSystemMessage("send() failed");
else if (numBytes != byteStringLen)
DieWithUserMessage("send()", "sent unexpected number of bytes");
shutdown(sock,SHUT_WR); // further sends are disallowed yet recieves are still possible
//----------------------------------- Recieve Cookie ----------------------------------------
On Unix systems recv and send are just special cases of the read and write that accepts additional flags. (Windows also emulates this with Winsock).
You shouldn't assume that one recv corresponds to one send because that's generally isn't true (just like you can read a file in multiple parts, even if it was written in a single write). Instead you should start each "message" with a header that tells you how long the message is, if it's important to know what were the separate messages, or just read the stream like a normal file, if it's not important.
TCP is a byte-stream protocol, not a message protocol. There is no guarantee that what you write with a single send() will be received via a single recv(). If you need message boundaries you must implement them yourself, e.g. with a length-word prefix, a type-length-value protocol, or a self-describing protocol like XML.
You're experiencing a TCP congestion avoidance optimization commonly referred to as the Nagle algorithm (named after John Nagle, its inventor).
The purpose of this optimization is to reduce the number of small TCP segments circulating over a socket by combining them together into larger ones. When you write()/send() on a TCP socket, the kernel may not transmit your data immediately; instead it may buffer the data for a very short delay (typically a few tens of milliseconds), in case another request follows.
You may disable Nagle's algorithm on a per-socket basis, by setting the TCP_NODELAY option.
It is customary to disable Nagle in latency-sensitive applications (remote control applications, online games, etc..).
I'm writing a C program to transfer a file of fixed size, a little over 2Mb, from a server to a client. I'm using TCP sockets on Linux and the code I wrote is the following:
Server (sender)
while (1) {
int nread = read(file, buffer, bufsize);
if (nread == 0) // EOF
break;
if (nread < 0) {
// handle errors
}
char* partial = buffer;
while (nread > 0) {
int nwrite = write(socket, partial, nread);
if (nwrite <= 0) {
// handle errors
}
nread -= nwrite;
partial += nwrite;
}
}
// file sent
shutdown(socket, SHUT_WR);
Client (receiver)
while (filesize > 0) {
nread = read(socket, buffer, bufsize);
if (nread == 0) {
// EOF - if we reach this point filesize is still > 0
// so the transfer was incomplete
break;
}
if (nread < 0) {
// handle errors
}
char* partial = buffer;
while (nread > 0) {
nwrite = write(file, partial, nread);
if (nwrite <= 0) {
// handle errors
}
nread -= nwrite;
partial += nwrite;
filesize -= nwrite;
}
}
if (filesize > 0) {
// incomplete transfer
// handle error
}
close(socket);
When testing the code on my laptop (both client and server "are" on localhost and the communication happen on the loopback interface), sometimes the client exits because read received an EOF, and not because it received all filesize bytes. Since i use a shutdown on the server, this should mean that there is no other data to read.
(Note that the server sent all the bytes and executed the shutdown correctly)
Can you explain me why this is happening?
Where are the missing bytes gone?
-----
EDIT 1 - Clarifications
Some users asked a couple of clarifications so i am posting the answers here:
The program is using TCP blocking sockets
The filesize is a fixed value and is hardcoded in both client and server.
No special socket options as, for example, SO_LINGER are enabled/used.
When the error occur, the server (sender) has already sent all the data and executed the shutdown correctly.
The error, as of today, never happened when testing the application with the client and the server on different machines (transfer over a real network interface and not a loopback interface)
EDIT 2
User Cornstalks pointed me to a really interesting article about the, non always reliable, behaviours of TCP.
The article, which is well worth a read, describe a few tricks useful when sending an unknown amount of data between TCP sockets. The tricks described are the followings:
Take advantage of the SO_LINGER option on the sender. This will help to keep the socket open, upon a call to close(2) or shutdown(2), until all the data has successfully been sent.
On the receiver, beware of pending readable data before the actual receiving loop. It could lead to an immediate reset being sent.
Take advantage of shutdown(2) to signal the receiver the the sender has done sending data.
Let the receiver know the size of the file that will be sent before actually sending the file.
Let the receiver acknowledge the sender that the receiving loop is over. This will help to prevent the sender from closing the socket too soon.
After reading the article, i upgraded my code to implement the tricks number 1 and 5.
This is how i implemented trick number 5:
Server (sender)
// sending loop ...
// file sent
shutdown(socket, SHUT_WR);
// wait acknowledgement from the client
ack = read(socket, buffer, bufsize);
if (ack < 0) {
// handle errors
}
Client (receiver)
// receiving loop..
if (filesize > 0) {
// incomplete transfer
// handle error
}
// send acknowledgement to the server
// this will send a FIN and trigger a read = 0 on the server
shutdown(socket, SHUT_WR);
close(socket);
What about tricks number 2, 3 and 4?
Trick number 2 is not needed because as soon as the server accepts the connection the application proceed to the file transfer. NO extra messages are exchanged.
Trick number 3 is already implemented
Trick number 4 is also already implemented. As mentioned earlier the file size is hardcoded, so there is no need to exchange it.
Did this solve my original problem?
NO my problem was not solved. The error is still happening, and as of today, it only happened when testing the application with both client and server on localhost.
What do you think?
Is there a way to prevent this?
You're:
assuming that read fills the buffer, even though
you're defending magnificently against write() not writing the entire buffer.
You need to do (1), and you don't need to do (2) because you're in blocking mode and POSIX assures that write() doesn't return until all the data is written.
A simple version of both loops:
while ((nread = read(inFD, buffer, 0, sizeof buffer)) > 0)
{
write(outFD, buffer, 0, nread);
}
if (nread == -1)
; // error
A more correct version would check the result of write() for errors of course.
I'm trying to troubleshoot a strange C programming problem.
I'm entering a URL on a client program and then transferring that URL to a server program. The only issue is, that when the server program receives the URL it's missing it's first two characters. So, if the url is http://www.google.com what the server reports is receiving is "tp://www.google.com."
The weird thing is that it's not some partial send problem. I'm checking on the number of bytes sent and it's claiming that the entire message is sent. The issue is that on the receiving end it's only claiming that it's getting a small chunk of the data. The receiver reports back that it's received the message length -2. Here is the code on the receiving end:
printf("%s \n", "Connected. Receive length of URL to wget.");
if ((messageSize = recv(acceptDescriptor, &urlLength, sizeof (int), 0)) == -1) {
perror("recv URL length");
exit(1);
}
urlSizeInt = atoi(urlLength);
char url[urlSizeInt];
printf("%s %d \n", "urlSizeInt: ", urlSizeInt);
printf("%s \n", "Receive URL to wget.");
if((messageSize = recv(acceptDescriptor, &url, 13, 0)) == -1) {
perror("recv URL");
exit(1);
}
Sending code:
printf("%s \n", "Connected");
//connected to first stepping stone in the chain.
//transfer the length of the URL
if (send(socketDescriptor, urlLengthStr, strlen(urlLengthStr), 0) == -1){
perror("send URL Length");
exit(0);
}
//transfer the URL
printf("%s %d \n", "strenlen(url): ",strlen(url));
printf("%s %s \n", "url: ",url);
int sent;
int totalSent=0;
if((sent=send(socketDescriptor, url, strlen(url), 0))==-1){
perror("send URL");
exit(0);
}
printf("%s %d \n", "sent: ",sent);
Send Output:
Connected
strenlen(url): 13
url: http://www.cs
sent: 13
Receive Output:
Connected. Receive length of URL to wget.
urlSizeInt: 13
Receive URL to wget.
messageSize: 11
URL Received: tp://www.cs
Code to encode the length as a char for sending:
char* url = "http://www.cs";
int urlLength = strlen(url);
char* urlLengthStr;
sprintf(urlLengthStr, "%d", urlLength);
Thanks for posting the full code. the problem lies in the way you send the UrlLength. Because you always recv sizeof(int) bytes, your first read is consuming the first bytes of the sent URL.
Start from the beginning - assume you don't send the URL Length.
Problem : URLs are variable length. How can the receiver know when it's read it all?
Solution : Send the length before
This is fine, unless you encode the length as a string, because that introduces another problem
Problem : The URL length may be variable length ("1", "12","1234"). How can the receiver know when it's read it all?
Solution : Haven't we been here before somewhere...
There are a couple of ways out of this recursive problem:
Solution a : Encode the URL length as a fixed size field. (you can just send the binary representation of an int, although beware of byte ordering problems - or you could encode it as a fixed-width ascii field , e.g, "00000124"
TX (ignoring byte ordering issues mentioned by JimR)
int urlLength = strlen(url);
send(socketDescriptor, &urlLength, sizeof(int), 0)
RX:
int urlLength;
recv(socketDescriptor, &urlLength, sizeof(int), 0)
Solution b: Use a termination character (often null, or linefeed) to indicate the end of the URL. Just read bytes in a loop until the terminator is reached. This also solves the 'partial recv' problem you would otherwise encounter.
When you're reading from a socket, you're reading a stream, not a file.
See here for good guidelines for writing network code.
Even though the sending side may have sent all the data at once, that does not guarantee that you will see all the data at once. You have to loop on recv, taking into account the number of bytes received for each call. If recv returns 0 the socket was closed or there was an error and you will receive no more data from that socket.
With those things in mind... Kinda sorta pseudo code, I didn't test this, but hope it gives you the idea:
int expectedLength = readLengthFromSocket( socket, sizeof( int ) );
int bytesRead = 0;
char buffer[expectedLength];
bytesRead = recv( socket, buffer, ... );
runningLength = 0;
if( bytesRead < 1 )
// Socket closed or there was an error, handle that here
else
{ runningLength += bytesRead;
while( runningLength < expectedLength )
{
bytesRead = recv( socket, buffer + runningLength, ... );
if( bytesRead < 1 )
// Socket closed or there was an error, handle that here
break;
else
runningLength += bytesRead;
}
}
Note that reading and writing int, long, long long, short and the unsigned variants over the network typically require byte swapping. Reading a buffer of bytes does not.
See here for an explanation.
Hi i have written a server application which accepts a name from the client which usually is a file name.It opens the file ,reads the contents into a buffer and then transmits the buffer over the ethernet using send().But the problem arises in the client side where all the bytes are not received successfully.I receive only a part of what i send.
For your reference ,here's the code snippet for the server side:
Server:
fp = fopen(filename,"r+");
strcpy(str,"");
fseek(fp, 0L, SEEK_END);
size = ftell(fp);
fseek(fp, 0L, SEEK_SET);
fread(str, size, 1,fp);
fclose(fp);
printf("Size of the file is : %d\n",size);
sprintf(filename, "%d", size);
n = send(nsd, filename, strlen(filename), 0);
while(size > 0){
n = send(nsd, str, strlen(str), 0);
printf("%d bytes sent successfully\n",n);
if(n == 0) break;
sentbytes = sentbytes + n;
size = size - sentbytes;
}
Please help me with writing the client app.I am currently confused about how to go about writing it.Shall i place the recv() part in a while(1) loop so that the client keeps running until all the bytes have been received successfully?
EDITED
For starters, you could both read from the file and write to the socket in chunks at the same time.
Since, you are transferring data over TCP, remember that data is transferred reliably as a stream and not as messages. So, don't make assumptions about how the data is recv'd except for the order.
Here is how it could be written:
open socket
open file
size_of_file = read_file_size(file);
send(socket, &size_of_file, sizeof(int), ...)
while (all is written)
read fixed chunk from file
write as much was read to the socket
cleanup // close file, socket
As for the recv part, I think it is best you send the file size over as an integer and keep reading in a while loop until you have recv'd as many bytes as you are sending from the server.
It's like this:
recv(socket, &size_of_msg, sizeof(int), ...)
while(all is read)
read fixed chunk from the socket
cleanup
Well I see atleast some issue with the way you are sending message over socket.
First from the man page of fread:
The function fread() reads nmemb elements of data, each size bytes
long, from the stream pointed to by stream, storing them at the loca-
tion given by ptr.
and what you are trying is this:
fread(str, size, 1,fp);
I assume what you meant was
fread(str, 1,size,fp);
Though it shold not casue the issue.
But the problem lies here:
n = send(nsd, str, strlen(str), 0);
printf("%d bytes sent successfully\n",n);
if(n == 0) break;
sentbytes = sentbytes + n;
size = size - sentbytes;
Though you are decreasing 'size' by decreasing by number of bytes successfully send, where are you extending str to point to new buffer location where data will be send.This will only resend initial bytes of the buffer repeatedly.
str += n; //Assuming str is char*
will solve your issue.
Using strlen doesn't seem appropriate. You've read the file, you know how long it is, so why do strlen? Either you'll just get the same result (so it's redundant) or you'll get something else (so it's a bug).
"Shall i place the recv() part in a while(1) loop so that the client keeps running until all the bytes have been received successfully?"
Something like that. Never presume that a recv() call got everything that was sent -- tcp/ip breaks messages into packets at a lower level, and recv() will return after reading whatever amount of data has actually been received at whatever point. You don't have to worry about that directly, except in so far as you do need to use some kind of protocol to indicate how long a message is so the receiver knows how much to read, then eg.:
char buffer[4096];
int msgsz = 600, // see below
sofar = 0,
cur;
while (sofar < msgsz) {
cur = recv (
socket_fd,
&buffer[sofar],
msgsz - sofar,
0
);
if (cur == -1) {
// error
break;
} else if (cur == 0) {
// disconnected
break;
}
sofar += cur;
}
WRT msgsz, you would include this somewhere in a fixed length header, which is read first. A simple version of that might be just 4 bytes containing a uint32_t, ie, an int with the length. You could also use a null terminated string with a number in it, but that means reading until '\0' is found.