Related
Here is my code:
FILE *responseStorage = fopen("response", "w");
if(responseStorage == NULL){
//error handling
}
int receivedTotal = 0;
while(1){
ssize_t received = recv(sockfd, buf, BUFSIZ, 0);
if( received == -1 ){
//error handling
}
if( received == 0 ){
//end of stream
break;
}
receivedTotal += received;
if(fwrite(buf, received, 1, responseStorage) != 1){
//error handling
}
memset(buf, '\0', sizeof(buf));
}
fclose(responseStorage);
FILE *responseFile = fopen("response", "r");
char responseArray[receivedTotal];
if(fread(responseArray, receivedTotal, 1, responseFile) == 0){
//error
}
I am calling a ssize_t received = recv(sockfd, buf, BUFSIZ, 0);, recieving data from server, saving how much data I received in receivedTotal += received; and writing that data to with fwrite(buf, received, 1, responseStorage) to my file FILE *responseStorage. After that at the end of the stream loop breaks and I open responseStorage file in r mode, make an array of the size receivedTotal, char responseArray[receivedTotal];, and with fread(responseArray, receivedTotal, 1, responseFile write that data from responeStorage file to responseArray.
Is there a way to write dirrectly to a responseArray? I have to validate response later on so I need it in an array. I know I would have to dynamically alocate the space for array with malloc. I want to avoid using receivedTotal and responseStorage.
You're already reading from your socket into buf, so all you have to do is write buf to a dynamically allocated string rather than responseStorage. Like you say you just have ti handle memory space to fit your response.
The inefficient but really easy way to do this is to reallocate storage every time you read. You can allocate storage for the sum of the previous response reads and the new string in buf, then write both strings to newly allocated space. You know the sum of the length of these strings +1 for the null byte, so you don't have to worry much about available allocated space. This is however pretty expensive because the reads get copied over and over again.
The slightly more complex way to do it would be to allocate one probably fairly large main response string buffer, keep track of its length total allocated space and use strncat to continue to concatenate buf until it's length exceeds the remaining space in response, +1 (for terminating nil byte). When there's not enough room, you can call realloc to obtain more memory. realloc is not very effficient (by C standards) because it is likely to need to allocate different space, copy existing data, and then return a new pointer.
if you wanted to be really clever, you could allocate one big buffer, and send read a pointer at the offset of the next available spot in buffer. You still might need to grow buffer but at least you don't need to copy it. buf then becomes your response array. This is the implementation I'll demonstrate:
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
int main(int argc, char**argv){
size_t size = 2;
size_t len = 0;
char* fullbuf = malloc(sizeof(char)*size);
while(1) {
ssize_t b = read(STDIN_FILENO, fullbuf + len, sizeof(char) * (size - len-1) );
if( b < 0) {
perror("Couldn't read from stdin: ");
exit(2);
} else if( b == 0 ){
break;
}
if( b + len + 1 >= size) {
// time to allocate more memory
size = size * 2;
fullbuf = realloc(fullbuf, sizeof(char) * size);
if( fullbuf == NULL ){
fprintf(stderr, "Couldn't allocate %zd more bytes of memory\n", size);
exit(1);
}
}
len += b;
}
fullbuf[len] = '\0'; //terminating null space
printf("%s", fullbuf);
}
For this demonstration I read from stdin instead of a socket, but same idea. I read only as much data as is available in buf. When it's full (but the terminating byte), I double its space. Note that I set fullbuf to the output of realloc - it may be the same address, it may not.
To prove it works, I started at the rather insane 2 byte buffer, and double from there so there's lots of realloc calls. I grabbed 32k of lorem ipsum to use as input.
$ du -h file.txt
32K file.txt
$ shasum -a 256 file.txt
346f2adbd1fdca6bf3b03fb0a4d2fd0030e3363e9a9c5d1e22747e1bcc316e37 file.txt
$ ./t < file.txt | shasum -a 256
346f2adbd1fdca6bf3b03fb0a4d2fd0030e3363e9a9c5d1e22747e1bcc316e37 -
Awesome, if the shasums are the same, that means I outputted file.txt exactly.
In Windows, most efficient way is to use VirtualAlloc, reserving a very large chunk and committing initially one page (4096 bytes). If BUFSIZ is larger, commit more pages to make it fit. Assign the address to responsiveArray. Read the first packet in there. For the next recv calls, keep committing extra pages and adjust the recv buffer address, making sure, that there is least BUFSIZ free bytes available. You will have continuous address space, no fragmentation, no hogging extra memory, no rewriting, and no writing to disk.
I'm using sockets in C, I have a client that will send a message and the size of the message and the server that will receive this message in buffers of shorter size. The message is being sent correctly, but I'm having problems when I try to concatenate all the buffers in a single string.
The client:
char *buffer = "Hello from the client you're receiving this message";
int bytes_sent;
long msgSize = strlen(buffer);
printf("Buffer Size: %ld\n", msgSize);
int msgFileSize = send(SocketFD, &msgSize, sizeof(msgSize),0);
bytes_sent = send(SocketFD, buffer, strlen(buffer), 0);
The server:
char buffer[8];
ssize_t recsize;
long msgSize;
int msize = recv(ConnectFD, &msgSize, sizeof(msgSize), 0);
int total_byt = 0;
printf("Full Message Size: %ld\n", msgSize);
char full_msg[msgSize];
while (total_byt < msgSize) {
recsize = recv(ConnectFD, (void*)buffer, sizeof buffer, 0);
total_byt += recsize;
strcat(full_msg, buffer);
if (recsize < 0) {
fprintf(stderr, "%s\n", strerror(errno));
exit(EXIT_FAILURE);
}
printf("recsize: %d\n ", (int)recsize);
sleep(1);
printf("datagram: %.*s\n", (int)recsize, buffer);
}
printf("full message: %s\n", full_msg);
The output of the server looks like this:
Full Message Size: 51
recsize: 8
datagram: Hello fr
recsize: 8
datagram: om the c
recsize: 8
datagram: lient yo
recsize: 8
datagram: u're rec
recsize: 8
datagram: eiving t
recsize: 8
datagram: his mess
recsize: 3
datagram: age
full message: ��9�Hello from the client you're receiving this message mess
Few problems here.
strlen returns length excluding the null char.
long msgSize = strlen(buffer);
Thus you need to have 1 additional place to hold null char.
char full_msg[msgSize+1];
full_msg[0] = '\0';
There is undefined behavior with strcat as buffer is not null terminated.
recsize = recv(ConnectFD, (void*)buffer, sizeof buffer, 0);
strcat(full_msg, buffer);
use
strncpy(full_msg, buffer, sizeof buffer);
Finally null terminate the full_msg after loop to be sure.
while (total_byt < msgSize) {
….
}
full_msg[total_byt] = '\0';
It looks like your full message variable is not initialized. It is starting out with random data in it, which is causing your unexpected results. Use the following code to initialize it:
char full_msg[msgSize] = "";
Also, using strcat in the way you are is not safe. You must keep track of how much space is left in full_msg or you will have a buffer overrun situation. strncat is the function that can solve this problem.
There's absolutely no point reading into a separate buffer and then concatenating that into another buffer. Just read directly into full_msg.
Even if you were going to append block by block, strcat is not the right way to do it. strcat needs to start by finding the current end of the output buffer, which it can only do by sequentially scanning from the beginning looking for a NUL byte. As the buffer gets more and more data, those scans get longer and longer, leading to quadratic time complexity. This particular unnecessarily quadratic append is often called a "Schlemiel the Painter" algorithm.
The scan is unnecessary because you already know where the new data should be appended, since you carefully keep track of how many bytes you've already read. So each successive block should be placed total_byt bytes past the beginning of the buffer. (That is, at location full_msg+totalbyt.) You also know how long the data to append is, so you can use memcpy to put the newly-read chunk in the right place.
Uding memcpy will also avoid the problems which will occur if data can contain NUL bytes. (Since strcat returns immediately when it encounters a NUL, your current code will not work on messages which do include NUL.)
Note thatrecv does not NUL-terminate the input received, so your strcat will also do the wrong thing if the recv doesn't fill the buffer (and it only works with the buffer as written because it happens to be the case that
There is at least one other problem with your code: nothing guarantees that recv will stop reading at the end of a message, since the sockets library doesn't know where TCP messages end. (UDP message endpoints are marked, but you can't receive a UDP message in multiple chunks.) Again, since you keep track of the number of bytes read, and you know the length of the message (once you've read that data, at least), you can easily work out what the maximum number of bytes to read is.
This is for a Linux system, in C. It involves network programming. It is for a file transfer program.
I've been having this problem where this piece of code works unpredictably. It either is completely successful, or the while loop in the client never ends. I discovered that this is because the fileLength variable would sometimes be a huge (negative or positive) value, which I thought was attributed to making some mistake with ntohl. When I put in a print statement, it seemed to work perfectly, without error.
Here is the client code:
//...here includes relevant header files
int main (int argc, char *argv[]) {
//socket file descriptor
int sockfd;
if (argc != 2) {
fprintf (stderr, "usage: client hostname\n");
exit(1);
}
//...creates socket file descriptor, connects to server
//create buffer for filename
char name[256];
//recieve filename into name buffer, bytes recieved stored in numbytes
if((numbytes = recv (sockfd, name, 255 * sizeof (char), 0)) == -1) {
perror ("recv");
exit(1);
}
//Null terminator after the filename
name[numbytes] = '\0';
//length of the file to recieve from server
long fl;
memset(&fl, 0, sizeof fl);
//recieve filelength from server
if((numbytes = recv (sockfd, &fl, sizeof(long), 0)) == -1) {
perror ("recv");
exit(1);
}
//convert filelength to host format
long fileLength = ntohl(fl);
//check to make sure file does not exist, so that the application will not overwrite exisitng files
if (fopen (name, "r") != NULL) {
fprintf (stderr, "file already present in client directory\n");
exit(1);
}
//open file called name in write mode
FILE *filefd = fopen (name, "wb");
//variable stating amount of data recieved
long bytesTransferred = 0;
//Until the file is recieved, keep recieving
while (bytesTransferred < fileLength) {
printf("transferred: %d\ntotal: %d\n", bytesTransferred, fileLength);
//set counter at beginning of unwritten segment
fseek(filefd, bytesTransferred, SEEK_SET);
//buffer of 256 bytes; 1 byte for byte-length of segment, 255 bytes of data
char buf[256];
//recieve segment from server
if ((numbytes = recv (sockfd, buf, sizeof buf, 0)) == -1) {
perror ("recv");
exit(1);
}
//first byte of buffer, stating number of bytes of data in recieved segment
//converting from char to short requires adding 128, since the char ranges from -128 to 127
short bufLength = buf[0] + 128;
//write buffer into file, starting after the first byte of the buffer
fwrite (buf + 1, 1, bufLength * sizeof (char), filefd);
//add number of bytes of data recieved to bytesTransferred
bytesTransferred += bufLength;
}
fclose (filefd);
close (sockfd);
return 0;
}
This is the server code:
//...here includes relevant header files
int main (int argc, char *argv[]) {
if (argc != 2) {
fprintf (stderr, "usage: server filename\n");
exit(1);
}
//socket file descriptor, file descriptor for specific client connections
int sockfd, new_fd;
//...get socket file descriptor for sockfd, bind sockfd to predetermined port, listen for incoming connections
//...reaps zombie processes
printf("awaiting connections...\n");
while(1) {
//...accepts any incoming connections, gets file descriptor and assigns to new_fd
if (!fork()) {
//close socket file discriptor, only need file descriptor for specific client connection
close (sockfd);
//open a file for reading
FILE *filefd = fopen (argv[1], "rb");
//send filename to client
if (send (new_fd, argv[1], strlen (argv[1]) * sizeof(char), 0) == -1)
{ perror ("send"); }
//put counter at end of selected file, and find length
fseek (filefd, 0, SEEK_END);
long fileLength = ftell (filefd);
//convert length to network form and send it to client
long fl = htonl(fileLength);
//Are we sure this is sending all the bytes??? TEST
if (send (new_fd, &fl, sizeof fl, 0) == -1)
{ perror ("send"); }
//variable stating amount of data unsent
long len = fileLength;
//Until file is sent, keep sending
while(len > 0) {
printf("remaining: %d\ntotal: %d\n", len, fileLength);
//set counter at beginning of unread segment
fseek (filefd, fileLength - len, SEEK_SET);
//length of the segment; 255 unless last segment
short bufLength;
if (len > 255) {
len -= 255;
bufLength = 255;
} else {
bufLength = len;
len = 0;
}
//buffer of 256 bytes; 1 byte for byte-length of segment, 255 bytes of data
char buf[256];
//Set first byte of buffer as the length of the segment
//converting short to char requires subtracting 128
buf[0] = bufLength - 128;
//read file into the buffer starting after the first byte of the buffer
fread(buf + 1, 1, bufLength * sizeof(char), filefd);
//Send data too client
if (send (new_fd, buf, sizeof buf, 0) == -1)
{ perror ("send"); }
}
fclose (filefd);
close (new_fd);
exit (0);
}
close (new_fd);
}
return 0;
}
Note: I've simplified the code a bit, to make it clearer I hope.
Anything beginning with //... represents a bunch of code
You seem to be assuming that each send() will either transfer the full number of bytes specified or will error out, and that each one will will pair perfectly with a recv() on the other side, such that the recv() receives exactly the number of bytes sent by the send() (or error out), no more and no less. Those are not safe assumptions.
You don't show the code by which you set up the network connection. If you're using a datagram-based protocol (i.e. UDP) then you're more likely to get the send/receive boundary matching you expect, but you need to account for the possibility that packets will be lost or corrupted. If you're using a stream-based protocol (i.e. TCP) then you don't have to be too concerned with data loss or corruption, but you have no reason at all to expect boundary-matching behavior.
You need at least three things:
An application-level protocol on top of the network-layer. You've got parts of that already, such as in how you transfer the file length first to advise the client about much content to expect, but you need to do similar for all data transferred that are not of pre-determined, fixed length. Alternatively, invent another means to communicate data boundaries.
Every send() / write() that aims to transfer more than one byte must be performed in a loop to accommodate transfers being broken into multiple pieces. The return value tells you how many of the requested bytes were transferred (or at least how many were handed off to the network stack), and if that's fewer than requested you must loop back to try to transfer the rest.
Every recv() / read() that aims to transfer more than one byte must be performed in a loop to accommodate transfers being broken into multiple pieces. I recommend structuring that along the same lines as described for send(), but you also have the option of receiving data until you see a pre-arranged delimiter. The delimiter-based approach is more complicated, however, because it requires additional buffering on the receiving side.
Without those measures, your server and client can easily get out of sync. Among the possible results of that are that the client interprets part of the file name or part of the file content as the file length.
Even though you removed it from that code I'll make an educated guess and assume that you're using TCP or some other stream protocol here. This means that the data that the servers sends is a stream of bytes and the recv calls will not correspond in the amount of data they get with the send calls.
It is equally legal for your first recv call to just get one byte of data, as it is to get the file name, file size and half of the file.
You say
When I put in a print statement,
but you don't say where. I'll make another educated guess here and guess that you did it on the server before sending the file length. And that happened to shake things enough that the data amounts that were sent on the connection just accidentally happened to match what you were expecting on the client.
You need to define a protocol. Maybe start with a length of the filename, then the filename, then the length of the file. Or always send 256 bytes for the filename regardless of how long it is. Or send the file name as a 0-terminated string and try to figure out the data from that. But you can never assume that just because you called send with X bytes that the recv call will get X bytes.
I believe the issue is actually a compound of everything you and others have said. In the server code you send the name of the file like this:
send (new_fd, argv[1], strlen (argv[1]) * sizeof(char), 0);
and receive it in the client like this:
recv (sockfd, name, 255 * sizeof (char), 0);
This will cause an issue when the filename length is anything less than 255. Since TCP is a stream protocol (as mentioned by #Art), there are no real boundaries between the sends and recvs, which can cause you to receive data in odd places where you are not expecting them.
My recommendation would be to first send the length of the filename, eg:
// server
long namelen = htonl(strlen(argv[1]));
send (new_fd, &namelen, 4, 0);
send (new_fd, argv[1], strlen (argv[1]) * sizeof(char), 0);
// client
long namelen;
recv (sockfd, &namelen, 4, 0);
namelen = ntohl(namelen);
recv (sockfd, name, namelen * sizeof (char), 0);
This will ensure that you are always aware of exactly how long your filename is and makes sure that you aren't accidentally reading your file length from somewhere in the middle of your file (which is what I expect is happening currently).
edit.
Also, be cautious when you are sending sized numbers. If you use the sizeof call on them, you may be sending and receiving different sizes. This is why I hard-coded the sizes in the send and recv for the name length so that there is no confusion on either side.
Well, after some testing, I discovered that the issue causing the problem did have something to do with htonl(), though I had still read the data incorrectly in the beginning. It wasn't that htonl() wasn't working at all, but that I didn't realize a 'long' has different lengths depending on system architecture (thanks #tofro). That is to say the length of a 'long' integer on 32-bit and 64-bit operating systems is 4 bytes and 8 bytes, respectively. And the htonl() function (from arpa/inet.h) for 4-byte integers. I was using a 64-bit OS, which explains why the value was being fudged. I fixed the issue by using the int32_t variable (from stdint.h) to store the file length. So the main issue in this case was not that it was becoming out of sync (I think). But as for everyone's advice towards developing an actual protocol, I think I know what exactly you mean, I definitely understand why it's important, and I'm currently working towards it. Thank you all for all your help.
EDIT: Well now that it has been several years, and I know a little more, I know that this explanation doesn't make sense. All that would result from long being larger than I expected (8 bytes rather than 4) is that there's some implicit casting going on. I used sizeof(long) in the original code rather than hardcoding it to assume 4 bytes, so that particular (faulty) assumption of mine shouldn't have produced the bug I saw.
The problem is almost certainly what everyone else said: one call to recv was not getting all of the bytes representing the file length. At the time I doubted this was the real cause of the behaviour I saw, because the file name (of arbitrary length) I was sending through was never partially sent (i.e. the client always created a file of the correct filename). Only the file length was messed up. My hypothesis at the time was that recv mostly respected message boundaries, and while recv can possibly only send part of the data, it was more likely that it was sending it all and there was another bug in my code. I now know this isn't true at all, and TCP doesn't care.
I'm a little curious as to why I didn't see other unexpected behaviour as well (e.g. the file name being wrong on the receiving end), and I wanted to investigate further, but despite managing to find the files, I can't seem to reproduce the problem now. I suppose I'll never know, but at least I understand the main issue here.
I have the following code in server.c
while (1) {
char msg[1024];
recv(fd, msg, 1024, 0);
}
From client, first, I sent "hello world".
And then I received "hello world" in server.
I then sent "hexx" to server.
But I received "hexxo world" on server.
It seems as if msg wasn't cleared fully.
Please let me know what I'm doing wrong.
Cheers.
You are allocating 1024 bytes for message and never zeroing that data. You are just reading into the same buffer over and over. First pass it reads "hello world" in the second pass you are putting "hexx" in the same starting address resulting in "hexxo world"
As DoxyLover mentions you can null terminate the newly read string and get what you are looking for.
tcp recv not clearing old data
Who said it would? Not that it should be necessary. You're ignoring the count returned by recv(). After calling recv(), there are three possibilities:
Return value of -1: an error; call perror().
Return value of 0: end of stream: close the socket and exit the read loop.
Return value is positive: you have received exactly that many bytes into your buffer, and you should not assume that any data beyond that count is valid.
At present you're doing none of these things correctly.
read() does not null terminate the buffer. Therefore, you are seeing the old data left over in the buffer.
What you want is something like:
while (1) {
char msg[1024];
ssize_t n = recv(fd, msg, 1023, 0);
if (n >= 0)
msg[n] = '\0';
}
Note that I am limiting the read to one less than the size of the buffer to allow space for the null byte. Obviously, this only works with text data. With binary data, you need to record the return value from read() and use that as a byte count when processing the buffer. For example, to copy data from one socket to another:
while (1) {
char msg[1024];
ssize_t n = recv(fd, msg, 1024, 0);
if (n > 0)
send(second_fd, msg, n, 0);
}
All of this is very simplified - you need error checking, check the return from read() to make sure any bytes were received, check the return from send() to make sure all of the byte were sent, etc.
Hi i have written a server application which accepts a name from the client which usually is a file name.It opens the file ,reads the contents into a buffer and then transmits the buffer over the ethernet using send().But the problem arises in the client side where all the bytes are not received successfully.I receive only a part of what i send.
For your reference ,here's the code snippet for the server side:
Server:
fp = fopen(filename,"r+");
strcpy(str,"");
fseek(fp, 0L, SEEK_END);
size = ftell(fp);
fseek(fp, 0L, SEEK_SET);
fread(str, size, 1,fp);
fclose(fp);
printf("Size of the file is : %d\n",size);
sprintf(filename, "%d", size);
n = send(nsd, filename, strlen(filename), 0);
while(size > 0){
n = send(nsd, str, strlen(str), 0);
printf("%d bytes sent successfully\n",n);
if(n == 0) break;
sentbytes = sentbytes + n;
size = size - sentbytes;
}
Please help me with writing the client app.I am currently confused about how to go about writing it.Shall i place the recv() part in a while(1) loop so that the client keeps running until all the bytes have been received successfully?
EDITED
For starters, you could both read from the file and write to the socket in chunks at the same time.
Since, you are transferring data over TCP, remember that data is transferred reliably as a stream and not as messages. So, don't make assumptions about how the data is recv'd except for the order.
Here is how it could be written:
open socket
open file
size_of_file = read_file_size(file);
send(socket, &size_of_file, sizeof(int), ...)
while (all is written)
read fixed chunk from file
write as much was read to the socket
cleanup // close file, socket
As for the recv part, I think it is best you send the file size over as an integer and keep reading in a while loop until you have recv'd as many bytes as you are sending from the server.
It's like this:
recv(socket, &size_of_msg, sizeof(int), ...)
while(all is read)
read fixed chunk from the socket
cleanup
Well I see atleast some issue with the way you are sending message over socket.
First from the man page of fread:
The function fread() reads nmemb elements of data, each size bytes
long, from the stream pointed to by stream, storing them at the loca-
tion given by ptr.
and what you are trying is this:
fread(str, size, 1,fp);
I assume what you meant was
fread(str, 1,size,fp);
Though it shold not casue the issue.
But the problem lies here:
n = send(nsd, str, strlen(str), 0);
printf("%d bytes sent successfully\n",n);
if(n == 0) break;
sentbytes = sentbytes + n;
size = size - sentbytes;
Though you are decreasing 'size' by decreasing by number of bytes successfully send, where are you extending str to point to new buffer location where data will be send.This will only resend initial bytes of the buffer repeatedly.
str += n; //Assuming str is char*
will solve your issue.
Using strlen doesn't seem appropriate. You've read the file, you know how long it is, so why do strlen? Either you'll just get the same result (so it's redundant) or you'll get something else (so it's a bug).
"Shall i place the recv() part in a while(1) loop so that the client keeps running until all the bytes have been received successfully?"
Something like that. Never presume that a recv() call got everything that was sent -- tcp/ip breaks messages into packets at a lower level, and recv() will return after reading whatever amount of data has actually been received at whatever point. You don't have to worry about that directly, except in so far as you do need to use some kind of protocol to indicate how long a message is so the receiver knows how much to read, then eg.:
char buffer[4096];
int msgsz = 600, // see below
sofar = 0,
cur;
while (sofar < msgsz) {
cur = recv (
socket_fd,
&buffer[sofar],
msgsz - sofar,
0
);
if (cur == -1) {
// error
break;
} else if (cur == 0) {
// disconnected
break;
}
sofar += cur;
}
WRT msgsz, you would include this somewhere in a fixed length header, which is read first. A simple version of that might be just 4 bytes containing a uint32_t, ie, an int with the length. You could also use a null terminated string with a number in it, but that means reading until '\0' is found.