I try to download a tar file from a remote webserver, but I get only false data. Every runtime an other bytesize and with a HTTP header:
//now it is time to receive the page
memset(buf, 0, sizeof(buf));
FILE *fp;
if ((fp = fopen(FILENAME, "wb"))==NULL) {
perror("Error open file");
}
else {
while((tmpres = recv(xmlsock, buf, 65536, 0)) > 0){
fwrite(buf, sizeof(buf), 1, fp);
memset(buf, 0, tmpres);
}
}
if(tmpres < 0) {
perror("Error receiving data");
}
fclose(fp);
close(xmlsock);
My sent header is:
GET /updates/xxxxx.tar HTTP/1.1
Host: www.yyyyyyyy.de
User-Agent: HTMLGET 1.1
Connection: close
Can anybody help me?
Try:
fwrite(buf, 1, tmpres, fp);
You're always writing sizeof(buf) bytes in your original code, but you only read tmpres bytes, so you're writing a lot of cruft every time (there is no guarantee how big the packets are that you receive and in how many recv calls your data will be split up)
Two problems: The first is that you write the complete buffer to the file even if not all of it was filled by the recv Call. The other problem is that you receive everything, without regards to the HyperText Transfer Protocol which naturally includes headers.
For the first problem the solution is very simple, as you already have a variable whose value is the amount of data received.
For the second problem, you need to learn HTTP and how it separates the headers and the data.
Related
I'm making a TCP socket programming.
client:
a client will receive a list of files from a server --> send a file name to the server --> receive the file from the server
My problems:
the server sends the list of files correctly. I printed out all and all of them are sent well. however, the client receives it well but while the loop isn't finished even if the server sending while loop is finished.
[client]
while ((read_cnt=read(sd, buf, BUF_SIZE)) != 0) {
read_cnt=read(sd, buf, BUF_SIZE);
printf("%s\n", buf);
}
[server]
while ((entry=readdir(dir)) != NULL) {
sprintf(buf, "%s\n", entry->d_name);
write(clnt_sd, buf, strlen(buf));
}
the server sends 17 messages but the client receives 15 messages and while loop is not finished. when while loop receives "finish", make client finishes the while loop --> still it receives 15 messages even if the server sent 17 messages.
[client]
while (1) {
read_cnt=read(sd, buf, BUF_SIZE);
fwrite((void*)buf, 1, read_cnt, fp);
printf("%s\n", buf);
if (strstr(buf, "fin") != NULL) {
break;
}
total_read += read_cnt;
pcnt = ((total_read/ file_size) * 100.0)/10;
printf("(%.0d %%) Send %.0f / %.0f bytes.\n", pcnt*10, total_read, file_size);
}
[server]
FILE *fp;
fp = fopen(file_name, "r");
if (fp == NULL) {
printf("File not Exist");
exit(1);
}
fseek(fp, 0, SEEK_END);
int file_size = ftell(fp);
sprintf(buf, "%d", file_size);
write(clnt_sd, buf, read_cnt);
fseek(fp, 0, SEEK_SET);
while (feof(fp) == 0) {
read_cnt = fread((void*)buf, 1, BUF_SIZE, fp);
write(clnt_sd, buf, read_cnt);
}
fclose(fp);
I tried memset, bzero to make buffer empty but it didn't work.
I think the problem is the client part because when I checked server side by using print, they are sending well and finished the while loops. But I don't know what's the problem in receiving process on the client side. Please, Let me know what's the problem is.
the server sends 17 messages but the client receives 15 messages
Completely normal.
TCP is a stream protocol, not a message passing protocol. What you "send" on one side may not be the exact number of bytes received at the other end. You have to expect messages to arrive chunked, coalesced, or segmented between and within read/recv calls. You can't even assume that the initial "size" write you send as a 4-byte message is received within the same read call either as an entire message.
Implement your client side code to expect read and recv to return a random number of bytes up to the size of the buffer you pass in. If you get less bytes than expected, you need to make another read/recv call to receive the rest - adjusting your destination buffer pointer as needed. Or use the MSG_WAITALL flag with recv (instead of read) to get all the data in one call. You still need to check the result of the buffer to make sure you got the expected number of bytes, that the stream didn't close, and to validate that an error (-1 return) didn't occur.
EDIT: Changed the while loop condition from > 0 to !=0. There are also 2 or more newlines being output on the terminal after printing the content, but in the file there aren't.
I'm doing a file transfer from the client to the server. The contents are transferred successfully, but when receiving it doesn't exit the while loop. Is my while condition somehow wrong? After I force stop the program, i can see that the file has actually been copied successfully, it just does not exit this while loop after writing everything to the file.
This is the code where I'm having trouble:
ssize_t bytes_read = 0;
int error;
while((bytes_read = read(sd, buf, sizeof(buf))) != 0){
printf("writing: %s\n", buf);
if((error = fwrite(buf, 1, bytes_read, fp)) < 0)
{
printf("error");
}
if((bytes_read = read(sd, buf, sizeof(buf))) < 0){
break;
}
}
Tried to keep it as minimal as possible, as I'm sure there's something very simplistically wrong here.
EDIT: I tried it with a larger file, and it seems that its missing some content here and there but not in a consistent pattern.
EDIT: Here is the client side.
else if(strcmp(shortCommand, "put") == 0){
char *tmp = buf + 4;
char filename[MAX_BLOCK_SIZE];
strcpy(filename, "filename ");
strcat(filename, tmp);
FILE *fp;
printf("File name: %s\n", tmp);
fp = fopen(tmp, "rb");
if(fp == NULL){
printf("ERROR: Requested file does not exist.\n");
}
else{
printf("Client sending filename...\n");
if ((nw = write(sd, buf, sizeof(buf))) < 0){ //sending the file name to the client first
printf("Error sending client's filename.\n");
}
printf("Client sending file...\n");
size_t bytes_read = 0;
ssize_t bytes_written = 0;
while((bytes_read = fread(buf, 1, sizeof(buf), fp)) != 0){ //sending the file contents
if ((bytes_written = write(sd, buf, bytes_read)) < 0){
printf("Error sending client file.\n");
}
}
fclose(fp);
}
}
The server code assumes that it will observe read() to return 0 after it reads all the data that the client transfers, but that will not happen just because the client stops sending data. read() returns 0 to indicate that the end of the file has been reached, but "file" in that sense means the read side of its socket. Until the client closes the peer socket, the server will not perceive EOF on its socket. And that's perfectly reasonable because until then, the client might send more data, proving that indeed EOF had not previously been reached. Therefore, if the server tries to perform a (blocking) read on the socket after receiving all the data then it will block until the client sends more or closes the connection, or until it detects an error, is interrupted by a signal, etc..
If the client doesn't actually intend to send any further data after the file content then it should close() that socket after sending all the bytes of the file content, or at least shutdown() the write side.
If, on the other hand, the client is keeping the socket open because it intends to send more data later, then you need a new strategy. In that case the end of the file content does not coincide with the logical end of the stream, and you therefore need a different mechanism to communicate to the server where the end of the file data is. This is where communication protocols such as HTTP come in, but for your purpose, you don't need a protocol nearly as complicated as HTTP.
It might suffice, say, for the client to send data to the server as one or more chunks, each comprising a byte count followed by the specified number of bytes of content. You could insist that the whole file content always go into one chunk, but if you want to allow splitting the file into multiple chunks then you could use a chunk length of zero to signal the end of the data.
I am trying to simulate client/server communication to send and receive files using sockets and http requests.
The client sends an http request to the server saying that they want to download a simple .txt file and the server sends the file data back to the client with an http response header.
However, the data written into the received text file also contains the http header information which it shouldn't.
How do I effectively separate the http header from the actual data?
Here's what the code looks like:
Server:
//receives an http header request from client saying they want to download a .txt file
bzero(buffer, 256);
sprintf(buffer, "HTTP/1.1 200 OK\r\n Date:%s\r\n Content-Type:%s\r\n Content-Length:%s\r\n\r\n", time, content, length);
data = write(incoming_socket, buffer, strlen(buffer)); // first sends an http response header
bzero(buffer, 256);
int data;
while((data = fread(buffer, sizeof(char), 256, fs)) > 0){
if(send(cli_socket, buffer, strlen(buffer), 0) < 0){ // sends the actual file data to client in this while cycle
perror("ERROR: Could not send file.");
break;
exit(EXIT_FAILURE);
}
if (block_size < 256){
if (ferror(fs)){
perror("ERROR: Failed while sending data.");
exit(EXIT_FAILURE);
}
bzero(buffer, 256);
break;
}
bzero(buffer, 256);
}
Client:
bzero(buffer, 256);
data = read(client_socket, buffer, 256); // first receive the http header response
bzero(buffer, 256);
while (1){
data = recv(client_socket, buffer, 256, 0); // receive data
fwrite(buffer, sizeof(char), data, fr); // write data into the file
bzero(buffer, 256);
if (data == 0 || data < 256){
fclose(fr);
break;
}
if(data< 0){
printf("failed while copyng file!\n");
exit(1);
}
}
However, this fills the .txt file with the http response as well, not just the data.
Now, I know that I should probably look for /r/n/r/n in the header response in order to separate the header and write just the actual data but, if someone could show me the best way to do this in my example with my particular buffer, I'd be very grateful!
Unfortunately, you've chosen a very compelex protocol to implement. Your only choice is to read the documentation for HTTP version 1.1 and follow the specification precisely. The documentation will tell you how to identify the end of the header. Note that you must support chunked encoding. It is required for HTTP 1.1 compliance.
But you've really chosen an H-bomb to kill an ant here. HTTP 1.1 is a complex, high-performance protocol and a terrible choice for just messing around transferring a file.
A few more mistakes:
if(send(cli_socket, buffer, strlen(buffer), 0) < 0){ // sends the actual file data to client in this while cycle
The strlen function is only for C-style strings, not arbitrary data. The recv function returns the number of bytes read for a reason.
data = recv(client_socket, buffer, 256, 0); // receive data
fwrite(buffer, sizeof(char), file_block_size, fr); // write data into the file
Surely the number of bytes you write should be the number of bytes you read (which you helpfully stored in a variable, data, but didn't use), not some other number.
If you really want to make HTTP 1.1 client and server code, you really should start by thoroughly understanding both the HTTP 1.1 specification and example client and server code. But that's a terrible way to learn how to send and receive files because HTTP 1.1 is so complicated.
I'm writing a toy client/server app and sending data over a UDP stream. Everything is working well when I send small messages like "test" or something like that. But when I go to transfer files, I seem to be transferring too much data.
The file that I am transferring is a little less than 4k in size (at least according to an ls command). But when my sendto operation completes, my confirmation message shows that I have sent exactly 5000 bytes (which is the max size of my buffer).
I'd rather not break the sendto up if I don't need to since UDP can potentially transfer messages out of order. Anyone see what I am doing wrong? I think that is has to do with my use of sizeof(buffer), but I don't know how to transfer exactly the number of bytes that I want. Thank you!
Here is the relevant part of the code. Please let me know if you need more:
char buffer[5000];
char path[] = "myfile";
FILE* fp;
if ((fp = fopen(path, "rb")) == NULL) {
printf(stdout, "Error opening the file: %s\n", strerror(errno));
exit(1);
}
memset(&buffer[0], 0, sizeof(buffer));
do {
fread(buffer, 1, MAXBUFSIZE, fp);
} while (!feof(fp));
if ( (numbytes = sendto(sockfd, buffer, sizeof(buffer), 0,
(struct sockaddr *)&server_addr, sizeof(server_addr))) < 0)
{
perror("send error");
}
fprintf(stderr, "client sent %d bytes to server\n", numbytes);
As per the sendto command, UDP will send the same amount of data as specified in the send length; the third param is the send length: http://linux.die.net/man/2/sendto . The example is specifying sizeof(buffer) as the send length and sizeof(buffer) is 5000. Accordingly, UDP sends 5000 bytes from the buffer (which is actually the entire buffer). If we want to send the exact number of bytes as present in the file, then we should read the number of bytes in the file, and pass that as the send length.
numbytes = sendto(sockfd, buffer, sizeof(buffer), 0,
I'm fairly new to socket programming in C, so the code below may have a ton of newbie mistakes.
I'm trying to make a client-server application in which the server will transfer a file to the client using an UDP socket. Both the client and the server will run on Linux hosts. It's an assignment, so it has to be done that way. Other client-server communications may use a TCP socket, but the file transfer MUST be via UDP. The program works correctly for small files, but if I try to send a slightly larger file (say, a 600 kb text file), the client will stop receiving the packets, even though the server will send them all. Here's the file transfer part of the server code:
FILE* myFile;
long fileSize, readBytes, sentBytes, sizeCheck;
uint32_t encodedFileSize;
myFile = fopen(fileName, "rb");
if(myFile == NULL)
{
perror("Error when opening file.");
exit(1);
}
fseek(myFile, 0, SEEK_END);
fileSize = ftell(myFile);
encodedFileSize = htonl(fileSize);
rewind(myFile);
sizeCheck = 0;
write(myTCPSocket, &encodedFileSize, sizeof(encodedFileSize));
if(fileSize > 255)
{
while(sizeCheck < fileSize)
{
readBytes = fread(bufferRW, 1, 256, myFile);
sentBytes = sendto(sockfdUDP, bufferRW, readBytes, 0, (struct sockaddr*)&cli_addr, udpAddressSize);
sizeCheck += sentBytes;
}
}
else
{
readBytes = fread(bufferRW, 1, 256, myFile);
sentBytes = sendto(sockfdUDP, bufferRW, readBytes, 0, (struct sockaddr*)&cli_addr, udpAddressSize);
}
if(fileSize == sizeCheck)
{
printf("Success.\n");
}
else
{
printf("Fail.\n");
}
fclose(myFile);
fflush(stdout);
close(sockfdUDP);
As you can see, I used a TCP socket to send the client the file size. Here's the client code:
FILE *myFile;
long receivedBytes, writtenBytes, sizeCheck;
long fileSize, realFileSize;
char ack2[5] = "Ok";
sockfdUDP = socket(AF_INET, SOCK_DGRAM, 0);
read(socketTCP, &fileSize, sizeof(long));
realFileSize = ntohl(fileSize);
myFile = fopen(fileName, "wb");
if(myFile == NULL)
{
perror("Error when creating file.");
exit(1);
}
sizeCheck = 0;
if((realFileSize) > 255)
{
while(sizeCheck < (realFileSize))
{
receivedBytes = recvfrom(sockfdUDP, bufferRW, 256, 0, (struct sockaddr*)&serv_addr, &serv_addr_size);
writtenBytes = fwrite(bufferRW, 1, receivedBytes, myFile);
fflush(myFile);
sizeCheck += writtenBytes;
}
}
else
{
receivedBytes = recvfrom(sockfdUDP, bufferRW, 256, 0, (struct sockaddr*)&serv_addr, &serv_addr_size);
fwrite(bufferRW, 1, receivedBytes, myFile);
fflush(myFile);
}
if(realFileSize == sizeCheck)
{
printf("Success.");
}
else
{
printf("Fail.");
}
fclose(myFile);
close(sockfdUDP);
The "bufferRW" buffer was originally declared as char bufferRW[256] and passed to the function as an argument. Same goes for other undeclared variables.
Like I said before, the server will (apparently) send the whole file without any issues. However, the client will stop receiving packets after it's written about 423936 bytes (this may vary between executions). It'll just stay at the recvfrom line, without reading anything.
Now, I'm sure the problem is not being caused by a faulty connection since I'm testing both processes on the same host. And before you ask "What is it with the 256 byte packet size?", there's this weird bug that will throw me a segmentation fault on the realFileSize = ntohl(fileSize); client line if I use a buffer size of, say, 1500.
Could you please tell me what am I missing here?
EDIT: I'm trying with different file sizes now. It seems to handle files larger than 256 bytes without issue (it enters and exits the while loops correctly on both client and server), but the client will start having problems when the file is bigger than, say, 300 kb.
EDIT 2: I just debugged the program. Apparently, the server sends the entire file before the client can even enter its while loop.
EDIT 3: I think I know what's causing the issue. It seems like if the server sends a bunch of packets before the client starts reading, the client will read up to 278 packets, regardless of their size. If I try sending, say, 279 before the client starts reading it won't read the 279th packet. So if the server sends its packets fast enough, the number of packets that the client hasn't read yet will exceed 278 and the client won't finish reading all of the packets. Any ideas on how to fix this?
long* fileSize declared a pointer to a long, but in your code, it's pointing nowhere. In fact, it's pointing to a random address. You should declare it as long fileSize, and call read(socketTCP, &fileSize, sizeof(long)) instead.
You should check the return value of read, write, etc, to guarantee they did not fail. For example, sendto returns -1 on error. You're ignoring this, and incrementing sizeCheck with this value anyway.
UDP is not a reliable protocol for file transfers, but if you cannot do without it, you better implement some controls that TCP already gives you for free, like packet reordering, data checksum, etc. And that can be a very complex task by itself.
Compile your code with -Wall -Wextra. The compiler will give you hints about what could be potentially wrong. I see you're still using *fileSize in a comparison, which is clearly wrong.
After you fix the *fileSize issue, your loop-condition is still using the wrong value (due to fileSize = ntohl(fileSize)). You need to store this value in another variable, or change your loop-condition to use the real file size.
Regarding your EDIT 3, you need to somehow synchronise your client & server, so they can start the transmission at the same time. However, a sender that is much faster than the receiver will still cause packet loss. To solve this, you'll also need to implement packet acknowledgement, and retransmit a packet if the sender doesn't receive an ACK for a respective sent packet after a timeout. This is something TCP already does for you.
A simpler (but not completely reliable) way, would be to slow down the sending process a bit - maybe using nanosleep between each call to sendto.