I am trying to simulate client/server communication to send and receive files using sockets and http requests.
The client sends an http request to the server saying that they want to download a simple .txt file and the server sends the file data back to the client with an http response header.
However, the data written into the received text file also contains the http header information which it shouldn't.
How do I effectively separate the http header from the actual data?
Here's what the code looks like:
Server:
//receives an http header request from client saying they want to download a .txt file
bzero(buffer, 256);
sprintf(buffer, "HTTP/1.1 200 OK\r\n Date:%s\r\n Content-Type:%s\r\n Content-Length:%s\r\n\r\n", time, content, length);
data = write(incoming_socket, buffer, strlen(buffer)); // first sends an http response header
bzero(buffer, 256);
int data;
while((data = fread(buffer, sizeof(char), 256, fs)) > 0){
if(send(cli_socket, buffer, strlen(buffer), 0) < 0){ // sends the actual file data to client in this while cycle
perror("ERROR: Could not send file.");
break;
exit(EXIT_FAILURE);
}
if (block_size < 256){
if (ferror(fs)){
perror("ERROR: Failed while sending data.");
exit(EXIT_FAILURE);
}
bzero(buffer, 256);
break;
}
bzero(buffer, 256);
}
Client:
bzero(buffer, 256);
data = read(client_socket, buffer, 256); // first receive the http header response
bzero(buffer, 256);
while (1){
data = recv(client_socket, buffer, 256, 0); // receive data
fwrite(buffer, sizeof(char), data, fr); // write data into the file
bzero(buffer, 256);
if (data == 0 || data < 256){
fclose(fr);
break;
}
if(data< 0){
printf("failed while copyng file!\n");
exit(1);
}
}
However, this fills the .txt file with the http response as well, not just the data.
Now, I know that I should probably look for /r/n/r/n in the header response in order to separate the header and write just the actual data but, if someone could show me the best way to do this in my example with my particular buffer, I'd be very grateful!
Unfortunately, you've chosen a very compelex protocol to implement. Your only choice is to read the documentation for HTTP version 1.1 and follow the specification precisely. The documentation will tell you how to identify the end of the header. Note that you must support chunked encoding. It is required for HTTP 1.1 compliance.
But you've really chosen an H-bomb to kill an ant here. HTTP 1.1 is a complex, high-performance protocol and a terrible choice for just messing around transferring a file.
A few more mistakes:
if(send(cli_socket, buffer, strlen(buffer), 0) < 0){ // sends the actual file data to client in this while cycle
The strlen function is only for C-style strings, not arbitrary data. The recv function returns the number of bytes read for a reason.
data = recv(client_socket, buffer, 256, 0); // receive data
fwrite(buffer, sizeof(char), file_block_size, fr); // write data into the file
Surely the number of bytes you write should be the number of bytes you read (which you helpfully stored in a variable, data, but didn't use), not some other number.
If you really want to make HTTP 1.1 client and server code, you really should start by thoroughly understanding both the HTTP 1.1 specification and example client and server code. But that's a terrible way to learn how to send and receive files because HTTP 1.1 is so complicated.
Related
I'm making a TCP socket programming.
client:
a client will receive a list of files from a server --> send a file name to the server --> receive the file from the server
My problems:
the server sends the list of files correctly. I printed out all and all of them are sent well. however, the client receives it well but while the loop isn't finished even if the server sending while loop is finished.
[client]
while ((read_cnt=read(sd, buf, BUF_SIZE)) != 0) {
read_cnt=read(sd, buf, BUF_SIZE);
printf("%s\n", buf);
}
[server]
while ((entry=readdir(dir)) != NULL) {
sprintf(buf, "%s\n", entry->d_name);
write(clnt_sd, buf, strlen(buf));
}
the server sends 17 messages but the client receives 15 messages and while loop is not finished. when while loop receives "finish", make client finishes the while loop --> still it receives 15 messages even if the server sent 17 messages.
[client]
while (1) {
read_cnt=read(sd, buf, BUF_SIZE);
fwrite((void*)buf, 1, read_cnt, fp);
printf("%s\n", buf);
if (strstr(buf, "fin") != NULL) {
break;
}
total_read += read_cnt;
pcnt = ((total_read/ file_size) * 100.0)/10;
printf("(%.0d %%) Send %.0f / %.0f bytes.\n", pcnt*10, total_read, file_size);
}
[server]
FILE *fp;
fp = fopen(file_name, "r");
if (fp == NULL) {
printf("File not Exist");
exit(1);
}
fseek(fp, 0, SEEK_END);
int file_size = ftell(fp);
sprintf(buf, "%d", file_size);
write(clnt_sd, buf, read_cnt);
fseek(fp, 0, SEEK_SET);
while (feof(fp) == 0) {
read_cnt = fread((void*)buf, 1, BUF_SIZE, fp);
write(clnt_sd, buf, read_cnt);
}
fclose(fp);
I tried memset, bzero to make buffer empty but it didn't work.
I think the problem is the client part because when I checked server side by using print, they are sending well and finished the while loops. But I don't know what's the problem in receiving process on the client side. Please, Let me know what's the problem is.
the server sends 17 messages but the client receives 15 messages
Completely normal.
TCP is a stream protocol, not a message passing protocol. What you "send" on one side may not be the exact number of bytes received at the other end. You have to expect messages to arrive chunked, coalesced, or segmented between and within read/recv calls. You can't even assume that the initial "size" write you send as a 4-byte message is received within the same read call either as an entire message.
Implement your client side code to expect read and recv to return a random number of bytes up to the size of the buffer you pass in. If you get less bytes than expected, you need to make another read/recv call to receive the rest - adjusting your destination buffer pointer as needed. Or use the MSG_WAITALL flag with recv (instead of read) to get all the data in one call. You still need to check the result of the buffer to make sure you got the expected number of bytes, that the stream didn't close, and to validate that an error (-1 return) didn't occur.
EDIT: Changed the while loop condition from > 0 to !=0. There are also 2 or more newlines being output on the terminal after printing the content, but in the file there aren't.
I'm doing a file transfer from the client to the server. The contents are transferred successfully, but when receiving it doesn't exit the while loop. Is my while condition somehow wrong? After I force stop the program, i can see that the file has actually been copied successfully, it just does not exit this while loop after writing everything to the file.
This is the code where I'm having trouble:
ssize_t bytes_read = 0;
int error;
while((bytes_read = read(sd, buf, sizeof(buf))) != 0){
printf("writing: %s\n", buf);
if((error = fwrite(buf, 1, bytes_read, fp)) < 0)
{
printf("error");
}
if((bytes_read = read(sd, buf, sizeof(buf))) < 0){
break;
}
}
Tried to keep it as minimal as possible, as I'm sure there's something very simplistically wrong here.
EDIT: I tried it with a larger file, and it seems that its missing some content here and there but not in a consistent pattern.
EDIT: Here is the client side.
else if(strcmp(shortCommand, "put") == 0){
char *tmp = buf + 4;
char filename[MAX_BLOCK_SIZE];
strcpy(filename, "filename ");
strcat(filename, tmp);
FILE *fp;
printf("File name: %s\n", tmp);
fp = fopen(tmp, "rb");
if(fp == NULL){
printf("ERROR: Requested file does not exist.\n");
}
else{
printf("Client sending filename...\n");
if ((nw = write(sd, buf, sizeof(buf))) < 0){ //sending the file name to the client first
printf("Error sending client's filename.\n");
}
printf("Client sending file...\n");
size_t bytes_read = 0;
ssize_t bytes_written = 0;
while((bytes_read = fread(buf, 1, sizeof(buf), fp)) != 0){ //sending the file contents
if ((bytes_written = write(sd, buf, bytes_read)) < 0){
printf("Error sending client file.\n");
}
}
fclose(fp);
}
}
The server code assumes that it will observe read() to return 0 after it reads all the data that the client transfers, but that will not happen just because the client stops sending data. read() returns 0 to indicate that the end of the file has been reached, but "file" in that sense means the read side of its socket. Until the client closes the peer socket, the server will not perceive EOF on its socket. And that's perfectly reasonable because until then, the client might send more data, proving that indeed EOF had not previously been reached. Therefore, if the server tries to perform a (blocking) read on the socket after receiving all the data then it will block until the client sends more or closes the connection, or until it detects an error, is interrupted by a signal, etc..
If the client doesn't actually intend to send any further data after the file content then it should close() that socket after sending all the bytes of the file content, or at least shutdown() the write side.
If, on the other hand, the client is keeping the socket open because it intends to send more data later, then you need a new strategy. In that case the end of the file content does not coincide with the logical end of the stream, and you therefore need a different mechanism to communicate to the server where the end of the file data is. This is where communication protocols such as HTTP come in, but for your purpose, you don't need a protocol nearly as complicated as HTTP.
It might suffice, say, for the client to send data to the server as one or more chunks, each comprising a byte count followed by the specified number of bytes of content. You could insist that the whole file content always go into one chunk, but if you want to allow splitting the file into multiple chunks then you could use a chunk length of zero to signal the end of the data.
I have some problems when trying to receive http response message of a website.
This is my function:
void Reveive_response(char *resp, SSL *ssl) {
const int BUFFER_SIZE = 1024;
char response[1048576];
char *buffer = NULL; // to read from ssl
char *check = (char *) malloc(BUFFER_SIZE*sizeof(char));
int bytes; // number of bytes actually read
int received = 0; // number of bytes received
buffer = (char *) malloc(BUFFER_SIZE*sizeof(char)); // malloc
memset(response, '\0', sizeof(response)); // response
assign = '\0'
do{
memset(buffer, '\0', BUFFER_SIZE); // empty buffer
bytes = SSL_read(ssl, buffer, BUFFER_SIZE);
if (bytes < 0) {
printf("Error: Receive response\n");
exit(0);
}
if (bytes == 0) break;
received += bytes;
printf("Received...%d bytes\n", received);
strncat(response, buffer, bytes); // concat buffer to response
} while (SSL_pending(ssl)); // while pending
response[received] = '\0';
printf("Receive DONE\n");
printf("Response: \n%s\n", response);
free(buffer);
strcpy(resp, response); // return via resp
}
When I call the function, it seems like the response message is not complete. Like this:
Received...1014 bytes
Received...1071 bytes
Receive DONE
Response:
HTTP/1.1 200 OK
<... something else....>
Vary: Accept-Encoding
Content-Type: text/html
Conne
Then if i call the function again, it returns:
Received...39 bytes
Receive DONE
Response:
ction: keep-alive
Content-Length: 0
The field Connection was split. Why my function didn't receive all the response message? I used do while loop inside. Please tell me where did i go wrong? Thank you.
There is nothing wrong. This is simply how TCP works. It is a streaming transport, it has no concept of message boundaries. There is no 1-to-1 relationship between the number of bytes sent and the number of bytes read. Your reading receives arbitrary bytes, which you are then responsible for processing as needed. Keep reading, buffering and parsing the HTTP data as you go, until you discover the end of the response (see RFC 2616 Section 4.4 Message Length for details). Looping on SSL_pending() is not sufficient (or correct).
In this case, you have to read CRLF-delimited lines one at a time until you reach a CRLF/CRLF pair indicating the end of the response headers, then you need to analyze the headers you have received to know whether a response body is present and how to read it, as it may be in one of several different encoded formats. If present, you can then read the body (decoding it as you go along) until you reach the end of the body as specified by the headers.
See the pseudo-code I posted in my answer to the following question:
Receiving Chunked HTTP Data With Winsock
That said, you really should not be implementing HTTP (let alone HTTPS) manually to begin with. HTTP is not trivial to implement from scratch, and neither is SSL/TLS for that matter. You have dived head-first into a deep well without understand some important basics of network programming and OpenSSL programming. You should use an existing HTTP/S library instead, such as libcurl, and let it handle the details for you so you can focus on your code's business logic and not its communications logic.
To receive from the requested web server and transmit it to the client, I am doing the following,
while(1) {
bzero(buffer,65536); //Character buffer of 64KB
ret_val = recv(sockfd, buffer, 65535,0); //sockfd is the socket between web server and proxy server
if(ret_val < 0)
error("Error Reading data from requested server");
send_ret_val = send_all(sock, buffer, strlen(buffer), 0);//sockfd is socket between proxy server and client
if(send_ret_val < 0)
error("Error returning data to client");
if(ret_val == 0)
break;
}
The function send() all transmits all the data there in the buffer and returns 0 else returns a negative value for an error.
The problem is that the server seems to be working fine for text data but cannot handle images and other binary data. When using firefox, I get the error, incompatible compression technique.
Is there a problem in this code or is there a problem somewhere else?
strlen(buffer) truncates when it found null character in the buffer.
Image data is binary data. Binary data may contain null characters in the middle of the image.
You must use the number of bytes received from the recv call to send bytes to client.
Modify following statement
send_ret_val = send_all(sock, buffer, strlen(buffer), 0);
to
send_ret_val = send_all(sock, buffer, ret_val, 0);
Handling images in a proxy server written in C
... is no different from handling any other type of data. If it doesn't work for images, it will break for other kinds of data as well.
bzero(buffer,65536); //Character buffer of 64KB
Cargo-cult programming. Remove it.
ret_val = recv(sockfd, buffer, 65535,0); //sockfd is the socket between web server and proxy server
There's no reason for the length supplied to be different from sizeof buffer here.
if(ret_val < 0)
error("Error Reading data from requested server");
This is only OK if error() prints or accesses errno prior to executing any other system calls, and if it magically causes this loop to exit. After this you need to add:
else if (retval == 0)
break; // end of stream
Then:
send_ret_val = send_all(sock, buffer, strlen(buffer), 0);//sockfd is socket between proxy server and client
This assumes that the data received is null-terminated, which isn't even valid in the case of a text message. It is completely wrong in the case of an image. Change to:
send_ret_val = send_all(sock, buffer, retval , 0);//sockfd is socket between proxy server and client
if(send_ret_val < 0)
error("Error returning data to client");
Again this is only OK if error() prints or accesses errno prior to executing any other system calls, and if it magically causes this loop to exit.
if(ret_val == 0)
break;
You have this in the wrong place.
The function send() all transmits all the data there in the buffer and returns 0 else returns a negative value for an error.
No. It returns -1 or the number of bytes transferred. The only way it can return zero is if you supplied a zero length, which would be completely pointless.
I try to download a tar file from a remote webserver, but I get only false data. Every runtime an other bytesize and with a HTTP header:
//now it is time to receive the page
memset(buf, 0, sizeof(buf));
FILE *fp;
if ((fp = fopen(FILENAME, "wb"))==NULL) {
perror("Error open file");
}
else {
while((tmpres = recv(xmlsock, buf, 65536, 0)) > 0){
fwrite(buf, sizeof(buf), 1, fp);
memset(buf, 0, tmpres);
}
}
if(tmpres < 0) {
perror("Error receiving data");
}
fclose(fp);
close(xmlsock);
My sent header is:
GET /updates/xxxxx.tar HTTP/1.1
Host: www.yyyyyyyy.de
User-Agent: HTMLGET 1.1
Connection: close
Can anybody help me?
Try:
fwrite(buf, 1, tmpres, fp);
You're always writing sizeof(buf) bytes in your original code, but you only read tmpres bytes, so you're writing a lot of cruft every time (there is no guarantee how big the packets are that you receive and in how many recv calls your data will be split up)
Two problems: The first is that you write the complete buffer to the file even if not all of it was filled by the recv Call. The other problem is that you receive everything, without regards to the HyperText Transfer Protocol which naturally includes headers.
For the first problem the solution is very simple, as you already have a variable whose value is the amount of data received.
For the second problem, you need to learn HTTP and how it separates the headers and the data.