HTTP response message generated by server is not being recognized as a response - c

I'm programming a simple web server in C. The HTTP message the server generates is stored in a buffer and sent (via send()) as follows:
Scenario 1:
"200 OK\nContent-Type: text/html\nContent-Length: " + resource size in bytes + "\r\n\n"
where the resource size in bytes is converted to a char array using snprintf and then concatenated into the string.
Scenario 2:
"HTTP1.1 404 Not Found\r\nContent-Length: 0\r\n\n"
Scenario 3:
"HTTP1.1 405 Method Not Allowed\r\nAllow: GET, HEAD\r\n"
These are the headers, they are sent beforehand. The message body is sent afterwards as follows:
char resource[length];
int numRead;
while ( (numRead= read(filefd, resource, length)) > 0 )
send(client, resource, length, 0);
When I use wireshark, it doesn't recognize it as an HTTP response. When I use firefox, the web page continues loading until I shut down the server, at which point it displays the HTTP response instead of a webpage (index.html):
image
Do I have to encode the message before sending? Or is there something wrong with my message format?

There are many errors here which suggest that you just made up your own version of HTTP instead of reading the standard. Please refer to the standard for all the details if you want to implement HTTP.
In short:
"200 OK\nContent-Type: text/html\nContent-Length: " + resource size in bytes + "\r\n\n"
It should be HTTP/1.1 200 OK... and not just 200 OK.... All line ends must be \r\n and not \n and there must be a single \r\n at the end of the HTTP header.
"HTTP1.1 404 Not Found\r\nContent-Length: 0\r\n\n"
"HTTP1.1 405 Method Not Allowed\r\nAllow: GET, HEAD\r\n"
It should be HTTP/1.1 not HTTP1.1. And then all the other problems.
Again, if you really want to implement your own HTTP stack then study the standard. It is far more complex than you might think. Just because HTTP is a text based protocol does not mean that it is simple nor that any text is a correct HTTP message. Additionally implementations are tolerant in different ways, so just because it works with one client (i.e. browser) does not mean it is correct and that it will work with a different client too.

Related

Raw http server: send image issue

I'am working with kind of IoT device. Finaly I've got simple httpd server to work, and simple html pages works like a charm, but browser does not recognise images. I think this is http header issue, but i do not know what is exacly wrong.
For example, my test page look like this:
<html>
<head><title>test page</title></head>
<body>
hello world!
<img src="img.png">
</body>
</html>
If i go to http://de.vi.ce.ip/ 2 reqests are generated:
GET / HTTP/1.1\r\n
Accept text/html, application/xhtml+xml, */*\r\n
Accept-Language: en-EN\r\n
...
GET /img.png HTTP/1.1\r\n
Accept image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5\r\n
Accept-Language: en-EN\r\n
...
To witch my server responds with:
HTTP/1.0 200 OK\r\n
Content-Type: text/html\r\n
Content-Length: 131\r\n
\r\n
<page data>
HTTP/1.0 200 OK\r\n
Content-Type: image/png\r\n
Content-Length: 5627\r\n
\r\n
<image binary data>
As the result i can see the text, but images are broken.
I've tryed few more parameters like Connection: close, Accept-Ranges: bytes, Content-Location (path).
I've tryed jpeg image under Content-Type: image/jpeg with no luck. I'am certain that image sent correctly.
I've made exactly the same - raw http server for IoT and your response looks absolutely correct. Try check following:
You correctly flush the socket before closing it. If you call close() right after send(), you will likely encounter this problem - data has not been correctly written
The Content-Length should be exactly the size of your file. Make sure you are not counting \r\n bytes of http response. Browser may still wait for tail bytes
Finally, get the browser network logs :)
The request is asking for png
GET /img.png HTTP/1.1\r\n
Why not return the correct content type;
Content-Type: image/png\r\n
I was running into a very similar problem.
In my case when I thought I was using \r\n line terminators, I was actually only using \n; which worked fine in chromium for serving the text/html page, but was throwing net::ERR_INVALID_HTTP_RESPONSE error when serving the image/jpeg. So the page loaded, but the images were broken.
My fix was to make sure that everything was using \r\n as it was supposed to.

mg_send_response_line() in mongoose not working

I am trying to develop a server application using mongoose C library. In the initial stage of the my try, I am stuck at sending a response for an HTTP request. I am trying to send a simple response of status 200 using following line of code:
mg_send_response_line(nc, 200, "Access-Control-Allow-Origin: *");
printf("Response sent...\n");
But the response is not received to the client(Postman or web-browser).
There is no error, and even the printf line of Response sent is printed.
As against to this, following lines are getting executed successfully:
mg_http_send_error(nc,404, "Fatal Error!"); // I get this error at client side.
The basic simplest_web_server also works fine. Why is my single line of code sending response failing. I am not able to understand/ debug this.
Regards,
Neeraj.
The issue is that there is no content length or transfer encoding specified for the HTTP response and the server does not close the connection so the client hangs waiting for the response.
If you look through the source code, you will see that in mg_http_send_error(), the MG_F_SEND_AND_CLOSE flag is set but it is not set within mg_send_response_line() (though, like you, I assumed that this would be handled by the function).
To fix the issue in your context,
mg_send_response_line(nc, 200, "Access-Control-Allow-Origin: *");
printf("Response sent...\n");
nc->flag |= MG_F_SEND_AND_CLOSE;

Thunderbird Lightning caldav sync doesn't show any data/events

when i try to synchronize my caldav server implementation with Thunderbird 45.4.0 and Lightning 4.7.4 (one particular calendar collection) it doesnt show any data or events in the calendar though the last call of the sequence provided the data.
In the Thunderbird error log i can see one error:
Zeitstempel: 07.11.16, 14:21:12
Fehler: [calCachedCalendar] replay action failed: null,
uri=http://127.0.0.1:8003/sap/sports/webdav/appsvc/webdav/services/
server.xsjs/cal/_D043133/, result=2147500037, op=[xpconnect wrapped
calIOperation]
Quelldatei:
file:///Users/d043133/Library/Thunderbird/Profiles/hfbvuk9f.default/
extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-
js/calCachedCalendar.js
Zeile: 327
the call sequence is as follows (detailed content via gist-links):
Propfind Request - Response
Options Request - Response
Propfind Request - Response
Report Request - Response - Response Raw
The synchronization with other clients like macOS-calendar and ios-calendar works in principle and shows the data. Does anyone has a clue what is going wrong here?
Not sure whether that is the cause but I can see two incorrect things:
a) Your <href/> property has trailing spaces:
<d:href>/sap/sports/webdav/appsvc/webdav/services/server.xsjs/cal/_D043133/EVENT%3A070768ba5dd78ff15458f1985cdaabb1.ics
</d:href>
b) your ORGANIZER property is not a valid URI
ORGANIZER:_D043133
i was able to find the cause of the above issue by debugging Thunderbird as propsed by Philipp. The Report Response has http status code 200, but as it is a multistatus response Thunderbird/Lightning expects status code 207 ;-)
Thanks for the hints!

Transfer-Encoding: chunked-- Browser does not respond

I have made a very simple web server on my LINUX machine using TCP socket programming in C language.I am sending it a HTTP GET request from a browser(both chrome and mozilla ) from the local machine.
This problem is that when i do not set the header
Transfer-Encoding: chunked in the response , the browser successfully displays the webpage.
But when i keep this header , the browser does not respond, it says NO DATA IS AVAILABLE.
EDIT: It works for firefox now after i added the chunk size (446 bytes) as pointed by #RomanK.
But chrome becomes unresponsive.
Here is the code
responseIndex = add(response,"HTTP/1.1 200 OK",responseIndex);
responseIndex = add(response,"Transfer-Encoding: chunked",responseIndex);
responseIndex = add(response,"Content-Type: text/html",responseIndex);
response[responseIndex++]='\r';
response[responseIndex++]='\n';
updateIndex = add(response,"446",updateIndex);
responseIndex = add(response,filebuffer,responseIndex);
response[responseIndex++]='\0';
send(clntSock, response, strlen(response), 0) ;
close(clntSock);
exit(0);
Here, add is a function to append the second argument to response and then append "/r/n".
response is a string.
responseIndex is just an int to keep track of the current length of response.
filebuffer is a string which contains all the text of the html file to be sent.
Response :
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/html
446 (or 1EB)
<html>
BODY
</html>
The error code given by chrome is : ERR_INVALID_CHUNKED_ENCODING
Content-Length and chunked transfer encoding are mutually exclusive.
You should omit Content-Length and rather add the chunk size at the start of each chunk as per the Wikipedia article.
Or, in other words, you need to output the chunk size in hexadecimal before this line
responseIndex = add(response,filebuffer,responseIndex);
EDIT : Note that the you need to provide the size of the chunk only, not of the entire HTTP response. In your case it should be the size of the HTML body only; for example it looks like your sample body would be 30 or 31 in size in hex (not sure about the whitespace).
So, 3 points:
a) Use hex
b) Use lowercase
c) Use size of the chunk (in your case, the body, as you have a single chunk). Do not include the size of the HTTP meta-data.
It's also a bit questionable that you use chunks in the first place; they should be used only in cases where you do not know the response size when you start generating the response. Here you know the response size at the start and can use Content-Length without Transfer-Encoding: chunked.
The point of chunked transfer is (sorry for tautology) to send data in chunks. The browser doesn't know how many chunks to expect, so you need to tell it that some chunk is the last one. The protocol specifies that the last chunk should be of size 0:
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/html
446\r\n
Precisely 446 bytes of data
0\r\n

C HTTP Proxy, the browser shows "waiting for" indefinetely

I have a hard time implementing a proxy server in C. It works for a few first webpages but then I get blocked while waiting for a new request.
Design:
Firefox -> Proxy -> Webserver --.
Firefox <- Proxy <- Webserver <-'
So each request is a round-trip from the browser to the proxy and the server and then back. Until the response from the request comes back, nothing should happen. I use no pipelining, threads or anything like that but rather only recv() and send() in a linear serial manner (for simplicity and intuition). I also don't close any sockets as I want to have a persistent connection.
I expect to be able and fetch one whole webpage including subrequested resources like css, img, js, etc..
In my implementation I manage to fetch a few first requests for a webpage. Then it hangs at step 1.
Implementation:
puts("Waiting for user to connect..");
int sock_user = accept(sock, (struct sockaddr*)NULL, NULL);
int sock_host = -1;
printf("User connected.\n");
// Accept requests
while(1){
http_request req;
http_response resp;
// 1. Client ==> Proxy Server
http_parse_request(sock_user, &req); // uses recv()
// 2. Client Proxy ==> Server
if (sock_host < 0)
sock_host = proxy_connect_host(req.header->host);
write(sock_host, req.header->raw_data, req.header->raw_size);
// 3. Client Proxy <== Server
http_parse_response(sock_host, &resp); // uses recv()
// 4. Client <== Proxy Server
write(sock_user, resp.header->raw_data, resp.header->raw_size);
write(sock_user, resp.body ->first_block->data, resp.body ->first_block->size);
}
Log:
---- ......................................... ----
---- after succesfully responded to 4 requests ----
Client ==> Proxy Server
Received 389
Client Proxy ==> Server
Sending.. 389
Sent 389
Client Proxy <== Server
Got header 312
Got body 1437
Response total 1749
Client <== Proxy Server
Sending header.. 312
Sent 312
Sending body.. 1437
Sent 1437
Client ==> Proxy Server
---- Hangs/blocks here ----
Firebug:
Wireshark:
I have no intuition as to what the reason for that block is and I have spent a whole week trying to resolve this problem without a breakthrough.
Among things tried to resolve problem:
Sending some extra CRLF for each response body
Checked the return value from each recv() and send(). (in the log above,
the values getting printed are the return values from recv and
send)
I hope someone could give at least some direction as to how to troubleshoot this or my brain will soon explode :)
You have to be very careful not to read too much data. E.g. make sure that:
header is read only up to the double CRLF; store extra data and sent them with the body
sending of body starts not before the server has sent the complete header (does not apply to this GET case but is important for POST or CONNECT)
there are received and sent only Content-Length bytes from the body
This applies to both the client -> proxy request and the server -> proxy response.
Your sample code stays in an endless loop (while (1) ...). How do you abort this? Do you honor the "Proxy-Connection" header?

Resources