I have a simple C web server. i can connect to that web server by browsing in my web browser: http://127.0.0.1:5001/
What i want to be able to do is retrieve a file (locally) using GET.
so i want to be able to connect with http://127.0.0.1:5001/local/file.txt
and have the server send me the file.
I am having trouble understanding where i grab the the URL from within the C program? Once the socket is connected it reads into a buffer. to see what is being stored in the buffer i made it print out and got the following
GET /local/file.txt HTTP/1.1
Host: 127.0.0.1:5001
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
My question is, can i use the information that is allocated to the buffer as a way to recognise a GET command and open the file path that is located in that buffer? That is, search that buffer for the string GET /filepath/
I can't seem to find anything online to point me in the right direction.
I have read through Beej's Guide to Network Programming which was helpful, but didn't answer my question.
What is the best practice to open a file from a URL? Using a standard C library and UNIX functions and "translating" sockets by hand without the use of such things as libcurl etc.... i don't want to use libcurl, probably something along the lines of strstr etc... find keyword and open filepath...??
can i use something like this? so pass buffer into it...
if(strstr(buffer, "GET") != NULL)
{
// Get address
inf->addr = strdup(line+4);
except that this would output everything after GET to a structure pointer... is there a way to limit this? ie... stops after a space is recognised? so just before HTTP/1.1, i read that /0 is a terminating function... will strstr() stop after reaching a space?
Related
[Intro]]
I have been practicing C for 4 Months 5 times a Week, I am attempting to make the best possible guess for moving into web server/client programming for HTTP2. I have worked with HTML5 & Javascript I also work with HTTP2 & have used SPDY on a low leval.
[Info]
Debian GNU/Linux (Buster), V.10
Compiler - Clang
headers HTTP2(?)
Headers HTTP(sys/socket.h, sys/types.h, signal.h, arpa/inet.h)
Network Protocol - TCP, HTTP2, HTTP.
[Question]
When playing around with Java you notice a-lot of HTTP2 being used in Game engines. HTTP2 is being standardized more & more in current languages as of 2021.(FOCUS POINT->) C has a standard Library for working with HTP & TCP.
HTTP2 uses Encryption & as of my knowledge is based in "TCP". Is there a standard for C working with, "HTTP2 TCP" types. I need to be able to read the Text from HTTP2 & easily preform tasks(send(), recv().(<-FOCUS POINT END).
[FINAL]
Can I Realistically develop HTTP2 Programs Using C standard POSIX Library's.
How would I go about using HTTP2 in C & is it even posible.
HTTP is a protocol. It defines the way computers communicate with each other. There a lot of different protocols out there; HTTP/2 and HTTP is based on TCP/IP, which almost all computers use to connect to servers. TCP/IP specifies how to create a connection between a client and a specific server, and send raw bytes. HTTP(2) is a layer over this, which defines how a client should tell the server what resource it wants.
The API provided by base POSIX extends to only raw TCP/IP. Here's an example of what an application could send to a web server to request a web page (this is an HTTP/1 request, HTTP/2 is binary and encrypted, making it not human-readable)
GET /somepage HTTP/1.0
Host: example.com
However, it's kind of inconvenient to assemble these requests manually in your program, especially when you start doing things like sending parameters, encryption, etc. In this case, people should use some libraries on the machine. It's important that these are nonstandard. Probably one of the most common ones is libcurl.
Here's a libcurl tutorial. To summarize:
You can compile your program with libcurl by providing some flags to the compiler and the linker. You can get these by running curl-config --cflags and curl-config --libs.
Initialize libcurl with curl_global_init(bitfield). The bitfield can have CURL_GLOBAL_WIN32 and CURL_GLOBAL_SSL as its values. You probably need CURL_GLOBAL_SSL since you want to use encryption.
libcurl has two interfaces: easy and multi. I am going to cover easy.
Get an easy handle by calling curl_easy_init(). This handle allows you to make requests.
Set options on the handle by using curl_easy_setopt(handle, option, value). handle is the "easy handle" you obtained, option is a constant - for example CURLOPT_URL for setting the URL to request - and value is the value you want to set it to. It's often a string.
Set a callback function with option as CURLOPT_WRITEFUNCTION. It should have the signature size_t(void *buffer, size_t size, size_t nmemb, void *userp). It's called when libcurl receives data from the server. Usually, it's called many times. nmemb points out the size of the data in buffer. It should return the count of bytes it successfully processed. Normally this is nmemb.
Execute the request with curl_easy_perform(handle).
When you're done with every request you want to make, call curl_global_cleanup().
Within C standard library there is no part dedicated to http2.
However, working with http2 in C is definitely possible, well documented and widely used. For example:
curl - info: https://curl.se/docs/http2.html
nghttp2
More info about http2 and various programming languages can be found here: https://github.com/httpwg/http2-spec/wiki/Implementations
I have a simple networking program for sending and responding to HTTP requests/responses. However, if I wanted to send a HTTP or another request via a SOCKS5 proxy, how would I go about this? I'm using C Unix sockets.
I could solve this by creating a proxy server in Linux. However, my intention is so that I can send this to a proxy server I do not own so that it can be forwarded to the destination server. I couldn't seem to find a library. I found an RFC for it which I've read https://www.rfc-editor.org/rfc/rfc1928.txt , but i'm still not 100% unsure how to format my request.
Am I supposed to send the segments for the handshakes as hex? If so, would I send a string of hex 0F3504 or \x0F \x35 \x04 or 0x0F3504? Another question is do i need to denote that header = value in the message, or does the SOCKS5 server know what header i am referring to by the position of the byte it is looking at from the message I've sent?
Any clear up would be very much appreciated
Some time ago I wrote an open source C library that may help you: https://github.com/brechtsanders/proxysocket
Maybe you can use the library. Its quite easy, just replace the connect() with stuff from the library and for the rest you can keep the rest of your code that uses the socket that is returned.
Or you can take a peek in the code to see how it's done there.
I am trying to write a proxy server in C language under Linux. It was working fine (I had the perception that it was working fine) until I tried it for streaming media.
Lemme first tell the problem and then I'll jump onto streaming media.
To read the incoming data from the website and forward it to the actual client I do this
count = read(websitefd,buffer,BUFSIZ);
write(clientfd,buffer,count);`
in a continuous while loop until I read up all the data on that socket.
Now the problem is if the actual website sends an HTTP packet with content length field as 1025 bytes and other part of data in other packets then still I always wait for BUFSIZ(8192 bytes) and then I send 8192 bytes to the client machine all together. for normal octet-stream it works fine even though I know its not the right method, because I should forward the packets same as the actual server. So if actual server sends me 2 packet of sizes 1024 and 1024 bytes I send the client a packet of 2048 bytes with the first packet with the HTTP header saying that the content length is 900 bytes (rest all being the http header assuming) but actually I forward a packet of 2048 bytes to client. For Content Type: application/octet-stream it just downloads the whole thing and displays it either as image or html text or asks me to save it.
When the client request for a streaming media, because of the above reason the client is not able to play the video. So what should I do now ? Thanks for reading my question. Please help me out. :)
First, I strongly recommend using an existing proxy server as the base of any proxy system. The HTTP standard is quite complex, much more than you realize. If you are going to implement a proxy server, read RFC 2616 at least three times first.
Second, your proxy server must parse HTTP headers to figure out how much it must send. The three main ways to know how much data to send are as follows:
If a Content-Length header is present and no Transfer-Encoding header is present: The Content-Length header specifies how much data to relay in bytes. Just go into a loop copying.
If a Transfer-Encoding: chunked header is present: You must parse the chunked transfer encoding chunk headers. This encoding is frequently used for streaming data, where the total size is not known ahead of time. It's also often used for dynamic data generated by scripts.
If some other Transfer-Encoding header is present: Close the connection and report a 500 error, unless you know what that encoding is.
If a Content-Length header is not present, and no Transfer-Encoding header is present: Check for Connection: close (must be present, in HTTP/1.1) and Connection: keep-alive (must NOT be present in HTTP/1.0). If these conditions are violated, trigger a 500 error. Otherwise just keep passing data until the server closes the connection.
I'm deliberately making this a bit vauge - you MUST read the standard if you're implementing a proxy server from scratch, or you will certainly introduce browser incompatibilities and/or security holes! So please, don't do so. Use lighttpd or varnish or something as the core proxy server, and just write a plugin for whatever functionality you need.
I suppose media is transferred in chunks, i.e no Content-Length is present and data is sent until finished.
As bdonlan said please read how chunked data works,
And i agree HTTP is pretty nasty (due to many changes and interpretations in time)
I'm attempting to write my own http 1.1 server, just for fun and learning about more about HTTP, sockets, and threading.
I've gotten a good start i think with only delivering static pages (using c, which I would prefer to stay in for the time being). I have a test page I wrote a while ago and deliver it's ~50 files in 124ms according to chrome, without using threads or keep-alive sockets.
I've found it very difficult to get threading/keep-alive working at all. There are little to no resources on the web (that I can find in my hours of Googling) that explain keep-alive connections in detail. If anyone could recommend a good book on HTTP server programming, I would greatly appreciate it.
I've done some threading and socket programming before by making a simple chat program, so I have at least some experience with it.
The issue I'm having is that when I attempt to incorporate threads, the client browser sets up multiple connections. Somewhere along the line, the server gets confused and the client just sits there waiting for responses and the server stops doing anything. I send the Connection: Keep-Alive header, but that doesn't change anything and when I incorporate keep-alive it and create a loop for getting requests in the threaded function, it stalls until the connection is closed.
I would appreciate it if someone could give me some pseudo code on how to get keep alive/threading working for this so the client stops creating multiple connections at a time.
A brief description of whats going on:
main function
load in static pages to large array of fileinfo struct that hold the file data and length
create the socket
set it to listen to port 80
set it to listen for 10 connections at a time(i know this is low...)
start an endless loop
block while waiting for someone to connect
check if it's a localhost connection
shutdown the server
otherwise
start a thread(with pthread), sending it the socket variable
loop
Thread Function
setsock opt for 3 sec timeout on send/recv and enable Keep-alive
start endless loop
read in request
if request timed out, break the loop
Validate Request function call
Create Reponse function call
Send response
if request contained Connection: close header break the loop
loop
close socket
return
I would recommend looking at GNU libmicrohttpd. It focuses squarely on providing a framework upon which to build HTTP 1.1 servers. It is small and supports keep-alive with and without threading. (Personally I use it without threading. It has several threading models too.)
Even if you decide to write your web server from scratch, I would suggest looking at libmicrohttpd to gain insight in not only how the protocol works, but how the library models "the work flow" of a web server in a very clean way. I think it is a mistake to imagine that keep-alive implies threading and I think it is an impediment to understanding keep-alive.
(Regarding Apaches' credits as a web server, it is pretty huge, and there is a lot in there not related to protocols, but rather things like its plugin system and so on.)
I'd recommend grabbing the source for Apache and seeing how they handle it. There's not much point in psuedo code when you can see how the real thing works.
Perhaps you could look at Apache's code for some clues. It is written in C.
Hopefully someone will come along and give a more detailed answer :)
There are two problems here:
What if content is encoded:gzip...
Do I also need to change the header part to make the HTTP packet valid(checksums if any?)
UPDATE
Can someone with actual experience elaborate the steps involved?
I'm using winpcap and bpf tcp and src port 80 to filter the traffic,so my job lies in this callback function:
void packet_handler(u_char *param, const struct pcap_pkthdr *header, const u_char *pkt_data)
WinPcap doesn't allow you to change a packet that was already sent.
If the packet was sent, WinPcap won't prevent it from reaching its destination.
If you want to send another response - in addition to the response that was sent - I'm not sure what you're trying to achieve.
Decompress it with a GZIP decompresser.
Remove the Content-Encoding header and add a Content-Length header representing the new length in bytes.
That said, for a better answer you'll need to supply more context in the question. This is namely a smell. What is it you're trying to achieve and for which you think that modifying the HTTP response is the right solution?
libpcap is used for capturing. If you want to do modification and injection of network packets you need another library, such as libnet.
winpcap is an odd way to try modifying a TCP stream - you don't explain why you are trying to do this, but you should probably be able to achieve this by writing your own HTTP proxy instead. That way, you get presented with a straight datastream you can intercept, log and modify to your heart's content. Once you do that, strip out Accept-Encoding from the request headers, then you'll never need to deal with gzipped responses in the first place.
There are no HTTP checksums, but the lower layers do have checksums; by operating on the application level as a proxy server, you let the network stack deal with all this for you.