When i try to download a set of files using stream sockets over an HTTP protocol, it only gets data from the first file i try to download.
Assume a loop like the following...
char* file = (char*) malloc(enough_space);
char page[] = {"www.foobar.com"};
for(int n=0 ; n<10 ; n++)
{
sprintf(file, "file%i.html", n);
fopen(file, "wb");
sprintf(request, "GET %s HTTP/1.1\nHost: %s\n\n", file, page);
write( socket, request, strlen(request) );
read_file(output_file);
fclose(output_file);
}
Where a connection has been established first.
This code would give me file1.html, including its header from the server.. But only the first file, and this puzzles me.. What will i have to do in order to get them all?
Thanks up front.
HTTP was designed so that just a single file can be downloaded over a TCP connection. To download multiple files over one TCP connection, you could use HTTP Pipelining. You can read more here: HTTP pipelining request text example
Or you could just use one of the many libraries that will handle this, and many other caveats of HTTP for you: libcurl, libsoup...
Related
while(byte_count != 0){
byte_count = SSL_read(conn,get_buffer,sizeof(get_buffer));
printf("%s",get_buffer);
write_to_file(get_buffer,html,byte_count); // func to write to file
}
I've been trying to write a http/https client using sockets and SSL in C. The task is to get the HTML file of the landing page of a given website into a file on my system. I've handled the HTTP redirections and I was able to read only a portion of the HTTP payload since I've only called recv/SSL_read once. When I put this in a while loop it reads a few more 16kb segments and the connection times out. Is there any other way I can obtain whole of the HTML file ? (Sorry if this question seems vague, I'll be glad to make edits according to your responses)
This question already has answers here:
When does an HTTP 1.0 server close the connection?
(1 answer)
Detect end of HTTP request body
(6 answers)
HTTP header line break style
(3 answers)
Closed 3 years ago.
I'm trying to create a server in C sockets that will allow file uploads through HTTP. The problem I'm having is that I get a very good chunk of the HTTP content but then it just stops sending and begins hanging, perhaps waiting for a response or something? But since recv never hits 0 it never gets to my response. (not sure thats even the reason why).
I looked around on google but most answers are about receiving data and looping to receive more when I'm already doing that.
Heres the code:
fp = fopen("fileName", "a");
for(;;)
{
ret = recv(fd, buffer, BUFSIZE, 0);
if(ret == 0){
\\ Never gets here unless I cancel the web request manually (pressing x where refresh usually is)
printf("Finished recieving");
char* sendHeader = "HTTP/1.1 200 OK\nContent-Type: text/plain\nContent-Length: 8\n\nRecieved";
write(fd, sendHeader, strlen(sendHeader));
break;
}
if(ret == -1) {
printf("Error recieving");
break;
}
fprintf(fp, "%s", buffer);
}
fclose(fp);
Right now i'm just taking the file contents and appending it to a file.
The actual result's I'm getting is:
(using ... to abbreviate)
--WebKitFormBoundaryRMGRl...
Content-Dispotion: form-data; name="filetoUpload"; filename...;
Content-type: application/octet-stream
\n
Actual file contents
\n
--WebKitFormBoundaryRMGRl...
Content-Disposition: form data; name="submit"
Upload License
--WebKitFormBoundaryRMGRl...
Begins writing file contents again, writes about 10 lines, then hangs until I manually cancel request
When I print the byte values, i fill the buffer 2 times then the 3rd time it doesn't completely fill it and just hangs waiting?
Any ideas?
But since recv never hits 0 it never gets to my response ...
recv will return 0 if the client shuts down the connection. But the client will not shut down the connection since it want to receive the response (ok, it could shut down for writing then) and maybe wants to send more requests (HTTP persistent connection).
Instead you have to parse the HTTP request to figure out how much data the client will send in the body. The usual way to do this is by setting the Content-length header to the size of the body. If the size is not known up-front the client might use chunked transfer encoding though were each chunk is prefixed by its length (in hex).
Or in other words: if you are trying to implement HTTP then please make yourself familiar with the standard by studying it and not by making just assumptions. That's what standards are actually for.
So the other day I got a relatively simple assignment, which was to build some client and some server code that in which the server received a message and returned its current system time. It wasn't hard to do, i delivered and got an easy mark.
I started thinking a bit more on it and I decided to set out and try to send the contents of specific file server --> client (server sends contents to client). While i was building the code I tested every so often locally and it worked as intended, the real problem arose when I uploaded the server code to my server (hah) running Ubuntu. Started the server, everything OK, started the client, asked for "index.html" and BAM! half the file wasn't received.
The server prints it (I made it so it printed the contents of file as it sent so I could troubleshoot with more ease).
I have been searching for a bit now and every time I find something that looks useful it ends being in a different programming language and cant find any equivalents in C.
Using sleeps in both client and server code seems to solve this, but I figure it's not good practice.
The code is a mess so I'll include what I figure is relevant, I'll also include a link tot he full code. I really meant to improve it but so got demotivated while trying to fix this that I just made it worse.
Client side
printf("Please specify the filename: ");
fgets(msg,1000,stdin); // get message from std input
if(strcmp(msg,"\n")==0) {
printf("Wrong file name or format\n");
printf("Please specify the filename: ");
fgets(msg,1000,stdin); // get message from std input
}
while(strcmp(msg,"!stop\n")) {
msg[strlen(msg)-1]='\0';
write(sockfd,msg,strlen(msg));
FILE *fp = NULL;
char filecontent[1000];
bzero(filecontent,sizeof(filecontent));
while( (n = read(sockfd,filecontent,1000)) && strcmp(filecontent,"Over and out!")!=0 ) {
if(strcmp(filecontent,"No such file")!=0 && fp == NULL) {
fp = fopen(msg,"w");
}
printf("%s",filecontent);
if(fp !=NULL)
fprintf(fp, "%s",filecontent);
bzero(filecontent,sizeof(filecontent));
}
if(fp != NULL)
fclose(fp);
printf("\nPlease specify the filename: ");
fgets(msg,1000,stdin); // get message from std input
if(strcmp(msg,"\n")==0) {
printf("Wrong file name or format\n");
printf("Please specify the filename: ");
fgets(msg,1000,stdin); // get message from std input
}
}
Server side
char date[50];
time_t ticks;
struct tm *tinfo;
time(&ticks);
tinfo=localtime(&ticks);
strcpy(date,asctime(tinfo));
printf("DATA: %s\n",date);
write(newsocketfd,date,sizeof(date));
while( (n = read(newsocketfd,msg,1000)) && strcmp(msg,"!stop\n")!=0) {
//printf("MSG: %s\n",msg);
if(n<0)
error("ERROR READING");
/////////READING FILE/////////////
char *filename = malloc(sizeof(msg)+1);
strcpy(filename,msg);
printf("'server filename:%s'\n",filename);
FILE *fp = fopen( filename,"r");
if(fp == NULL) {
printf("No such file found\n");
write(newsocketfd,"No such file",sizeof("No such file"));
}
while( fp!=NULL && fgets(msg,1000,fp)!=NULL){
write(newsocketfd,msg,sizeof(msg));
msg[strlen(msg)-1]='\0';
printf("server: '%s'\n",msg);
bzero(msg,sizeof(msg));
}
bzero(msg,sizeof(msg));
bzero(filename,strlen(filename));
n = write(newsocketfd,"Over and out!",sizeof("Over and out!"));
printf("Over\n");
}
sorry for any headaches. Full code here.
Examples:
I think this pretty much shows the problem
My thinking was, the server reads the file, line by line, and sends its, line by line, to the client, when it's done the server sends "over" and the client stops reading from there, it seems however that the client never receives all the information or the "over" signal. Worth adding that this works perfectly fine if I run both codes on my local machine.
Welcome to the world of network programming! Network protocols are layered for a reason. When you send something on a TCP socket, and immediately close the socket, the delivery is unreliable: it may be correctly delivered to the peer, or may vanish because of race conditions.
The only reliable way is to only close the socket when the peer sends an acknowledgement that it could receive everything that was sent. Standard protocol use control messages for that, and you really should contemplate that, but if you do not need your server to be warned for client failures, you could simply have the client to close the connection when it has received "Over and out!". BTW, you should be aware that as TCP is a stream protocol, nothing can guarantee that the message will not be splitted in more than one read, or concatenated to other bytes. So you should keep the end of the previous read (size of the signal string minus one byte), concatenate next read to that and search the string anywhere in the buffer.
Another common way is to use a graceful shutdown: the sender uses shutdown(socket.SHUT_WR) to signal that the communication is over without closing the socket and waits (with a read) for the peer to close the socket when everything has been correctly delivered.
I need to save UDP packets to a file and would like to use the pcap format to reuse the various tools available (wireshark, tcpdump, ...).
There are some information in this thread but I can't find how to write the global file header 'struct pcap_file_header'.
pcap_t* pd = pcap_open_dead(DLT_RAW, 65535);
pcap_dumper_t* pdumper = pcap_dump_open(pd, filename);
struct pcap_file_header file_hdr;
file_hdr.magic_number = 0xa1b2c3d4;
file_hdr.version_major = 2;
file_hdr.version_minor = 4;
file_hdr.thiszone = 0;
file_hdr.sigfigs = 0;
file_hdr.snaplen = 65535;
file_hdr.linktype = 1;
// How do I write file_hdr to m_pdumper?
while( (len = recvmsg(sd, &msg_hdr, 0)) > 0 )
pcap_dump((u_char*)m_pdumper, &m_pcap_pkthdr, (const u_char*)&data);
How should I write the global file header?
If there is no specific pcap function available, how can I retrieve the file descriptor to insert the header using write()?
You shouldn't need to write that header, pcap_open_dead should do it for you. You only need to fill out and write that header yourself if you want to write the file directly instead of using pcap_dump and friends. There's an example here of a trivial program write out a pcap file with those functions.
original answer, concerning writing the file directly:
I can't remember exactly how this works, but I wrote a patch to redir a while ago that would write out pcap files, you may be able to use it as an example.
You can find it attached to this debian bug. (bug link fixed.)
Some of it is for faking the ethernet and IP headers, and may not be applicable as you're using pcap_dump_open and pcap_dump where as the patch linked above writes out the pcap file without using any libraries, but I'll leave this here anyway in case it helps.
If you are interested in UDP and TCP only, you should use DLT_EN10MB instead of DLT_RAW ( cf pcap_open_dead to simulate full UDP packets capture ).
It is much better when editing in WireShak.
Here is a telnet site:
telnet://202.85.101.136:8604/
It is from Hong Kong public library, can I write some programme to get the string / result from the telnet service, and send the request from C / Objective C? thz u.
Sure its possible. Telnet is a pretty simple protocol, you simply need to open a TCP socket and connect it to that IP and Port. When you first connect, the telnet server will send some negotiation requests using the binary protocol defined in RFC854, to which your client is expected to respond. Once negotiation is completed you communicate by simply sending and receiving ASCII data, normally a line at a time.
For a simple "get some data from a host" telnet sessions where you aren't trying to have a real interactive session, it sometimes works to simply accept all the servers negotiation settings to avoid implementing the whole negotiation protocol. To do this, just look for the server to send you several 3-byte commands in the format of: 0xFF 0xFD xx, which is basically the server telling you "I want you to use option X", just respond to this with 0xFF 0xFB xx, which basically is just you agreeing to whatever the server is asking for. Then when you get passed negotiations, you just have to receive lines with a socket read and send commands with a socket write.
If you have a telnet program already on your system, you can use it to do all the connection work for you. Here's a program for gnu/Linux that you can use as a starting point.
It uses popen to execute the system's telnet command. Then it just reads all data from the pipe (stdout if you just executed the telnet command by itself from the shell) and prints it. When there's no more data to read, it exits.
You can send data to the server by opening the pipe in rw mode instead of r and then writing like you would any other file. You could conditionally do stuff like scan your input for Username: and then send a username string too, for instance.
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
const char *cmd = "telnet 202.85.101.136 8604";
char buffer[256];
FILE *pipe = popen(cmd, "r");
if( !pipe ) { perror("popen"); exit(-1); }
while( fgets(buffer, sizeof(buffer), pipe) != NULL &&
!feof(pipe) )
{
if( ferror(pipe) ) { perror("fgets"); break; }
/* Here you do whatever you want with the data. */
printf("%s", buffer);
}
pclose(pipe);
return 0;
}
If you're using Windows, this link explains the alternative to popen.
There's also a program called Expect that can help you automate stuff like this.