So, I'm writing this simple HTTP client in C and I seem to be stuck on this problem - how do I strip the HTTP headers from the response? After all, if I get a binary file I can't just write the headers out to my output file. I can't seem to go in once the data is already written to a file because linux screams when you try to even view the first few lines of a binary file, even if you know they're just text HTTP headers.
Now, here's the rub (well, I suppose the whole thing is a rub). Sometimes the whole header doesn't even in come in on the first response packet, so I can't even guarantee that we'll have the whole header in our first iteration (that is, iteration of receiving an HTTP response. We're using recv(), here), which means I need to somehow... well, I don't even know. I can't seem to mess with the data once it's already written to disk, so I need to deal with it as it's coming in, but we can't be sure how it's going to come in, and even if we were sure, strtok() is a nightmare to use.
I guess I'm just hoping someone out there has a better idea. Here's the relevant code. This is really stripped down, I'm going for MCVE, of course. Also, you can just assume that socket_file_descriptor is already instantiated and get_request contains the text of our GET request. Here is it:
FILE* fp = fopen("output", "wb"); // Open the file for writing
char buf[MAXDATASIZE]; // The buffer
size_t numbytes; // For the size of the response
/*
* Do all the socket programming stuff to get the socket file descriptor that we need
* ...
* ...
*/
send(socket_file_descriptor, get_request, strlen(get_request), 0); // Send the HTTP GET request
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
/* I either need to do something here, to deal with getting rid of the headers before writing to file */
fwrite(buf, 1, numbytes, fp); // Write to file
memset(buf, 0, MAXDATASIZE); // This just resets the buffer to make room for the next packet
}
close(s);
fclose(fp);
/* Or I need to do something here, to strip the file of its headers after it's been written to disk */
So, I thought about doing something like this. The only thing we know for sure is that the header is going to end in \r\n\r\n (two carriage returns). So we can use that. This doesn't really work, but hopefully you can figure out where I'm trying to go with it (comments from above removed):
FILE* fp = fopen("output", "wb");
char buf[MAXDATASIZE];
size_t numbytes;
int header_found = 0; // Add a flag, here
/* ...
* ...
*/
send(socket_file_descriptor, get_request, strlen(get_request), 0);
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
if (header_found == 1) { // So this won't happen our first pass through
fwrite(buf, 1, numbytes, fp);
memset(buf, 0, MAXDATASIZE);
}
else { // This will happen our first pass through, maybe our second or third, the header doesn't always come in in full on the first packet
/* And this is where I'm stuck.
* I'm thinking about using strtok() to parse through the lines, but....
* well I just can't figure it out. I'm hoping someone can at least point
* me in the right direction.
*
* The point here would be to somehow determine when we've seen two carriage returns
* in a row and then mark header_found as 1. But even if we DID manage to find the
* two carriage returns, we still need to write the remaining data from this packet to
* the file before moving on to the next iteration, but WITHOUT including the
* header information.
*/
}
}
close(s);
fclose(fp);
I've been staring at this code for three days straight and am slowly losing my mind, so I really appreciate any insight anyone is able to provide. To generalize the problem, I guess this really comes down to me just not understanding how to do text parsing in C.
The second self-answer is better than the first one, but it still could be made much simpler:
const char* pattern = "\r\n\r\n";
const char* patp = pattern;
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
for (int i = 0; i < numbytes; i++) {
if (*patp == 0) {
fwrite(buf + i, 1, numbytes - i, fp);
break;
}
else if (buf[i] == *patp) ++patp;
else patp = pattern;
}
/* This memset isn't really necessary */
memset(buf, 0, MAXDATASIZE);
}
That looks like a general solution, but it's not really: there are values for pattern for which it might fail to see a terminator under particular circumstances. But this particular pattern is not problematic. You might want to think about what sort of pattern would cause a problem before taking a look at the more general solution.
So, I know this is not the most elegant way to go about this, but... I did get it. For anyone who finds this question and is curious about at least an answer, here it is:
int count = 0;
int firstr_found = 0;
int firstn_found = 0;
int secondr_found = 0;
int secondn_found = 0;
FILE* fp = fopen("output", "wb");
char buf[MAXDATASIZE];
size_t numbytes;
int header_found = 0;
/* ...
* ...
*/
send(socket_file_descriptor, get_request, strlen(get_request), 0);
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
if (header_found == 1) {
fwrite(buf, 1, numbytes, fp);
}
else {
// These buf[i]'s are going to return as integers (ASCII)
// \r is 13 and \n is 10, so we're looking for 13 10 13 10
// This also needs to be agnostic of which packet # we're on; sometimes the header is split up.
for (int i = 0; i < numbytes; i++) {
if (firstr_found == 1 && firstn_found == 1 && secondr_found == 1 && secondn_found == 1) { // WE FOUND IT!
header_found = 1;
// We want to skip the parts of the buffer we've already looked at, that's header, and our numbytes will be decreased by that many
fwrite(buf + i, 1, numbytes - i, fp);
break;
}
if (buf[i] == 13 && firstr_found == 0) { // We found our first \r, mark it and move on to next iteration
firstr_found = 1;
continue;
}
if (buf[i] == 10 && firstr_found == 1 && firstn_found == 0) { // We found our first \n, mark it and move on
firstn_found = 1;
continue;
}
else if (buf[i] != 13 && buf[i] != 10) { // Think about the second r, it'll ignore the first if, but fail on the second if, but we don't want to jump into this else block
firstr_found = 0;
firstn_found = 0;
continue;
}
if (buf[i] == 13 && firstr_found == 1 && firstn_found == 1 && secondr_found == 0) {
secondr_found = 1;
continue;
}
else if (buf[i] != 10) {
firstr_found = 0;
firstn_found = 0;
secondr_found = 0;
continue;
}
if(buf[i] == 10 && firstr_found == 1 && firstn_found == 1 && secondr_found == 1 && secondn_found == 0) {
secondn_found = 1;
continue;
}
}
}
memset(buf, 0, MAXDATASIZE);
count++;
}
close(s);
fclose(fp);
Adding another answer because, well I suppose I think I'm clever. Thanks to #tadman for the idea of a counter. Look here (I'm going to shave off a lot of the bloat and just do the while loop, if you've looked at my other code blocks you should be able to see what I mean here) ...
/* ...
* ...
*/
int consec_success = 0;
while ((numbytes = recv(socket_file_descriptor, buf, MAXDATASIZE - 1, 0)) > 0) {
if (header_found == 1) {
fwrite(buf, 1, numbytes, fp);
}
else {
for (int i = 0; i < numbytes; i++) {
if (consec_success == 4) {
header_found = 1;
fwrite(buf + i, 1, numbytes - i, fp);
break;
}
if (buf[i] == 13 && consec_success % 2 == 0) {
consec_success++;
}
else if (buf[i] == 10 && consec_success % 2 == 1) {
consec_success++;
}
else {
consec_success = 0;
}
}
}
memset(buf, 0, MAXDATASIZE);
}
/* ...
* ...
*/
Related
I am trying to make a small proxy service.
Now the problem is that I can not show the image from web service.
I use Rio_readlineb to read image data back.But it could not show up.
while ((Rio_readlineb(&rio_client, buf, MAXLINE)) !=0){
Rio_writen(fd, buf, strlen(buf));
}
But when I use Rio_readnb to read.The problem is solved.
while ((size = Rio_readnb(&rio_client, body, MAXLINE)) != 0)
Rio_writen(fd, body, size);
I do not know where is the problem.
readnb
ssize_t rio_readnb(rio_t *rp, void *usrbuf, size_t n)
{
size_t nleft = n;
ssize_t nread;
char *bufp = usrbuf;
while (nleft > 0) {
if ((nread = rio_read(rp, bufp, nleft)) < 0)
return -1; /* errno set by read() */
else if (nread == 0)
break; /* EOF */
nleft -= nread;
bufp += nread;
}
return (n - nleft); /* return >= 0 */
}
rio_readlineb
ssize_t rio_readlineb(rio_t *rp, void *usrbuf, size_t maxlen)
{
int n, rc;
char c, *bufp = usrbuf;
for (n = 1; n < maxlen; n++) {
if ((rc = rio_read(rp, &c, 1)) == 1) {
*bufp++ = c;
if (c == '\n') {
n++;
break;
}
} else if (rc == 0) {
if (n == 1)
return 0; /* EOF, no data read */
else
break; /* EOF, some data was read */
} else
return -1; /* Error */
}
*bufp = 0;
return n-1;
}
Is the rio_readlineb cut off the data? But If MAXLINE is too small,rio_readnb also will cut off the data.So where is the problem?
I assume that you are trying to implement a HTTP 1.0 proxy.
rio_readlineb is looking for the newline character (\n, hex value 0x0A, decimal value 10) in the data is is reading. When it has read that character, it returns successfully, indicating the number of characters read. This works fine for reading textual data, like HTTP headers, where each header line is terminated by the newline character.
However, image data is not terminated by a newline character. In fact, images can contain this character anywhere within their data. The moment rio_readlineb finds \n in the image data, it returns, but the buffer likely does not contain the entire image yet. So, you're sending a corrupt image (too short) to the proxy client.
In HTTP 1.0, the server writes the response headers and then data to the socket. After it has written all the data, it closes the socket. You need to read until end-of-file to get all the image data. This is what your Rio_readnb & Rio_writen loop is doing.
I wrote a C application for a socialization network and also a simple room-based chat. I used ncurses, sockets and basic networking stuff.
The problem is that my function uses select() to read from server socket AND stdin so when I start to write a message, the output window freezes and only shows messages from other clients after I hit enter.
I tried everything possible .. Is there a way to fix this ?
I also tried to force nocbreak().It works okay but if I do that, when I write the message, the echoing is disabled and nothing shows up in the input window as I type, even though the message is there but like "invisible".
Here is the code :
ssize_t safePrefRead(int sock, void *buffer)
{
size_t length = strlen(buffer);
ssize_t nbytesR = read(sock, &length, sizeof(size_t));
if (nbytesR == -1)
{
perror("read() error for length ! Exiting !\n");
exit(EXIT_FAILURE);
}
nbytesR = read(sock, buffer, length);
if (nbytesR == -1)
{
perror("read() error for data ! Exiting !\n");
exit(EXIT_FAILURE);
}
return nbytesR;
}
ssize_t safePrefWrite(int sock, const void *buffer)
{
size_t length = strlen(buffer);
ssize_t nbytesW = write(sock, &length, sizeof(size_t));
if (nbytesW == -1)
{
perror("write() error for length ! Exiting !\n");
exit(EXIT_FAILURE);
}
nbytesW = write(sock, buffer, length);
if (nbytesW == -1)
{
perror("write() error for data ! Exiting !\n");
exit(EXIT_FAILURE);
}
return nbytesW;
}
void activeChat(int sC, const char *currentUser, const char *room)
{
char inMesg[513], outMesg[513];
char user[33];
int winrows, wincols;
WINDOW *winput, *woutput;
initscr();
nocbreak();
getmaxyx(stdscr, winrows, wincols);
winput = newwin(1, wincols, winrows - 1, 0);
woutput = newwin(winrows - 1, wincols, 0, 0);
keypad(winput, true);
scrollok(woutput, true);
wrefresh(woutput);
wrefresh(winput);
fd_set all;
fd_set read_fds;
FD_ZERO(&all);
FD_ZERO(&read_fds);
FD_SET(0, &all);
FD_SET(sC, &all);
wprintw(woutput, "Welcome to room '%s' \n Use /quitChat to exit !\n!", room);
wrefresh(woutput);
while (true)
{
memcpy( &read_fds, &all, sizeof read_fds );
if (select(sC + 1, &read_fds, NULL, NULL, NULL) == -1)
{
perror("select() error or forced exit !\n");
break;
}
if (FD_ISSET(sC, &read_fds))
{
memset(inMesg, 0, 513);
safePrefRead(sC, user);
safePrefRead(sC, inMesg);
wprintw(woutput, "%s : %s\n", user, inMesg);
wrefresh(woutput);
wrefresh(winput);
}
if (FD_ISSET(0, &read_fds))
{
//wgetnstr(winput, "%s", outMesg);
int a, i = 0;
while ( i < MAX_BUF_LEN && (a = wgetch(winput)) != '\n')
{
outMesg[i] = (char)a;
i++;
}
outMesg[i] = 0;
if (outMesg[0] == 0)
continue;
if (strcmp(outMesg, "/quitChat") == 0)
{
safePrefWrite(sC, outMesg);
break;
}
safePrefWrite(sC, outMesg);
delwin(winput);
winput = newwin(1, wincols, winrows - 1, 0);
keypad(winput, true);
wrefresh(winput);
}
}
delwin(winput);
delwin(woutput);
endwin();
}
-safePrefWrite and safePrefRead are wrappers for prexied read / write and error treating
-sC is the server socket.
LE: I tried using fork and threads. Using fork was behaving the same and threads were a disaster, the terminal was messed up.
Thank you.
modify the while(true) loop to only handle one char at a time for the stdin.
Which mostly means for stdin, read a single char:
if char is '\n' then handle as currently,
otherwise, just append char to the buffer to write.
Always, before appending a char to buffer to write, check that buffer is not full.
add code to handle the case where the buffer to write is full
end the function with this sequence:
delwin(winput);
delwin(woutput);
endwin();
endwin();
to end both windows.
Do not call endwin() during processing of the socket input.
Do not call endwin() when select() returns an error condition
the fd_set is not an intrinsic size in C, so use memcpy() to set
read_fds from all. suggest:
memcpy( &read_fds, &all, sizeof read_fds );
the parameter: currentUser is not used, suggest inserting the line:
(void)currentUser;
to eliminate a compiler warning message.
for readability, and ease of understandability, suggest #define the magic numbers 513 and 33 with meaningful names, then use those meaningful names throughout the code.
#define MAX_BUF_LEN (513)
#define MAX_USER_LEN (33)
this line: outMesg[i] = a; raises a compiler warning, suggest:
outMesg[i] = (char)a;
This line: while ( (a = wgetch(winput)) != '\n') can allow the buffer outMesg[] to be overrun, resulting in undefined behaviour and can lead to a seg fault event. suggest:
while ( i < MAX_BUF_LEN && (a = wgetch(winput)) != '\n')
Suggest posting the prototypes for the safePrefWrite() and safePrefRead() functions, similar to:
void safePrefRead( int, char * );
void safePrefWrite( int, char * );
As noted by #user3629249, there are several criticisms which can be applied to the sample code. However, OP's question is not addressed by those improvements.
OP seems to have overlooked these functions:
cbreak or raw, to make wgetch read unbuffered data, i.e., not waiting for '\n'.
nodelay or timeout, to control the amount of time wgetch spends waiting for input.
By the way, making select work with a curses program will make assumptions about the curses library internal behavior: getting that to work reliably can be troublesome.
Fixed it finally by using only the big loop.
Here is the code if anyone has the same problem in the future :
if (FD_ISSET(0, &read_fds))
{
inChar = wgetch(winput);
if (inChar == 27)
{
safePrefWrite(sC, "/quit");
break;
}
if (inChar == KEY_UP || inChar == KEY_DOWN || inChar == KEY_LEFT || inChar == KEY_RIGHT)
continue;
if (inChar == KEY_BACKSPACE || inChar == KEY_DC || inChar == 127)
{
wdelch(winput);
wrefresh(winput);
if (i != 0)
{
outMesg[i - 1] = 0;
i--;
}
}
else
{
outMesg[i] = (char)inChar;
i++;
}
if (outMesg[i - 1] == '\n')
{
outMesg[i - 1] = 0;
i = 0;
if (outMesg[0] == 0)
continue;
if (strcmp(outMesg, "/quit") == 0)
{
safePrefWrite(sC, outMesg);
break;
}
safePrefWrite(sC, outMesg);
delwin(winput);
winput = newwin(1, wincols, winrows - 1, 0);
keypad(winput, true);
wrefresh(winput);
memset(outMesg, 0, 513);
}
}
I also use raw() to disable signals and to treat the codes how I want.
Anything else above and below this "if" is just like in the 1st post.
I had a function that would receive data from an IRC server in 512-byte chunks and print it to the terminal window, it went like this:
int mainLoop(redchan_t *redchan)
{
int socketDescriptor = redchan->socketDescriptor, i, bytesReceived;
char workingBuffer[RECVBUFF] = {[0 ... RECVBUFF - 2] = '0', '\0'};
puts("Recieving data...");
do
{
if ((bytesReceived = recv(
socketDescriptor,
workingBuffer,
RECVBUFF,
0)) == 0)
exit(EXIT_FAILURE);
for (i = 0; i < bytesReceived; ++i)
printf("%c", workingBuffer[i]);
}
while(1);
return 0;
}
But I wanted to make it more orthogonal so I took out the call to recv and put it in its own routine which looks like this:
int redchanRecv(redchan_t *redchan, int size, char *buffer)
{
int totalBytes = 0, bytesReceived;
while (totalBytes < size)
{
if ((bytesReceived = recv(
redchan->socketDescriptor,
buffer + totalBytes,
size - totalBytes,
0)) <= 0)
cleanup_redchan(redchan, serverInfo);
totalBytes += bytesReceived;
}
return 0;
}
I would then just call redchanRecv() in each iteration of my main loop and then print the buffer.
But when I run it, it prints out almost everything, except for one line.
That last line never makes it to the terminal (does it even make it into the buffer?).
I really feel like I'm making a rookie mistake but hell if I can see it.
How can I solve this?
stdout is line-buffered by default in your system, add a fflush(stdout) call to have your last line printed.
I tried and looked up TONS of pages over the net, found NOTHING that fits windows and is working always, I tried this one over the TCP protocol (streaming byte-by-byte untill you bump into 3 - I tried it on files with no 3 in it :PPPP and by 3 I mean the ASCII value 3 and not the digit '3').
Server side:
int sendFile(SOCKET s, const char* file_path)
{
FILE* fp = fopen(file_path, "rb");
int i, err = 0, bytesSent, isOk = 1;
char ch = 0;
if(!fp)
{
fclose(fp);
return 1;
}
while(ch != EOF && isOk)
{
fread(&ch, sizeof(char), 1, fp);
if(ch != EOF)
{
bytesSent = send(s, &ch, sizeof(char), 0);
if(bytesSent <= 0)
{
return 1;
}
}
else
{
isOk = 0;
}
}
ch = 3;
bytesSent = send(s, &ch, sizeof(char), 0);
fclose(fp);
return 0;
}
Client side:
int recvFile(SOCKET s, const char* file_path)
{
FILE* fp = fopen(file_path, "wb");
int bytesRecieved;
char ch;
if(!fp)
{
fclose(fp);
return 1;
}
bytesRecieved = recv(s, &ch, sizeof(char), 0);
if(bytesRecieved <= 0)
{
return 1;
}
while(ch != 3)
{
fwrite(&ch, sizeof(char), 1, fp);
putch(ch);
bytesRecieved = recv(s, &ch, sizeof(char), 0);
if(bytesRecieved <= 0)
{
return 1;
}
}
fclose(fp);
return 0;
}
The sockets are functioning well and sending and receiving well (I'm talking about sending regular messages, without the functions).
It's not returning 1, it's just turns into an infinite loop.
No idea why it's not working, any idea ? I'm totally desperate.
Instead of comparison with EOF, you should use the feof function while reading from file. EOF is just an error code returned by some functions, not an actual character in the file.
Also, I notice that the thing with the character 3 seems a way to signal the end of the file. You should consider normalizing the data transfer. You could send at the beginning of the communication the size of the file, and the client reads exactly the size of the file, to avoid having problems with files that contain that character (3 is still a valid character).
For inspiration, take a look at an old project of mine that implements this behavior (although it's linux only).
I once fell in trouble reading on tcp socket by chunks smaller than what was used for writing. On a socket you write packets of a determined length (write or send), and implementation may discard the end of the packet if read size is shorter.
From man page on recv : All three routines [recv, recvfrom, recvmsg] return the length of the message on successful completion. If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from.
When reading, you should allways use a buffer of a size at least equals of the longest buffer used in writing.
You could dump what you receive to confirm.
From the below piece of code, why I am getting Reading Socket for response
int Read(int sock, char *p, int size)
{
int remain, read=0;
remain = size;
while (remain > 0 ) {
if ((read = recv(sock, p, remain, 0)) < 0) {
/* Error */
return(read);
} else if (read == 0 || *p == 0x0a) {
/* EOF */
break;
}
remain -= read;
p += read;
}
return(size - remain);
}
while (!done)
{
printf("***Reading Socket for response***");
rsplen= Read(myVsHandle.sock,(char *)encXMLResponse,MAX_RSP_LEN);
if (rsplen < 0 )
{
printf("Internal Communication Error");
return -1;
}
else if (rsplen >0)
printf("Revieved response");
done++;
return 0;
else if (rsplen == 0)
{
printf("Reading socket");
}
You are waiting for MAX_RSP_LEN bytes to be read - is there that many bytes to be read? Maybe your process is stuck in a blocking read().
Also depending on the sort of socket you are recv()ing from, there is no guarantee on the amount of data you will read, so specifically looking for a value 0x0a may not work.
Your problem could be that you are not ending your output with a newline. Try ending your outputs with a newline (\n). stdout is line buffered, so you may not see anything for a long time if you don't output a newline.
Another possibility is that you don't return from Read() unless you read the specified number of bytes. Depending upon the value of MAX_RSP_LEN, and the amount of data available, Read() may wait forever.
Also, your test: *p == 0x0a looks suspicious. What are you testing here?
Edit: There is another "bug":
else if (rsplen >0)
printf("Revieved response");
done++;
return 0;
else...
You are missing curly braces. In the current form, the code shouldn't compile. Please post actual code.
This:
if ((read = recv(sock, p, remain, 0)) < 0) {
Should be
if ((read = recv(sock, p, remain, 0)) > 0) { // Greater then 0, because recv returns the number of bytes received if successful, if it fails -1.
You're missing curly braces around the:
else if(rsplen > 0)
... statements
It should be:
...
}else if (rsplen >0){
printf("Revieved response");
done++;
return 0;
} ...