Hello I've got a problem trying to create a twitter console app. When getting a response I've got some thrash in it.
Here my code is:
char new_request[1024] = "GET /1.1/statuses/user_timeline.json?count=4&screen_name=twitterapi HTTP/1.1\r\nHost: api.twitter.com\r\nUser-Agent: twitter-terminal-app$
strcat(new_request, bearer);
strcat(new_request, "\r\nAccept-Encoding: gzip\r\n\r\n\0");
BIO_write(bio, new_request, strlen(new_request));
printf("%s\n", new_request);
p = BIO_read(bio, ans, 2047); // Getting header
ans[p] = 0;
printf("%s\n", ans);
char ans2[100000] = "";
p = BIO_read(bio, ans2, 10000); // Getting body
BIO_should_retry(bio);
FILE *file = fopen("result.txt", "w+");
fputs(ans2, file);
printf("%s\n%i\n", ans2, p);
The answer I have in body looks like that:
▒▒is▒HǿJ▒_
▒▒▒▒V▒▒▒▒.▒▒T▒▒f
▒Tm▒m
▒P▒▒1▒|▒}▒%▒▒▒▒Q^▒▒▒▒?▒cx{Չ
F%▒C*;▒▒ŴD/#▒L▒▒▒▒?L▒A▒▒▒N▒▒ĝ▒▒▒▒$▒El▒▒▒▒X▒▒B0▒~%▒▒5▒˲#Y)GY▒ctz▒h&-▒▒▒O>AG▒▒▒l6b▒z▒:K
▒T▒\▒▒2+▒b▒H▒&B ▒▒▒B▒qV▒▒▒▒▒▒
▒▒a_▒l▒?#▒o▒▒▒W▒▒u%▒▒▒;▒1M▒v▒▒L▒▒
Maybe the answer is encoded somehow and that's the problem. Tried to serf dev.twitter.com but didn't find any answer. If I use BIO_gets() instead of BIO_read() the answer is -2. Any ideas?
strcat(new_request, "\r\nAccept-Encoding: gzip\r\n\r\n\0");
...
p = BIO_read(bio, ans, 2047); // Getting header
...
p = BIO_read(bio, ans2, 10000); // Getting body
... Maybe the answer is encoded somehow
You've declared in your HTTP request that you support data compressed with gzip and that's why the server has send you the data compressed. If you would not only read and ignore the HTTP response header but actually take a look at it you would probably notice:
Content-Encoding: gzip
Apart from explicitly allowing compressed data you are doing a HTTP/1.1 request. This means that you would also need to be able to deal with chunked transfer encoding. HTTP/1.1 also implies that by default that connections are persistent so you would need to properly parse the header to find out where the response really ends instead of relying on connection end. You also cannot rely on a fixed size of the header or that header and body can be read with separate BIO_read calls. For example you might need multiple reads for the body or the body might already be included in the single read you do for the header.
Unless you really want to deal with all these problems yourself I recommend you better use an existing library which implements this properly and thus gets you the correct response reliably and not by chance.
If you instead want to learn how this is done I recommend you start with learning more about HTTP, i.e. by reading the wikipedia entry and then continue with all the standards referenced there. I suggest to start with HTTP/1.0 since this is simpler than HTTP/1.1.
Related
I am working on a project where I need to send back 302 reply. Everything seems to work, except I can't remove certain headers, i.e. From, Contact, etc. (I don't want to remove them completely, but rather substitute with my own version of it). I use KEMI with Lua to do so:
KSR.hdr.remove("From")
As I mentioned, this does not work (while other functions from hdr work fine in the same context, namely KSR.hdr.append_to_reply(...).
I decided to look at the Kamailio source code and found following lines of code in kemi.c file:
int sr_kemi_hdr_remove(sip_msg_t *msg, str *hname)
{
...
anchor=del_lump(msg, hf->name.s - msg->buf, hf->len, 0);
if (anchor==0) {
LM_ERR("cannot remove hdr %.*s\n", hname->len, hname->s);
return -1;
}
}
return 1;
}
Looking at the last parameter that del_lump takes, it is of type _hdr_types_t which describes an enum of different header types. Now, in particular to me, there were three headers I was working with:
From (type 4)
Contact (type 7)
Other (type 0)
So my question is, why does that function is hardcoded to take only OTHER headers, but not other ones, i.e. From and Contact? Is that to safeguard from breaking the SIP request (inadvertently removing required headers)?
And as a follow up question, is it even possible to remove From and Contact from reply messages?
I assume the 302 is generated by Kamailio, then several headers are copied from the incoming request, like From, To, Call-Id, CSeq. Therefore if you want a different From in the generated reply, change it in the request and then do msg_apply_changes().
Contact headers for redirect (3xx) replies are generated from the destination set of the request (modified R-URI and branches that can be created by append_branch(), lookup("location") etc.).
More headers can be added to the generated replies using append_to_reply().
Note that I gave the name of the functions for the native kamailio.cfg, but you can find them exported to Kemi as well (by core or textops, textopsx modules).
char get_buffer[10000];
SSL_write(conn,https_request_get,strlen(https_request_get));
printf("GET Sent...\n");
byte_count = SSL_read(conn,get_buffer,sizeof(get_buffer));
printf("recv()'d %d bytes of data in get_buff\n",byte_count);
printf("%s",get_buffer);
fprintf(html, "%s",get_buffer);
fwrite(get_buffer,sizeof(get_buffer),1,html); //html is the file pointer
I've written a C program with sockets that "downloads" the landing page HTML of a given website from the HTTP response. I was able to store the entire HTTP response in a char array and was also able to print it in the console using printf("%s",buffer_name).
Now I am trying to write the same array into a file ( fprintf()), but it only prints the HTTP response excluding the HTML of the page. I understand that there can be null characters and I also tried
fwrite(buffer,sizeof(buffer),1,file_ptr)
which gave me the same output as before (didn't put HTML into the file).
Can anyone help me out with this ?
I'm working with an old version of mongoose (open source web server) in C, which did not provide native access to the requests payload. In order to support POST and PUT requests, I manually modified it: after mongoose reads the headers, I check if Content-Length is set and, if so, I read again from the socket for Content-Lenght characters.
findCL = strstr(conn->buf, "Content-Length:");
if (findCL)
{
// skip "Content-Length:" string
findCL += 15 * sizeof(char);
findCLEnd = (char*)strchr(findCL, delimiter);
sizeLen = findCLEnd - findCL;
strncpy(CLSize, findCL, sizeLen);
CLSize[sizeLen] = '\0';
size = strtoll(CLSize, NULL, 10);
if (size > 0)
{
conn->content_len = read_request(NULL, conn->client.sock, conn->ssl,
conn->buf, conn->buf_size, &conn->data_len);
conn->content_len = size;
perror("recv");
body = (char*)malloc(sizeof(char) * (size + 1));
strncpy(body, conn->buf + conn->request_len, size);
body[size] = '\0';
}
}
So far so good, even if the code is not that beautiful it does the dirty job. Problem is, while in debug the code works fine, but when the code runs as a simple background process the body is not parsed correctly: sometimes the resulting body is truncated, some other times it is just empty. It seems that the problem is caused by the fast queries from the clients.
Not a real answer, but that's how I solved.
The web server module started as a Mongoose web server; I ported it to Civetweb some months ago, hoping that the latest versions of the project also supported the parsing of the body. It was not so, and I had to implement it manually. After some times I discovered that Civetweb had issues in serving Javascript files to IE8 browsers (for some mysterious reason). I reverted to Mongoose and all worked, except the body parsing, which brought to the code above and the subsequent errors. I finally solved by reverting back to Civetweb, stable version 1.9.1, keeping the manual body parsing procedure. This solved both the truncated requests body and the truncated served javascript files. Probably the first version of Civetweb was a not-so-stable beta, although I wonder how did it manage to work for months.
I still haven't checked the diffs between the two versions, but I expect it to be something related to maximum response sizes depending on platform or request headers or whatever else.
I have a simple Restlet service hosted on AppEngine. This performs basic CRUD operations with strings and is working well with all sorts of UTF-8 characters when I test it with curl (for all the verbs).
This is consumed by a simple restlet client hosted in a servlet on another AppEngine app:
// set response type
resp.setContentType("application/json");
// Create the client resource
ClientResource resource = new ClientResource(Messages.SERVICE_URL + "myentity/id");
// Customize the referrer property
resource.setReferrerRef("myapp");
// Write the response
resource.get().write(resp.getWriter());
The above is pretty much all I have in the servlet. Very plain.
The servlet is invoked via jquery ajax, and the json that I get back is well formed and everything, but the problem is that UTF-8 encoded strings are coming back scrambled, for example:
Université de Montréal becomes Universit?? de Montr??al.
I tried adding this line in the servlet (before everything else):
resp.setCharacterEncoding("UTF-8");
But the only diference is that instead of getting ?? I get Universitￃﾩ de Montrￃﾩal (I don't even know what kind of characters those are, asian I suppose).
I am 100% sure the restlet service is OK, because other than debugging it line by line I am able to test it from cmd line with curl and it's returning well formed strings.
By looking at the http header of the response from firefox (when calling the servlet via javascript) I can see the encoding is indeed UTF-8, as expected. After hours of struggling reading every possible related article I came across this restlet discussion and noticed that indeed I do have Transfer-Encoding: chunked on the http header of the response. I tried the proposed solutions (override ClientResource.toRepresentation, didn't do any good so I tried restlet 2.1 as susggested with ClientResource.setRequestEntityBuffering(true), no luck there either) but I am not convinced my issue is related to Transfer-Encoding: chunked at all.
At this point I am out of ideas, and I would really appreciate any suggestions! O_o
UPDATE:
I tried doing a manual GET with a classic UrlConnection and the string is coming back alright:
URL url = new URL(Messages.SERVICE_URL + "myentity/id");
URLConnection conn = url.openConnection();
InputStream is = conn.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
resp.getWriter().print(writer.toString());
So much for being all RESTful and fancy ...but still I have no clue why the original version doesn't work! :/
I tried doing a manual GET with a classic UrlConnection and the string is coming back alright:
URL url = new URL(Messages.SERVICE_URL + "myentity/id");
URLConnection conn = url.openConnection();
InputStream is = conn.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
resp.getWriter().print(writer.toString());
So much for being all RESTful and fancy ...but still I have no clue why the original version doesn't work! :/
Does your response contain the appropriate "Content-Type" header? It should be something like "Content-Type: application/json; charset=UTF-8" (note the charset).
Try starting your development server and retrieving your resource from the command line using cURL and inspecting the headers, e.g. curl -i http://localhost:8080/myentity/id. In theory browsers should assume UTF-8 for JSON, but I wouldn't trust on that.
I am writing an action in a controller where in a certain case, I want to output raw image data directly, and want to set the header content-type appropriate. However I think the header is already being set earlier by CakePHP (I am setting render to be false).
Is there a way to get around this? Thanks!
As said before, CakePHP does not send headers when render is false. Beware though, that any code doing an 'echo' will send headers (except you are using output-buffering). This includes messages from PHP (warnings etc.).
Sending the file can be done in numerous ways, but there are two basic ways:
Send the file using plain PHP
function send_file_using_plain_php($filename) {
// Avoids hard to understand error-messages
if (!file_exists($filename)) {
throw RuntimeException("File $filename not found");
}
$fileinfo = new finfo(FILEINFO_MIME);
$mime_type = $fileinfo->file($filename);
// The function above also returns the charset, if you don't want that:
$mime_type = reset(explode(";", $mime_type));
// gets last element of an array
header("Content-Type: $mime_type");
header("Content-Length: ".filesize($filename));
readfile($filename);
}
Use X-Sendfile and have the Webserver serve the file
// This was only tested with nginx
function send_file_using_x_sendfile($filename) {
// Avoids hard to understand error-messages
if (!file_exists($filename)) {
throw RuntimeException("File $filename not found");
}
$fileinfo = new finfo(FILEINFO_MIME);
$mime_type = $fileinfo->file($filename);
// The function above also returns the charset, if you don't want that:
$mime_type = reset(explode(";", $mime_type));
// gets last element of an array
header("Content-Type: $mime_type");
// The slash makes it absolute (to the document root of your server)
// For apache and lighttp use:
header("X-Sendfile: /$filename");
// or for nginx: header("X-Accel-Redirect: /$filename");
}
The first function occupies one PHP-process / thread while the data is being send and supports no Range-Requests or other advanced HTTP-features. This should therefore only be used with small files, or on very small sites.
Using X-Sendfile you get all that, but you need to know which webserver is running and maybe even a change to the configuration is needed. Especially when using lighttp or nginx this really pays off performance-wise, because these webservers are extremly good at serving static files from disk.
Both functions support files not in the document-root of the webserver. In nginx there are so called "internal locations" (http://wiki.nginx.org/HttpCoreModule#internal). These can be used with the X-Accel-Redirect-Header. Even rate-throtteling is possible, have a look at http://wiki.nginx.org/XSendfile.
If you use apache, there is mod_xsendfile, which implements the feature needed by the second function.
It's not $this->render(false), it's $this->autoRender=false; The header is not sent in the controller action unless you echo something out.
If render is false, cake will not send a header.
You can rely on plain ol' php here.
PNG:
header('Content-Type: image/gif');
readfile('path/to/myimage.gif');
JPEG:
header('Content-Type: image/jpeg');
readfile('path/to/myimage.jpg');
PNG:
header('Content-Type: image/png');
readfile('path/to/myimage.png');