I am using libcurl to get http pages and I'm stuck with a problem.
The page I am trying to get is:
http://newsnow.in/news/tunisians-head-to-elect-president-in-runoff-u-t-san-diego
If you open the page you will receive an error stating that the page isn't redirecting properly. When I try to fetch it libcurl simply hangs, no error whatsoever. I also set the timeout which didn't work, I guess since it received the redirect message.
Here is my setting:
curl_easy_setopt(curl, CURLOPT_URL, "http://newsnow.in/news/tunisians-head-to-elect-president-in-runoff-u-t-san-diego"); // should be changed later with setLink()
/* example.com is redirected, so we tell libcurl to follow redirection */
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:31.0) Gecko/20100101 Firefox/31.0");
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20);
/* send all data to this function */
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, filePtr);
And start it with the usual way:
res = curl_easy_perform(curl);
any ideas on what might solve the problem?
For anyone encountering this problem in the future, following #SSC 's comment I added the verbose flag:
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
And found out the page enters an infinite loop of redirects, this was easily solved using the max redirect flag:
curl_easy_setopt(curl, CURLOPT_MAXREDIRS, 3L);
Related
I’m trying to use libcurl to call Rest API.
My server env : Oracle tuxedo(pro *c), AIX 7.1
It does work using command “curl” on prompt.
I can also see the whole log by using verbose option.
But it keeps stopping when I tried to use it on client compiling with libcurl.
According to log.
It says…
Trying 123.456.789.00:443…
Connected to “api url”(123.456.789.00) port 443 (#0)
ALPN, offering http/1.1
Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
successfully set certificate verify locations :
CAfile: /var/ssl/cert.pem
CApath: /var/ssl/certs/
And it stopped here!!!!
When I use curl command on prompt
It says exactly same,
But keeps going…
TLSv1.2 (OUT), TLS header, Certificate Status (22):
TLSv1.2 (OUT), TLS handshake, Client hello (1):
…
…
Etc…
I have no idea what causes this
And what is the difference…
Can anybody give me some advice please?
Or is there any other way to call RestAPI easily on Pro *C or C?
========================================
All I did is using sample of libcurl.
static sample(){
CURL *curl;
CURLcode res;
curl = curl_easy_init();
struct curl_slist *list = NULL;
if(curl){
curl_easy_setopt(curl, CURLOPT_URL, "https://ApiUrl.here");
list = curl_slist_append(list, "Content-Type: application/json");
list = curl_slist_append(list, "ApiKey : realKey");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list);
curl_easy_setopt(curl, CURLOPT_SSLVERIFYPEER, 1L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1L);
res = curl_easy_perform(curl);
curl_slist_free_all(list);
if(res != CURLE_OK){
fprintf(stderr, "curl_easy_perform() failed : %s \n", curl_easy_strerror(res));
}
curl_easy_cleanup(curl);
}
}
list = curl_slist_append(list, "ApiKey : realKey");
I think that line is the issue.
RFC7230 3.2.4. Field Parsing
No whitespace is allowed between the header field-name and colon. In
the past, differences in the handling of such whitespace have led to
security vulnerabilities in request routing and response handling. A
server MUST reject any received request message that contains
whitespace between a header field-name and colon with a response code
of 400 (Bad Request). A proxy MUST remove any such whitespace from a
response message before forwarding the message downstream.
Remove the space:
list = curl_slist_append(list, "ApiKey: realKey");
I'm using curl-7.64.1 (https://curl.haxx.se/windows/) 32 bit, with OpenSSL 1.1.1b (32 bit) and tidy-5.0.0 for XML parsing.
When I'm home making any connection work fine, but when I'm at School, connected to the Wifi of this one. All the connection are refused.
I tried to make a connection to https://www.google.fr/ and with CURLOPT_VERBOSE I get on the screen the message :
Trying 172.217.18.227...
TCP_NODELAY set
connect to 172.217.18.227 port 443 failed: Connection refused
Failed to connect to www.google.fr port 443: Connection refused
Closing connection 0
Failed to connect to www.google.fr port 443: Connection refused
I tried first to change the port, to 8080. I wrote https://www.google.fr:8080 as URL for libCurl.
But still have :
Failed to connect to www.google.fr port 8080: Connection refused
After I tried to setup a proxy in the LibCurl connection. But it didn't change anything.
CURL *curl;
char curl_errbuf[CURL_ERROR_SIZE];
TidyDoc tdoc;
TidyBuffer docbuf = {0};
TidyBuffer tidy_errbuf = {0};
int err;
curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, "https://www.google.fr:8080");
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errbuf);
//curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
tdoc = tidyCreate();
tidyOptSetBool(tdoc, TidyForceOutput, yes);
tidyOptSetInt(tdoc, TidyWrapLen, 4096);
tidySetErrorBuffer(tdoc, &tidy_errbuf);
tidyBufInit(&docbuf);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &docbuf);
curl_easy_setopt(curl, CURLOPT_PROXY, "http://212.129.52.155:8080");
err = curl_easy_perform(curl);
I would like make the connection work on the Wifi of my School.
EDIT 1 :
Following the advise in this post : Can servers block curl requests?
I exacted the headers from Chrome and added it into LibCurl.
struct curl_slist *list = NULL;
list = curl_slist_append(list, "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
list = curl_slist_append(list, "accept-encoding: gzip, deflate, br");
list = curl_slist_append(list, "accept-language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7");
list = curl_slist_append(list, "cache-control: max-age=0");
list = curl_slist_append(list, "upgrade-insecure-requests: 1");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list);
I enable the UserAgent settings :
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36");
And I added cookies :
curl_easy_setopt(curl, CURLOPT_COOKIESESSION, 1L);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt");
But after all the connection still failed.
If your connection is fine, this error is probably due to a proxy server: is your school using a proxy?
If so, you need to explicitely tell curl to make this request through your proxy. This can be done with
curl_easy_setopt(curl, CURLOPT_PROXY, "http://proxy:80");
You can find more informations on proxy there: https://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html
Alternatively, you can try to use this in command line with curl -x http://proxy_server:proxy_port --proxy-user username:password -L http://url
I am trying to interface with Amazon Cloud Backup RESTful API. And in order to remove a file on server side, I need to use a PUT request.
Below shell scripts works well:
curl -v -X PUT https://cdws.us-east-1.amazonaws.com/drive/v1/trash/161kKftTTTuRee1hicOhAw --header "Authorization: Bearer Atza|IQEBLjAsAhQ5zx7pKp9PCgCy6T1JkQjHHOEzpwIUQM"
But now I want to realize it using libcURL. And below is my code:
int curlRequest(char *url, char *header){
CURL *curl;
CURLcode res;
struct curl_slist *chunk=NULL;
curl_global_init(CURL_GLOBAL_DEFAULT);
curl=curl_easy_init();
if(curl){
chunk=curl_slist_append(chunk, header);
curl_easy_setopt(curl, CURLOPT_HTTPHEAD, chunk);
curl_easy_setopt(curl, CURLOPT_PUT, 1L);
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res=curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_slist_free_all(chunk);
}
Return 0;
}
Then I got a reply as follow and the program keep waiting:
< HTTP/1.1 100 Continue
I know that if I upload file using PUT method, I need to use CURLOPT_READFUNCTION. But actually I do not need to upload any data. How can I fix this issue?
I had this same issue. You need to use this:
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
instead of:
curl_easy_setopt(curl, CURLOPT_PUT, 1L);
I am crawling a page by libcurl. I need to use specific IP to get page. this ip has been made by the DNS resolver. So I can skip the getaddrinfo in libcurl and cost less time.
I have asked a question How can I use libcurl function "curl_easy_setopt(CURL *handle, CURLOPT_DNS_LOCAL_IP4, char *address);" but I found this is not what I want.
You can "pre-populate" libcurl's DNS cache with CURLOPT_RESOLVE, and then you can keep using the host name in the URL just like normal.
Here's a little sample telling curl example.com is at 127.0.0.1
CURL *curl;
struct curl_slist *host = NULL;
host = curl_slist_append(NULL, "example.com:80:127.0.0.1");
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_RESOLVE, host);
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
}
curl_slist_free_all(host);
Another option is to use the correct IP in the URL and send a custom Host: header that includes the correct host name.
(CURLOPT_DNS_LOCAL_IP4 sets "the local IPv4 address that the resolver should bind to" and is thus a completely different functionality)
Hello what I am trying to do is send post method twice, however when I send it a second time the information from the first time is also being included and I do not want that.
To illustrate what I mean, this is the code that sends using post method. (the handle curl was already created)
void process(char* transferBuffer) {
curl_easy_setopt(curl, CURLOPT_URL, "http://localhost/cpp.php");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, transferBuffer);
res = curl_easy_perform(curl);
if (res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
If I do something like:
process("name=John"); - webserver receives name=John
process("name=El"); - webserver receives name=John AND name=El
What I want to do is somehow clean previously used data;
the curl handle was already created ... What I want to do is somehow clean previously used data
All I can say is that if you want to reuse your curl handle - which is a best practice, you should reset it with curl_easy_reset before re-setting your options and re-performing the transfer.
Note that without the complete sample code (including the creation of your curl handle, etc) it is quite hard to provide a detailed answer.