I am new using libcurl and I need some help for simple procedures.
I want my program to read information from a website, which will return the information in the following format:
text | xxxxx | xxxxxxxxx
text | xxxxx | xxxxxxxxx
text | xxxxx | xxxxxxxxx
and I want to save that information into an array.
Then read the information from the array, but only the second and third column, just the numbers to use them with another function.
I know how to get the information using libcurl in C from the website and I get it printed in the terminal window, but I do not know how to save it into an array. I am stuck there...
For example with this code:
How would you save the information it returns into an array?
#include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http:www.example.com");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
I would appreciate any help I can get.
Thank you.
Take a look at CURLOPT_WRITEFUNCTION option of curl_easy_setopt(), and follow their examples.
size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata);
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_WRITEFUNCTION, write_callback);
Related
When trying to send an empty array using curl the array get's received as so emptyArr['']
with quote instead of being posted as an empty emptyArr[]
how do i post the emptyArr without the quote?
#include <curl/curl.h>
CURL *curl;
CURLcode res;
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if(curl) {
/* First set the URL that is about to receive our POST. This URL can
just as well be a https:// URL if that is what should receive the
data. */
curl_easy_setopt(curl, CURLOPT_URL, "http://someaddress.com");
/* Now specify the POST data */
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "emptyArr[]");
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
curl_global_cleanup();
Empty arrays don't really exist in URL-encoded parameters. When you send an array, it's sent as:
name[]=firstElement&name[]=secondElement&name[]=thirdElement
An empty array means you don't send any of these, but then there's no parameter at all.
It's the responsibility of the server code to handle the nonexisting parameter and treat it as an empty array.
When you write
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "emptyArr[]");
it's being treated as
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "emptyArr[]=");
so you're creating an array with one element whose value is an empty string.
You should simply leave out the parameter entirely, and the server should treat it as empty.
I'm trying to access an airport RSS feed using the curl library in C. Whenever I try to access it, though, I get access denied error. The following is the code I'm using. It's almost identical to the example code at https://curl.haxx.se/libcurl/c/simple.html:
#include <stdio.h>
#include <curl/curl.h>
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
char *feed_addr = "http://w1.weather.gov/xml/current_obs/KUCP.rss";
//airport not in the state I live
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, feed_addr);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
// Perform the request, res will get the return code
res = curl_easy_perform(curl);
// Check for errors
if(res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",curl_easy_strerror(res));
}
// always cleanup
curl_easy_cleanup(curl);
}
return 0;
}
I've tried this with other websites besides weather.gov (google, youtube, mit.edu) and they all work just fine. But when I tried this, even with other airport RSS feeds (which can be found at http://w1.weather.gov/xml/current_obs/seek.php?state=pa&Find=Find), I get the same access denied error. I also get the same error when I set feed_addr to any weather.gov page.
To make this stranger, when I tried to access the feed using Python3's urllib.request module, it worked just fine. I can also access it just fine with Google Chrome. So I can rule out the idea that it doesn't want me accessing the feed.
Is there something that I'm missing? Is there a way to get the feed via the curl library? Or is there a way using a different library?
Answering my own question (I did not originally plan to, but I solved the issue before someone else answered):
So here is the code that I got to work
#include <stdio.h>
#include <curl/curl.h>
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
char *feed_addr = "http://w1.weather.gov/xml/current_obs/KUCP.rss";
//airport not in the state I live
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, feed_addr);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
//line that solved my issue
curl_easy_setopt(curl, CURLOPT_USERAGENT, <string>);
/**
* The line as written will not work.
* I'm only using <string> as a stand-in for some personal information
* If you're having the same issue as I did, I explain how to solve it under the code
*/
// Perform the request, res will get the return code
res = curl_easy_perform(curl);
// Check for errors
if(res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",curl_easy_strerror(res));
}
// always cleanup
curl_easy_cleanup(curl);
}
return 0;
}
The way I solved this was finding a site that printed out my HTTP request. In this case, it was http://rve.org.uk/dumprequest. I visited the site like normal in one tab, and I used the program to get the source, then launched that in another tab. That was when I saw that several fields were present when I manually opened the page, but absent when I used the code.
So, I looked at curl_easy_setopt to see if there was a way to set those fields. It turns out, curl_easy_setopt has documentation for that and many more at the website https://curl.haxx.se/libcurl/c/curl_easy_setopt.html. At the suggestion of one of the comments, I looked at CURLOPT_USERAGENT first.
Since that one took a char *, I found the line in the HTTP request that started with User-Agent:, copy-and-pasted the rest, and that was <string> in the above line.
So if the request included the line:
User-Agent: Lord Voldemort (Tom Marvolo Riddle)
The line I included would be:
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Lord Voldemort (Tom Marvolo Riddle)");
I am trying to generate curl get request using c program .Here I need to store the response in a variable and I tried with the following code.
#include <stdio.h>
#include <stdlib.h>
#include <curl/curl.h>
function_pt(void *ptr, size_t size, size_t nmemb, void *stream){
char **response_ptr = (char**)stream;
*response_ptr = strndup(ptr, (size_t)(size *nmemb));
}
int main(void)
{
CURL *curl;
CURLcode res;
char *response =calloc(1,sizeof(char));
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, function_pt);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
res=curl_easy_perform(curl);
curl_easy_cleanup(curl);
printf("%s\n",response);
}
return 0;
}
The data I get form http get request is real time,so i need to continue with the get request and store value in a variable in an iterative manner,only so that I can use the data in all other parts of the program.But the following code works only once and then exit.
How can I do it? Are there any other methods to generate http get request?
You don't need to calloc() the pointer if you are going to strndup() the original string, assuming that the response is a string is not good because that is not necessarily true.
I would suggest a structure where you can also store the length of the response, so if it's not text but for example a jpeg file nothing bad will happen, and you should not call printf() unless you check from the response headers that the response is indeed text, and it will be nul terminated afaik.
I am currently writing a REST-like client that only is required to do PUT requests.
Problem:
Running the program is not giving me the correct results on the URL's API and I do not know why.
Using curl_easy_perform(curl) does not throw an error when called. But the expected result is not generated on the URL's API.
Using curl_easy_send(curl,..,..,..) throws a : unsupported protocol error
Assumption:
I am assuming the order in which I am using the curl_easy_opts is a problem? And I am even missing a couple of key lines?
I have been reading on here of how other people do PUT requests and have been using their methods.
Summary of Program:
My program prompts the user for some string/character data, and from that, I construct the strings myself such as the header and the payload. The header and payload are both in JSON format but the payload is simply a string ( in this case, a char *str = (char *)mallo.. etc). How the header is constructed is shown below.
My header is being constructed using
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept: application/json");
//there is more content being appended to the header
The CURL function calls :
//init winsock stuff
curl_global_init(CURL_GLOBAL_ALL);
//get a curl handle
curl = curl_easy_init();
if(curl){
//append the headers
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
//specify the target URL
curl_easy_setopt(curl, CURLOPT_URL, url);
//connect ( //i added this here since curl_easy_send() says it requires it. )
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY,1L);
//specify the request (PUT in our case)
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
//append the payload
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, payload);
res = curl_easy_perform(curl);
//res = curl_easy_send(curl, payload, strlen(payload),&iolen);
//check for errors
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
curl_easy_cleanup(curl);
}
You should not be using the CURLOPT_CONNECT_ONLY option or curl_easy_send() function, those are intended to be used for custom, non-HTTP protocols.
See this page for an example of how to do a PUT request with libcurl. Basically, you want to enable the CURLOPT_UPLOAD and CURLOPT_PUT options to say that you're doing a PUT request and to enable uploading a body with the request, and then you set the CURLOPT_READDATA and CURLOPT_INFILESIZE_LARGE options to tell libcurl how to read the data you're uploading and how big the data is.
In your case, if you already have the data in memory, then you don't need to read it out of a file, and you can just memcpy() it inside your read callback.
Example code copied below:
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2012, Daniel Stenberg, <daniel#haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at http://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <curl/curl.h>
/*
* This example shows a HTTP PUT operation. PUTs a file given as a command
* line argument to the URL also given on the command line.
*
* This example also uses its own read callback.
*
* Here's an article on how to setup a PUT handler for Apache:
* http://www.apacheweek.com/features/put
*/
static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
{
size_t retcode;
curl_off_t nread;
/* in real-world cases, this would probably get this data differently
as this fread() stuff is exactly what the library already would do
by default internally */
retcode = fread(ptr, size, nmemb, stream);
nread = (curl_off_t)retcode;
fprintf(stderr, "*** We read %" CURL_FORMAT_CURL_OFF_T
" bytes from file\n", nread);
return retcode;
}
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
FILE * hd_src ;
struct stat file_info;
char *file;
char *url;
if(argc < 3)
return 1;
file= argv[1];
url = argv[2];
/* get the file size of the local file */
stat(file, &file_info);
/* get a FILE * of the same file, could also be made with
fdopen() from the previous descriptor, but hey this is just
an example! */
hd_src = fopen(file, "rb");
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if(curl) {
/* we want to use our own read function */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
/* enable uploading */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* HTTP PUT please */
curl_easy_setopt(curl, CURLOPT_PUT, 1L);
/* specify target URL, and note that this URL should include a file
name, not only a directory */
curl_easy_setopt(curl, CURLOPT_URL, url);
/* now specify which file to upload */
curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);
/* provide the size of the upload, we specicially typecast the value
to curl_off_t since we must be sure to use the correct data size */
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
(curl_off_t)file_info.st_size);
/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
fclose(hd_src); /* close the local file */
curl_global_cleanup();
return 0;
}
I agree, don't use CUSTOMREQUEST. One detail that is being missed on every related to PUT and CURL I've seen here is that you NEED to set the file size, otherwise you'll get HTTP error 411.
Use CURLOPT_INFILESIZE or CURLOPT_INFILESIZE_LARGE for that.
See more details here:
How do I send long PUT data in libcurl without using file pointers?
I know this is a very old question, but in case somebody want to use libcurl with GLib and json-glib to send a JSON with PUT request.
Code below works for me:
#include <curl/curl.h>
#include <json-glib/json-glib.h>
//this is callback function for CURLOPT_READFUNCTION:
static size_t
curlPutJson ( void *ptr, size_t size, size_t nmemb, void *_putData )
{
GString *putData = ( GString * ) _putData;
size_t realsize = ( size_t ) putData->len;
memcpy ( ptr, putData->str, realsize );
return realsize;
}
/*now inside main or other function*/
//json_to_string ( jsonNode, FALSE ) is from json-glib to stringify JSON
//created in jsonNode
GString *putData = g_string_new ( json_to_string ( mainNode, FALSE ) );
//now goes curl as usual: headers, url, other options and so on
//and 4 most important lines
curl_easy_setopt ( curl, CURLOPT_READFUNCTION, curlPutJson );
curl_easy_setopt ( curl, CURLOPT_UPLOAD, 1L );
curl_easy_setopt ( curl, CURLOPT_READDATA, putData ); //GString
curl_easy_setopt ( curl, CURLOPT_INFILESIZE, putData->len ); //type long
Im Finding the cURL Documentation to read/Understand. what im trying to do here is post a postcode and return whether or not it is valid.
This is my function so far (what i managed to pull together from tutorials)
int checkPostCode(char postCode[5])
{
printf("checkPostCode\n");
char uidPath[200] = "http://zim.cs.uow.edu.au/~dfs/cgi-bin/postcodes.cgi?pcode=";
//Concatinate poth strings
strcat (uidPath,postCode);
//Output URl
printf("%s \n ", uidPath);
CURL *curl;
CURLcode res;
// Create our curl handle
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_URL, uidPath);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
the script returns either a "Yes" or a "No".
I just don't know what to enter into that if statement to get my result.
thanks in advance :).
It sounds like you want to read a response from the server. For that you'll need to enable the option CURLOPT_READFUNCTION and specify a callback that will receive the data from the network and store it in a array of char.