How can I hide curl_easy_perform output (in a shell)?
This is in regards to a C application.
Use CURLOPT_NOBODY in curl_easy_setopt().
Example:
...
CURL *curl;
CURLcode statusCode;
curl = curl_easy_init();
if(curl){
curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com/");
//CURLOPT_NOBODY does the trick
curl_easy_setopt(curl, CURLOPT_NOBODY, 1);
curl_easy_perform(curl);
...
Link to docs: http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTNOBODY
Set the CURLOPT_WRITEFUNCTION and/or CURLOPT_WRITEDATA options:
FILE *f = fopen("target.txt", "wb");
curl_easy_setopt(handle, CURLOPT_WRITEDATA, f);
By default, libcurl writes output to stdout. When you override this (which is what almost any application will do), it will write to another file or to pass chunks of output to a callback. See the documentation for CURLOPT_WRITEFUNCTION for more details.
As Joey said, CURLOPT_WRITEFUNCTION will allow you to completely disregard all output. Just set up a callback that does absolutely nothing if you want the data to just go away, without being written to any file descriptor.
For instance,
/* Never writes anything, just returns the size presented */
size_t my_dummy_write(char *ptr, size_t size, size_t nmemb, void *userdata)
{
return size * nmemb;
}
Then in your options:
curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, &my_dummy_write);
Or, point the file handle at a NULL device (a lot easier).
Related
I'm trying to develop an application in C with GTK for the GUI, libcurl to get data by a web API and cJSON to parse JSON.
My problem is that when I do my request with libcurl after gtk_init, the data that I get can not be parsing in JSON. Otherwise, if I get the data and parse it before gtk_init, the parsing works fine.
I have an example to demonstrate it, the first printf in the main returns the JSON correctly but the second printf which is after gtk_init returns NULL (to be exact, the parsing stops at the first decimal number and fail) :
initString :
void initString(String * s) {
s->len = 0;
s->ptr = malloc(s->len + 1);
if (s->ptr == NULL) {
fprintf(stderr, "malloc() failed\n");
exit(EXIT_FAILURE);
}
s->ptr[0] = '\0';
}
writeFunc :
size_t writeFunc(void *ptr, size_t size, size_t nmemb, String * s)
{
size_t newLen = s->len + size*nmemb;
s->ptr = realloc(s->ptr, newLen + 1);
if (s->ptr == NULL) {
fprintf(stderr, "realloc() failed\n");
exit(EXIT_FAILURE);
}
memcpy(s->ptr + s->len, ptr, size * nmemb);
s->ptr[newLen] = '\0';
s->len = newLen;
return size * nmemb;
}
My function to get data with libcurl :
char * getData(gpointer user_data)
{
curl_global_init(CURL_GLOBAL_ALL);
CURL * curl;
char * url = user_data;
CURLcode res;
String s;
struct curl_slist * headers = NULL;
curl = curl_easy_init();
if(curl)
{
initString(&s);
headers = curl_slist_append(headers, "Accept: application/json");
headers = curl_slist_append(headers, "Content-Type: application/json");
headers = curl_slist_append(headers, "charsets: utf-8");
//curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_easy_setopt(curl, CURLOPT_SSLVERSION, 6);
curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0");
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
//write data in a string
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeFunc);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
res = curl_easy_perform(curl);
}
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
curl_easy_cleanup(curl);
return s.ptr;
}
The main :
int main(int argc, char ** argv)
{
char * str;
cJSON * json;
str = getData("https://data.culture.gouv.fr/api/records/1.0/search/?dataset=liste-et-localisation-des-musees-de-france&facet=ville&sort=ville&facet=nomdep&refine.nomdep=AIN&rows=1");
json = cJSON_Parse(str);
printf("DATA : %s\n", cJSON_Print(json));//returns the JSON perfectly
gtk_init(&argc,&argv);
str = getData("https://data.culture.gouv.fr/api/records/1.0/search/?dataset=liste-et-localisation-des-musees-de-france&facet=ville&sort=ville&facet=nomdep&refine.nomdep=AIN&rows=1");
json = cJSON_Parse(str);
printf("ERROR : %s\n", cJSON_GetErrorPtr());//returns half of data
printf("DATA : %s\n", cJSON_Print(json));//returns NULL
gtk_main();
curl_global_cleanup();
return EXIT_SUCCESS;
}
I have tried to resolve this by creating threads with g_idle_add, gdk_threads_idle_add, gdk_threads_entry and gdk_threads_leave, pthread_create and pthread_join but nothing worked.
Does someone know how to resolve this problem ?
Thanks.
You need to narrow down your problem. In other words, you need "M" in MVCE. Currently you have three libraries:
Gtk+
cURL
cJSON
that might interact in arbitrary ways. What you need to check:
Does cURL return different data before and after gtk_init call? If no, then problem is not with cURL.
If you hardcode JSON data in your program and parse it with cJSON, are results different before and after gtk_init? If so, problem is in cJSON. If no, problem is not related to cJSON.
My guess is as follows. gtk_init does several things, amongst them sets locale. Since you fetch data from .fr domain, I assume that you have French locale set on your computer.
to be exact, the parsing stops at the first decimal number and fail
In French, decimal separator is comma , instead of period ., so I suppose that after GTK+ changes locale, cJSON starts to look for , in decimal numbers but it finds . and fails.
Possible solutions
As a workaround, call gtk_disable_setlocale before gtk_init. This might cause unintended consequences, eg. your program starts to display numbers in English format instead of French in its UI.
The real solution would be to raise bug in cJSON, as JSON parsing should not take locale into account when parsing numbers. JSON mandates period . as decimal separator.
I doubt very much that this has anything at all to do with the gtk_init() call. It sounds like memory corruption. This would likely be the case e.g. if you returned a pointer to data that was allocated and then cleaned up by curl. In this case both calls are wrong, you just happened to get away with it the first time as the memory was hanging around untouched in that case. Check the API for function calls you make. Some will allocate memory that needs to be freed, others don't.
I suggest you use build with -g and use gdb to test your code. This will help narrow down where the problem lies. Take a look and see if the strings are identical.
cURL returns the same data before and after, I checked it by putting data in 2 strings and strcmp returns 0.
Indeed, I'm french and there may be a confusion between , and . .
gtk_disable_setlocale solved my problem !
I am trying to generate curl get request using c program .Here I need to store the response in a variable and I tried with the following code.
#include <stdio.h>
#include <stdlib.h>
#include <curl/curl.h>
function_pt(void *ptr, size_t size, size_t nmemb, void *stream){
char **response_ptr = (char**)stream;
*response_ptr = strndup(ptr, (size_t)(size *nmemb));
}
int main(void)
{
CURL *curl;
CURLcode res;
char *response =calloc(1,sizeof(char));
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, function_pt);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
res=curl_easy_perform(curl);
curl_easy_cleanup(curl);
printf("%s\n",response);
}
return 0;
}
The data I get form http get request is real time,so i need to continue with the get request and store value in a variable in an iterative manner,only so that I can use the data in all other parts of the program.But the following code works only once and then exit.
How can I do it? Are there any other methods to generate http get request?
You don't need to calloc() the pointer if you are going to strndup() the original string, assuming that the response is a string is not good because that is not necessarily true.
I would suggest a structure where you can also store the length of the response, so if it's not text but for example a jpeg file nothing bad will happen, and you should not call printf() unless you check from the response headers that the response is indeed text, and it will be nul terminated afaik.
I am new using libcurl and I need some help for simple procedures.
I want my program to read information from a website, which will return the information in the following format:
text | xxxxx | xxxxxxxxx
text | xxxxx | xxxxxxxxx
text | xxxxx | xxxxxxxxx
and I want to save that information into an array.
Then read the information from the array, but only the second and third column, just the numbers to use them with another function.
I know how to get the information using libcurl in C from the website and I get it printed in the terminal window, but I do not know how to save it into an array. I am stuck there...
For example with this code:
How would you save the information it returns into an array?
#include <stdio.h>
#include <curl/curl.h>
int main(void)
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http:www.example.com");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
I would appreciate any help I can get.
Thank you.
Take a look at CURLOPT_WRITEFUNCTION option of curl_easy_setopt(), and follow their examples.
size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata);
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_WRITEFUNCTION, write_callback);
I am currently writing a REST-like client that only is required to do PUT requests.
Problem:
Running the program is not giving me the correct results on the URL's API and I do not know why.
Using curl_easy_perform(curl) does not throw an error when called. But the expected result is not generated on the URL's API.
Using curl_easy_send(curl,..,..,..) throws a : unsupported protocol error
Assumption:
I am assuming the order in which I am using the curl_easy_opts is a problem? And I am even missing a couple of key lines?
I have been reading on here of how other people do PUT requests and have been using their methods.
Summary of Program:
My program prompts the user for some string/character data, and from that, I construct the strings myself such as the header and the payload. The header and payload are both in JSON format but the payload is simply a string ( in this case, a char *str = (char *)mallo.. etc). How the header is constructed is shown below.
My header is being constructed using
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept: application/json");
//there is more content being appended to the header
The CURL function calls :
//init winsock stuff
curl_global_init(CURL_GLOBAL_ALL);
//get a curl handle
curl = curl_easy_init();
if(curl){
//append the headers
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
//specify the target URL
curl_easy_setopt(curl, CURLOPT_URL, url);
//connect ( //i added this here since curl_easy_send() says it requires it. )
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY,1L);
//specify the request (PUT in our case)
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
//append the payload
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, payload);
res = curl_easy_perform(curl);
//res = curl_easy_send(curl, payload, strlen(payload),&iolen);
//check for errors
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
curl_easy_cleanup(curl);
}
You should not be using the CURLOPT_CONNECT_ONLY option or curl_easy_send() function, those are intended to be used for custom, non-HTTP protocols.
See this page for an example of how to do a PUT request with libcurl. Basically, you want to enable the CURLOPT_UPLOAD and CURLOPT_PUT options to say that you're doing a PUT request and to enable uploading a body with the request, and then you set the CURLOPT_READDATA and CURLOPT_INFILESIZE_LARGE options to tell libcurl how to read the data you're uploading and how big the data is.
In your case, if you already have the data in memory, then you don't need to read it out of a file, and you can just memcpy() it inside your read callback.
Example code copied below:
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2012, Daniel Stenberg, <daniel#haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at http://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <curl/curl.h>
/*
* This example shows a HTTP PUT operation. PUTs a file given as a command
* line argument to the URL also given on the command line.
*
* This example also uses its own read callback.
*
* Here's an article on how to setup a PUT handler for Apache:
* http://www.apacheweek.com/features/put
*/
static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
{
size_t retcode;
curl_off_t nread;
/* in real-world cases, this would probably get this data differently
as this fread() stuff is exactly what the library already would do
by default internally */
retcode = fread(ptr, size, nmemb, stream);
nread = (curl_off_t)retcode;
fprintf(stderr, "*** We read %" CURL_FORMAT_CURL_OFF_T
" bytes from file\n", nread);
return retcode;
}
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
FILE * hd_src ;
struct stat file_info;
char *file;
char *url;
if(argc < 3)
return 1;
file= argv[1];
url = argv[2];
/* get the file size of the local file */
stat(file, &file_info);
/* get a FILE * of the same file, could also be made with
fdopen() from the previous descriptor, but hey this is just
an example! */
hd_src = fopen(file, "rb");
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if(curl) {
/* we want to use our own read function */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
/* enable uploading */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* HTTP PUT please */
curl_easy_setopt(curl, CURLOPT_PUT, 1L);
/* specify target URL, and note that this URL should include a file
name, not only a directory */
curl_easy_setopt(curl, CURLOPT_URL, url);
/* now specify which file to upload */
curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);
/* provide the size of the upload, we specicially typecast the value
to curl_off_t since we must be sure to use the correct data size */
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
(curl_off_t)file_info.st_size);
/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
fclose(hd_src); /* close the local file */
curl_global_cleanup();
return 0;
}
I agree, don't use CUSTOMREQUEST. One detail that is being missed on every related to PUT and CURL I've seen here is that you NEED to set the file size, otherwise you'll get HTTP error 411.
Use CURLOPT_INFILESIZE or CURLOPT_INFILESIZE_LARGE for that.
See more details here:
How do I send long PUT data in libcurl without using file pointers?
I know this is a very old question, but in case somebody want to use libcurl with GLib and json-glib to send a JSON with PUT request.
Code below works for me:
#include <curl/curl.h>
#include <json-glib/json-glib.h>
//this is callback function for CURLOPT_READFUNCTION:
static size_t
curlPutJson ( void *ptr, size_t size, size_t nmemb, void *_putData )
{
GString *putData = ( GString * ) _putData;
size_t realsize = ( size_t ) putData->len;
memcpy ( ptr, putData->str, realsize );
return realsize;
}
/*now inside main or other function*/
//json_to_string ( jsonNode, FALSE ) is from json-glib to stringify JSON
//created in jsonNode
GString *putData = g_string_new ( json_to_string ( mainNode, FALSE ) );
//now goes curl as usual: headers, url, other options and so on
//and 4 most important lines
curl_easy_setopt ( curl, CURLOPT_READFUNCTION, curlPutJson );
curl_easy_setopt ( curl, CURLOPT_UPLOAD, 1L );
curl_easy_setopt ( curl, CURLOPT_READDATA, putData ); //GString
curl_easy_setopt ( curl, CURLOPT_INFILESIZE, putData->len ); //type long
I am trying to download file using FTP, and in between if connection terminates then it should resume from where it was stop. My problem is that using following code snippet I able to continue download if I close connection and then again connect it, but if I am do so at server site then I am not able to resume download, and program goes into infinites state.
#include <stdio.h>
#include <curl/curl.h>
/*
* This is an example showing how to get a single file from an FTP server.
* It delays the actual destination file creation until the first write
* callback so that it won't create an empty file in case the remote file
* doesn't exist or something else fails.
*/
struct FtpFile {
const char *filename;
FILE *stream;
};
static size_t my_fwrite(void *buffer, size_t size, size_t nmemb, void *stream)
{
struct FtpFile *out=(struct FtpFile *)stream;
if(out && !out->stream) {
/* open file for writing */
out->stream=fopen(out->filename, "wb");
if(!out->stream)
return -1; /* failure, can't open file to write */
}
return fwrite(buffer, size, nmemb, out->stream);
}
int main(void)
{
CURL *curl;
CURLcode res;
struct FtpFile ftpfile={
"dev.zip", /* name to store the file as if succesful */
NULL
};
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if(curl) {
/*
* You better replace the URL with one that works!
*/
curl_easy_setopt(curl, CURLOPT_URL,
"ftp://root:password#192.168.10.1/dev.zip");
/* Define our callback to get called when there's data to be written */
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, my_fwrite);
/* Set a pointer to our struct to pass to the callback */
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &ftpfile);
/* Switch on full protocol/debug output */
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
if(CURLE_OK != res) {
/* we failed */
fprintf(stderr, "curl told us %d\n", res);
}
}
if(ftpfile.stream)
fclose(ftpfile.stream); /* close the local file */
curl_global_cleanup();
return 0;
}
Could any one tell me that how can I resume download if connection is closed by remote site.
Any help would be appreciated
Thanks,
Yuvi
Add a varaible to the ftpfile struct to aware your write function of the need to appeand and tell libcurl to resume the download from the end of the destination file by setting CURLOPT_RESUME_FROM to the number of bytes downloaded already:
struct FtpFile
{
const char *pcInfFil;
FILE *pFd;
int iAppend;
};
In main, if you want to resume:
curl_easy_setopt(curl, CURLOPT_RESUME_FROM , numberOfBytesToSkip);
If you the file doesn't already exist, or it is not a resumed download but a new download, be sure to set CURLOPT_RESUME_FROM back to 0.
In my_fwrite:
out->stream=fopen(out->filename, out->iAppend ? "ab":"wb");
P.S. if where you need to resume the file is larger than a long (2GB), look into CURLOPT_RESUME_FROM_LARGE and CURL_OFF_T_C()
In response to comment requesting additional information on how to know when a transfer failed:
After you call curl easy perform call:
CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ... );
Retrieve from the curl context:
CURLINFO_HEADER_SIZE
CURLINFO_CONTENT_LENGTH_DOWNLOAD
Add them together and make sure they equal
CURLINFO_SIZE_DOWNLOAD
If not try to reperform the context.
And be sure to use the latest version of curl, it should timeout after 60 seconds of not hearing from the FTP server it is downloading from.
You may use the CURLOPT_CONNECTTIMEOUT and CURLOPT_TIMEOUT parameters to specify a connect timeout and maximum execution time on a per-handle basis.
The other way (that only works if you're using the easy interface, not the multi one) is to use a socket option callback, that you can set using CURLOPT_SOCKOPTFUNCTION. In it, you must call setsockopt() for the SO_RCVTIMEO parameter to the maximum amount of time a connection can be idle before it should be dropped. i.e. if no bytes have been received in the last 5 seconds, then drop the connection.