Use of both easy and multi interfaces in libcurl for FTP - c

Question

I'm writing a program that continuously and recursively checks an FTP server for new files. When a file is detected, it is downloaded.
I wrote the all thing using the curl easy interface, since blocking calls to curl_easy_perform() are great for the control channel and listing operations. But when it comes to download files, the multi interface seems a lot more appropriate. I thought about switching the entire thing to multi, but it gets very complicated for directory listing.
So here's my question, can I use both interfaces, easy and multi inside the same thread ? If so, can they share the same connection to the server ?
EDIT 1
Instead of using curl_easy_perform(), is there a way to check for a single transfer status ? So I could use the curl_multi_* interface for all my transfers, and only check my LIST command status right after I perform it. This would allow me to simulate a blocking behavior, without interfering with my file transfers that would be handled and checked elsewhere.
From what I saw, the curl_multi_info_read() doesn't allow to do so :
When you fetch a message using this function, it is removed from the internal queue so calling this function again will not return the same message again.

Answer 1

Does this answer your question:
When an easy handle is setup and ready for transfer, then instead of using curl_easy_perform like when using the easy interface for transfers, you should add the easy handle to the multi handle with curl_multi_add_handle. You can add more easy handles to a multi handle at any point, even if other transfers are already running.
From libcurl - multi interface overview (ONE MULTI HANDLE MANY EASY HANDLES)

Answer 2

can I use both interfaces, easy and multi inside the same thread ?
yes absolutely, but note that the easy api is mostly blocking, and the multi api is mostly non-blocking, so if you combine them wrong, you might end up in a situation where your multi transfers are hanging/slow because your thread is blocking on a curl_easy~ call.
If so, can they share the same connection to the server ?
strictly speaking, yes, at least in some situations, but you really should let libcurl worry about connection-reuse details, unless you're in a micro-optimization phase (and given your questions, you're absolutely not)
is there a way to check for a single transfer status
check status of a single transfer from a curl_multi list of transfers?
idk to be honest, when i use curl_multi, i usually only check up on them when they're no longer active as reported by curl_multi_info_read() & co.. you could wrap each transfer in its own object with its own dedicated download thread, and keep track of each transfer with CURLOPT_WRITEFUNCTION & co,
this program will output
transfer #1 is 4.70178% downloaded. running: true
transfer #2 is 6.51742% downloaded. running: true
transfer #3 is 6.14288% downloaded. running: true
transfer #4 is 6.01199% downloaded. running: true
transfer #0 is 12.3027% downloaded. running: true
transfer #1 is 8.73407% downloaded. running: true
transfer #2 is 14.0515% downloaded. running: true
transfer #3 is 12.8638% downloaded. running: true
transfer #4 is 11.8516% downloaded. running: true
(...)
transfer #0 is 94.8156% downloaded. running: true
transfer #1 is 88.5291% downloaded. running: true
transfer #2 is 98.8117% downloaded. running: true
transfer #3 is 92.01% downloaded. running: true
transfer #4 is 100% downloaded. running: false
transfer #0 is 100% downloaded. running: false
transfer #1 is 100% downloaded. running: false
transfer #2 is 100% downloaded. running: false
transfer #3 is 100% downloaded. running: false
it keeps track of each individual transfer in its own thread, and the main thread can easily check up on any individual transfers by doing transfers[x]->status ~
#include <iostream>
#include <thread>
#include <string>
#include <string_view>
#include <atomic>
#include <vector>
#include <memory>
#include <curl/curl.h>
class Curl_Transfer
{
public:
std::string url;
std::string response_headers;
std::string response_body;
CURL *ch = nullptr;
CURLcode curl_easy_perform_code = CURLcode(0);
bool running = true;
std::thread dedicated_thread;
int64_t expected_size = 0; // << content-length reported size
Curl_Transfer(std::string url) : url(url)
{
this->dedicated_thread = std::thread([&]() -> void
{
this->ch = curl_easy_init();
curl_easy_setopt(this->ch, CURLOPT_URL, this->url.c_str());
curl_easy_setopt(this->ch, CURLOPT_WRITEDATA,
this);
curl_easy_setopt(this->ch, CURLOPT_HEADERDATA,
this);
curl_easy_setopt(this->ch, CURLOPT_WRITEFUNCTION, this->WRITEFUNCTION_cb);
curl_easy_setopt(this->ch, CURLOPT_HEADERFUNCTION, this->HEADERFUNCTION_cb);
CURLcode code=curl_easy_perform(this->ch);
//std::cout << "code: " << code << std::endl;
this->curl_easy_perform_code = code;
this->running = false;
});
}
~Curl_Transfer()
{
std::cout << "DESTRUCTING!" << std::endl;
this->dedicated_thread.join();
curl_easy_cleanup(this->ch);
}
private:
// this function need to be static to be compatible with some C->C++ calling stuff... idk, but it also need access to this, so fthis=this...
static size_t WRITEFUNCTION_cb(const char *data, size_t size, size_t nmemb, Curl_Transfer *fthis)
{
CURL *ch = fthis->ch;
fthis->response_body.append(data, size * nmemb);
//std::cout << "got body data! " << size*nmemb << "\n";
return size * nmemb;
};
// this function need to be static to be compatible with some C->C++ calling stuff... idk, but it also need access to this, so fthis=this...
static size_t HEADERFUNCTION_cb(const char *data, size_t size, size_t nmemb, Curl_Transfer *fthis)
{
CURL *ch = fthis->ch;
//std::cout << "got headers! " << size*nmemb << "\n";
fthis->response_headers.append(data, size * nmemb);
std::string_view svd(data, size * nmemb);
const std::string_view needle = "Content-Length: ";
auto clp = svd.find(needle);
if (clp != std::string::npos)
{
svd = svd.substr(needle.size());
std::string fck(svd);
fthis->expected_size = std::stoll(fck, nullptr, 0);
}
return size * nmemb;
};
};
int main()
{
curl_global_init(~0); // << todo get proper constant
std::vector<Curl_Transfer *> transfers;
for (int i = 0; i < 5; ++i)
{
auto fck = new Curl_Transfer("http://speedtest.tele2.net/100MB.zip");
transfers.push_back((fck));
}
for (;;)
{
std::this_thread::sleep_for(std::chrono::seconds(5));
for (size_t i = 0; i < transfers.size(); ++i)
{
std::cout << "transfer #" << i << " is " << (double((transfers[i]->response_body.size()) / double(transfers[i]->expected_size))*100) << "% downloaded. running: " << (transfers[i]->running ? "true" : "false") << "\n";
}
}
}
there's probably a better way to do this though.. there has to be. but until someone smarter comes along, *this works at least...
apparently i did all the threading shit to avoid using the curl_multi api.. dafuq
you're using C, not C++... sorry, ofc you can do all of the above in C as well, but i'm not comfortable enough with C to enjoy re-writing that in C (anyone is free to re-write the code in C if they want to)