Getting 'access denied' when trying to access an airport RSS feed - c

Question

I'm trying to access an airport RSS feed using the curl library in C. Whenever I try to access it, though, I get access denied error. The following is the code I'm using. It's almost identical to the example code at https://curl.haxx.se/libcurl/c/simple.html:
#include <stdio.h>
#include <curl/curl.h>
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
char *feed_addr = "http://w1.weather.gov/xml/current_obs/KUCP.rss";
//airport not in the state I live
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, feed_addr);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
// Perform the request, res will get the return code
res = curl_easy_perform(curl);
// Check for errors
if(res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",curl_easy_strerror(res));
}
// always cleanup
curl_easy_cleanup(curl);
}
return 0;
}
I've tried this with other websites besides weather.gov (google, youtube, mit.edu) and they all work just fine. But when I tried this, even with other airport RSS feeds (which can be found at http://w1.weather.gov/xml/current_obs/seek.php?state=pa&Find=Find), I get the same access denied error. I also get the same error when I set feed_addr to any weather.gov page.
To make this stranger, when I tried to access the feed using Python3's urllib.request module, it worked just fine. I can also access it just fine with Google Chrome. So I can rule out the idea that it doesn't want me accessing the feed.
Is there something that I'm missing? Is there a way to get the feed via the curl library? Or is there a way using a different library?

Answer 1

Answering my own question (I did not originally plan to, but I solved the issue before someone else answered):
So here is the code that I got to work
#include <stdio.h>
#include <curl/curl.h>
int main(int argc, char **argv)
{
CURL *curl;
CURLcode res;
char *feed_addr = "http://w1.weather.gov/xml/current_obs/KUCP.rss";
//airport not in the state I live
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, feed_addr);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
//line that solved my issue
curl_easy_setopt(curl, CURLOPT_USERAGENT, <string>);
/**
* The line as written will not work.
* I'm only using <string> as a stand-in for some personal information
* If you're having the same issue as I did, I explain how to solve it under the code
*/
// Perform the request, res will get the return code
res = curl_easy_perform(curl);
// Check for errors
if(res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n",curl_easy_strerror(res));
}
// always cleanup
curl_easy_cleanup(curl);
}
return 0;
}
The way I solved this was finding a site that printed out my HTTP request. In this case, it was http://rve.org.uk/dumprequest. I visited the site like normal in one tab, and I used the program to get the source, then launched that in another tab. That was when I saw that several fields were present when I manually opened the page, but absent when I used the code.
So, I looked at curl_easy_setopt to see if there was a way to set those fields. It turns out, curl_easy_setopt has documentation for that and many more at the website https://curl.haxx.se/libcurl/c/curl_easy_setopt.html. At the suggestion of one of the comments, I looked at CURLOPT_USERAGENT first.
Since that one took a char *, I found the line in the HTTP request that started with User-Agent:, copy-and-pasted the rest, and that was <string> in the above line.
So if the request included the line:
User-Agent: Lord Voldemort (Tom Marvolo Riddle)
The line I included would be:
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Lord Voldemort (Tom Marvolo Riddle)");