Perl: Getting UTF-8 file with Mechanize content_file - file

Question

Creating a perl script (Strawberry Perl v5.32.0 on W10) to download my Google Calendar. Google provides a 'private' (no login required) url to a 'basic.ics' file.
When opening this url in my browser (firefox), a window pops up to download this 'basic.ics' file. When saved, the file is UTF-8 encoded.
In my script I'm using WWW::Mechanize get ":content_file" to download the file:
#!/usr/bin/perl -w
use WWW::Mechanize;
# URL modified for obvious reasons ...
my $url = 'https://calendar.google.com/path-to-private-calendar-file/basic.ics';
my $local_file_name = 'Calendar.ics';
my $mech = WWW::Mechanize->new();
$mech->get( $url, ":content_file" => $local_file_name );
The file received with $mech->get however, is written ANSI-encoded and contains 'gibberich' (not 'translated' UTF-8 data, I suppose).
How can I make the get :content_file create the local file UTF-8 encoded ?
Or do I just download the file as is and convert it to UTF-8 later ?
If so please point me in the right direction, because reading the ANSI-encoded file as UTF-8 does not do the trick ...

Answer 1

This has nothing to do with UTF-8 or character encodings. You are getting a gzip-compressed response.
If Compress::Zlib is available, WWW::Mechanize provides the following header by default:
Accept-Encoding: gzip
This allows the remote end to compress the response. If it does, the remote end will provide the following header in the response:
Content-Encoding: gzip
This is happening here, and you are saving the compressed response.
You could use :content_cb instead of :content_file to provide a callback that decompresses the data and stores it. Or you could simply request an uncompressed version by providing the following header:
Accept-Encoding: identity
This is done using
$mech->get($url,
Accept_Encoding => 'identity',
":content_file" => $local_file_name,
);
If you don't otherwise need the overhead of WWW::Mechanize, why not use its base class LWP::UserAgent. It doesn't provide an Accept-Encoding by default, so the server is unlikely to gzip the response
my $ua = LWP::UserAgent->new();
# Will work 99.999% of the time.
$ua->get($url, ":content_file" => $local_file_name);
# Definitely works.
$ua->get($url,
Accept_Encoding => 'identity',
":content_file" => $local_file_name,
);

Answer 2

Sorry for my late reply, but I had some other items to handle.
Thanks to ikegami and Hakon for your replies and solutions.
From your replies, I have distilled 4 methods that get a readable UTF-8 encoded non-zipped result file:
Mechanize + Content_file and encoding
UserAgent + Content_file, no encoding
UserAgent + Content_file and encoding
UserAgent + print content to file
In the last case, however, there was an extra CR at the end of each line (CRCRLF instead of CRLF), but nothing a little regex can't solve ...
Here is the result of my tests:
CASE 1: Mechanize + Content_file and encoding ...
Get: $mech1->get($url, Accept_Encoding => 'identity', ":content_file" => $fn1)
Calendar1.ics received, size = 78564 bytes
CASE 2: UserAgent + Content_file, no encoding ...
Get: $ua2->get($url, ":content_file" => $fn2);
Calendar2.ics received, size = 78564 bytes
CASE 3: UserAgent + Content_file and encoding ...
Get: $ua3->get($url, Accept_Encoding => 'identity', ":content_file" => $fn3);
Calendar3.ics received, size = 78564 bytes
CASE 4: UserAgent + print content to file ...
Get: my $res4 = $ua4->get($url);
...
my $content = $res4->content;
$content =~ s/\r[\n]*/\n/gm;
print $fh $content;
...
Calendar4.ics received, size = 78564 bytes

Answer 3

Yes, I also get some sort of encrypted file when using WWW::Mechanize, however LWP::UserAgent works fine for me:
use feature qw(say);
use strict;
use warnings;
use LWP::UserAgent;
my $fn = 'Calendar.ics';
my $url = 'https://calendar.google.com/calendar/XXXXXXXX/basic.ics';
my $ua = LWP::UserAgent->new();
my $res = $ua->get( $url );
if ($res->is_success) {
say "Saving file: '$fn'";
open ( my $fh, '>', $fn ) or die "Could not open file '$fn': $!";
print $fh $res->content;
close $fh;
}
else {
die $res->status_line;
}