SSL connection fails when stalled every ~5s - c

Question

I have a Perl streaming application that GET some file through HTTPS (for RadioParadise). The TCP flow is often paused for a few seconds because the player buffers a lot. As a result, the connection very often breaks/ends early.
I've taken various TCP logs and wireshark seems to be all confused and whines about Changing Cipher Spec and Ignored Unknown records. Obviously, the Change Cipher packets are wrong and do not match re-handshaking, there is something funny in the packets. What the logs surely shows is in the case of early termination, the server is closing the TCP connection (proper FIN frame)
I've written a small Perl example that throttles the download by using varous length pauses to show what happens. If the download is not paused by more than 1 sec, then this does not happen. When it is paused by 5s, it always happens at some point of the download. Server's openSSL is 1.0.2k and I've tried client 1.1.0j and various version from 0.9.8 to 1.0.2. I've tried also various Perl versions, no change
(it's just an example that actually shows the problem, not supposed to be proper code)
I've also written the same application in C to eliminate Perl as a potential problem and got exactly the same result.
I've also tried other HTTPS server and they do not have the same problem, it seems to be related to RP's. Maybe somebody would have an idea of what I should look at for further investigation as currently, I don't know in what direction to continue. Some general advice on why pausing for 5 sec would make any difference at TCP or SSL level (it's not a SO_KEEPALIVE issue).
NB: there is no packet lost by kernel in my tcpdump
use IO::Socket::SSL;
use Net::SSLeay;
use IO::Select;
$IO::Socket::SSL::DEBUG = 3;
$Net::SSLeay::trace = 3;
my $sock = IO::Socket::SSL->new(
Timeout => 15,
PeerAddr => '23.29.117.2',
PeerPort => 443,
SSL_startHandshake => 1,
SSL_verify_mode => Net::SSLeay::VERIFY_NONE() # SSL_VERIFY_NONE isn't recognized on some platforms?!?, and 0x00 isn't always "right"
) or do {
print("Couldn't create socket binding to $!");
return undef;
};
my $sel = IO::Select->new($sock);
my $start = time;
use constant SLEEP => 5;
syswrite ($sock, "GET /blocks/chan/0/4/1835801-1835807.flac HTTP/1.0\r\n\r\n");
sysread( $sock, my $data, 512 );
print $data;
$sock->blocking(0);
my $total;
my $sum;
while (1) {
my $bytes = sysread( $sock, my $data, 8192 );
next if !defined $bytes;
last if !$bytes;
$total += $bytes;
$sum += $bytes;
print "$total ", int ($total/(time() - $start)/1000), " kB/s \r" if time() - $start;
if ($sum > SLEEP*1024*1024/8) {
sleep(SLEEP);
$sum = 0;
}
}
print("\nfinished $total\n");
$sock->close();