Why looses this Armbian repeatedly connectivity? - armbian

Question

I have an Olimex Lime2 running an Armbian, headless. On this board I only care for SSH and MiniDLNA. I hope to get around to include the whole configuration, but one important bit might be that in /boot/armbianEnv.txt I put
extraargs=acpi=off
For one year now I experience very hard to debug issues with availability. The machine randomly stops to be accessible via ping or ssh. The issues are hard to debug because they seem to disappear when connecting monitor or keyboard while I can't find any trace of them when the system runs headless. While I got the problem mostly under control without knowing how, the Olimex still stops to respond now and then. This time I want to ask why.
I noticed that the Olimex stopped to provide DLNA access at 10/25th, ~2pm. I did not touch it to see if it recovers (which happens sometimes). This time the system remained unreachable for 2 days until I unplugged power.
Below you can find links to two logs. I would be very glad if anything suspicious in them could be pointed out so I can start resolving them.
One particular thing I wonder: Why did the system decided to reboot? There was no power outage at that day. I expect that a normal reboot would manifest itself in the logs, do they?
The logs:
/var/logs/messages: https://pastebin.com/qgRumreB
/var/logs/syslog: https://pastebin.com/U5jpHNHm
The logs are complete. I only removed lines in the beginning and at the end, but not in between.

Answer 1

Albeit I did not find a clear solution, I want to share what I have tried. Hopefully this will provide inspiration for others with similar problems.
One thing helping me a lot (which wasn't available back then) was to switch to a newer OS. I am running Armbian based on Ubuntu 18.04 and the logs are less cluttered now. Also, some minor details improved. If you landed here and still run Armbian Stretch, you should upgrade.
One reason for frequent crashes could be a corrupt file system, caused by frequent crashes :-(. Armbian mounts your SD card with
UUID=<uid> / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1
note the commit=600, which means that changes will be written only every 10 minutes. If your machine crashes inbetween, the file system might become corrupt. Thus, you might run fsck.ext4 on your SD card file system. In order to counteract the problem at all, you could:
run fsck at every boot
omit the commit setting. My SD card can handle the additional stress quite well.
What I think solved the problem for me is to put my external HDD, attached via SATA, to sleep. I was surprised that this does not happen automatically. Now I have the following section attached to my /etc/hdparm.conf:
/dev/disk/by-uuid/<uid> {
spindown_time = 60
write_cache = off
}
This tells hdparm to put the HDD to standby after 5 minutes of inactivity. Turning the write cache off is a safety measure against file system corruption again. Some disks reorder BtrFS write instructions when they shouldn't.
Something that I observed is that putting some load on the machine helped to keep the system alive. Unfortunately, that was true only until some point in time. However, if attaching Keyboard and Mouse or letting a script run throughout helps with restarts, you will have something to work with.
I use the following script to log information that might help me in case of a crash:
#!/bin/bash
# LICENSE: GPLv3 or later
set -euo pipefail
LOGFILE=/home/mgoerner/error-detection.log
function main() {
parse_cli_args "$#"
while true
do
print_debug_information >>"$LOGFILE" 2>&1
sync
sleep 3m
done
}
function parse_cli_args() {
if [[ $# -eq 1 ]]
then
arg="$1";shift
if [[ "$arg" == "--help" || "$arg" == "-h" ]]
then
print_usage
exit
fi
LOGFILE="$arg"
elif [[ $# -gt 1 ]]
then
echo "Please provide at most one argument!" >&2
exit 1
fi
}
function print_usage() {
cat <<EOF
$0 [LOGFILE]
EOF
}
function print_debug_information() {
echo
date
uptime
dmesg -uT | tail
ip addr show wlxd85d4c97e434
iwlist wlxd85d4c97e434 scan | egrep ESSID
hdparm -acdgkmurC /dev/sda
free
}
main "$#"
I let it start automatically after boot. Setting the sleep time under ~10 minutes used to let the crashes disappear, but does not anymore. Unfortunately, the error log produced by this script never helped getting any insight. The same goes for various logs from /var/log/. However, that might be different for you.
Furthermore, I suspect that my WiFi dongle does not like it warm. I reuse a childs shoe box as case and putting the dongle inside the closed box caused some connectivity problems.
Last but not least, I turned off automatic updates after reboots. Very often the crashes happened directly after some (mysterious) reboot. Having turned of automatic updates helped me getting rid of this altogether.