Server completely locking up every couple of days

[fulknerra]

Seems that every other day my unraid box is fully freezing and I have to hard power down, cant tell if its a kernel panic or something else. If I console into the box it just shows an unresponsive login prompt. Syslog (attached) shows a handful of error messages that seem concerning but nothing at the point of failure.

 

At first I was doing a backup of appdata/unraid every night, along with a TRIM, and plugin update check. I've since scheduled all my backups/trims to happen on different days, I also noticed that before every freeze the plugin updater was running, so I removed it, with these changes the freezing is still happening.

 

After the last crash I ran a parity check and noticed that it resolved 5K errors within the first few hours of running but none after that.

 

memtest passed 48/48 tests no errors

stresslinux showed no faults

ran full SMART tests on the drives, seems healthy.

 

Not sure what I should be looking at resolve this problem

Tower.log tower-diagnostics-20231122-0913.zip

[JorgeB]

Nov 20 05:02:29 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Nov 20 05:02:29 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Nov 20 05:02:29 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

[fulknerra]

I use a the br0 network so that I can assign individual IPs to every docker and have them be accessible by the host, does ipvlan still allow the same type of setup?

[JorgeB]

Yes, but you can keep using macvaln with v6.12.4 if you disable bridging, see the release notes.

[fulknerra]

On 11/23/2023 at 2:53 AM, JorgeB said:

Nov 20 05:02:29 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Nov 20 05:02:29 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Nov 20 05:02:29 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

This appears to have resolved the issue, going on more than 4 days of uptime now, which is the best I've seen all month. 

 

Thanks

[fulknerra]

On 11/23/2023 at 2:53 AM, JorgeB said:

Nov 20 05:02:29 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Nov 20 05:02:29 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Nov 20 05:02:29 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

 

I had another similar outage last week and this week after disabling macvlan, full lock up but no kernel panics and nothing really in the syslog. This time I was able to specifically correlate it to the APPDATA BACKUP plugin unfortunately as that doesn't log to syslog I can't pinpoint what action the backup plugin was doing, Disabling the functions of the backup has stopped all lockups so I believe it was a mix of macvlan and the backup.

 

I have the following settings enabled 

backuptype: stop,backup,start each container
delete older than: 7days
keep at least: 3 
appdata sources: 
/mnt/user/appdata
/mnt/cache/appdata

destination: /mnt/user/backup
use compression: yes, normal
verify: yes
ignore errors: no
backup flash: yes
backup VM Meta: yes

 

Checking on the backup directory for any of the previous backups, the container data is all there but what is missing is the flashbackup and the VM meta data, so I am assuming that the system locks up when it attempts to backup the flashdrive. I'm not entirely sure why this happens, but my system has been stable since disabling this function. 

 

leaving this information here for anyone else who might stumble across the same issue.