Unresponsive server - no Web UI, no SSH, no Ping

[potjoe]

 

Hi, 

 

For some time now, I've observed more and more often situations where my server became totally unresponsive : no Web UI, no VM running (no pfsense thus, but it's another story), no SSH connections, server not responding to ping. I noticed that when it happens power consumption raises by 50-70 % if the UPS is to be trusted. I usually try a quick press of power button, but it's not rebooting the server. I am forced to break the first rule and to force shutdown by pressing the reset button and force an unclean shutdown. 

 

The issue is that I'm running headless without GPU installed, and the Ryzen 1600 CPU I'm using does not support video output. Thus, I can't plug a monitor when the issue occurs. I've run a diagnostic after the reboot, but if I'm not mistaken it won't provide any useful information right ? I'm including it anyway, just in case.

This is the fourth time in two months it's happening, and I'm really out of ideas. Any help to troubleshoot the issue would be really great. Thank you for your time. 

P.s. : the pfsense being down when it happens, I plug in directly into the switch with a manual IP assignment to test connections. Since Unraid has a fixed IP, it should respond anyway.

 

Edited January 23, 2022 by potjoe

[JorgeB]

Make sure power supply idle control is correctly set.

[potjoe]

Thank you. I'll double check when at home, I'm almost sure to have follow instructions this page regarding ryzen cpus when I set up the server last year. Not sure about power supply idle control, but I remember disabling global C-State control. I've never experienced such behaviour in roughly a year and a half. 

Edited December 8, 2021 by potjoe

[potjoe]

Hi! So I finally had the time to check bios settings tonight. My memories were correct : when I set up the server a year ago, I disabled Global C-state control but did not touched Power Supply Idle Control. I adjusted them according to the recommendations : Power Supply Idle Control set to "Typical Current Idle" and re-enabled Global C-state control (set to auto). 

 

The situation where Unraid became unresponsive only started two months ago or so. I'm not familiar with the options you mentionned, thus I don't know if this was the source of the issue.

 

Current settings : 

Edited December 9, 2021 by potjoe

[potjoe]

Hi! After two weeks without issue, my server went unresponsive again this afternoon depite last time's BIOS changes. Weeks ago I enabled the syslog server to see if something was relevant in the last minutes preceding the bug. As you can see the last activity reported before the crash today (it happened somewhen between 15:30 and 15:59, time of hard reset when I had to hard reset again...), was the RAM-Disk synced : 

 

 

Dec 25 14:07:56 MonaLisa dhcpcd[1909]: br0: Router Advertisement from fe80:**************:c8c
Dec 25 14:08:01 MonaLisa dhcpcd[1909]: br0: Router Advertisement from fe80::**************:c8c
Dec 25 14:17:29 MonaLisa dhcpcd[1909]: br0: part of a Router Advertisement expired
Dec 25 14:17:29 MonaLisa dnsmasq[26780]: reading /etc/resolv.conf
Dec 25 14:17:29 MonaLisa dnsmasq[26780]: using nameserver 192.168.10.254#53
Dec 25 14:17:29 MonaLisa dnsmasq[26780]: using nameserver 2a01:**************1#53
Dec 25 14:17:29 MonaLisa dnsmasq[26780]: reading /etc/resolv.conf
Dec 25 14:17:29 MonaLisa dnsmasq[26780]: using nameserver 192.168.10.254#53
Dec 25 14:17:29 MonaLisa dnsmasq[26780]: using nameserver 2a01:**************#53
Dec 25 14:30:01 MonaLisa docker: RAM-Disk synced
Dec 25 15:00:01 MonaLisa docker: RAM-Disk synced
Dec 25 15:17:50 MonaLisa dhcpcd[1909]: br0: Router Advertisement from fe80::**************:c8c
Dec 25 15:17:58 MonaLisa dhcpcd[1909]: br0: Router Advertisement from fe80::**************:c8c
Dec 25 15:30:01 MonaLisa docker: RAM-Disk synced
Dec 25 15:59:58 MonaLisa kernel: mdcmd (36): set md_write_method 1
Dec 25 15:59:58 MonaLisa kernel: 
Dec 25 15:59:58 MonaLisa cache_dirs: Arguments=-p 1 -u -e Backup -e Enregistrements -e appdata -e domains -e isos -e system -l off
Dec 25 15:59:58 MonaLisa cache_dirs: Max Scan Secs=10, Min Scan Secs=1

 

This process is due to the tweak proposed by mgutt in this thread to avoid wearing out SSD's : 

 

 

I'm not sure at all it is related to the issue I'm having, just mentionning it. I don't have any clue on why I'm having these frequent locks of the server.

Thank you very much for your time! 

 

Edited December 25, 2021 by potjoe

[mgutt]

24 minutes ago, potjoe said:

I'm not sure at all it is related to the issue I'm having

Don't think so. I'm using this since I released my tweak without any problems.

 

26 minutes ago, potjoe said:

15:30 and 15:59

I can not see any logs which show a restart of the server?! The first entry is something as follows:
 

Quote

 

<date> xxx kernel: Linux version 123456-Unraid ...

 

 

 

[potjoe]

Thanks @mgutt! Actually there is no sign of the restart because I copied the lines from the syslog server local record (That I enabled from Unraid settings to be stored in appdata). In this log, we can see the last record before the lock : 

Dec 25 15:30:01 MonaLisa docker: RAM-Disk synced

I think the first lines  "<date> xxx kernel: Linux version 123456-Unraid" are missing because rsyslogd had not started yet.

I confirm that I had to reset the server and that it restarted : here is a new Diagnostic file with the full logs this time. 

 

Edit : in the logs from the diagnostic archive, we can see that the server restarted : 

Dec 25 15:57:17 MonaLisa kernel: Linux version 5.10.28-Unraid (root@Develop) (gcc (GCC) 9.3.0, GNU ld version 2.33.1-slack15) #1 SMP Wed Apr 7 08:23:18 PDT 2021

 

 

Edited January 23, 2022 by potjoe
Added logs