UNRAID WEBUI UNresponsive, can't reboot or shutdown, 3 cores 100%

[tessierp]

Hi, I'm faced with what I consider to be a very serious issue. I was trying to access a folder on my share and Windows Explorer started to hang. I was curious to see if something was wrong so went to UNRAID's WebUI.. I noticed that 3 cores were in the RED (used 100%) and then it started to not work, got a Internal Server Error (500)... I ran this command once "/etc/rc.d/rc.php-fpm restart" which helped... I proceeded to my share again, same thing happens except now "/etc/rc.d/rc.php-fpm restart" wont work, reboot wont work.. HTOP reports several cores 100%.

 

Thinking it was probably the PLEX container I tried to stop the container from SSH but every command is locking up at this point..

 

I seem to be faced with no other choice but to forcefully shutdown my server.... Has anyone every experienced anything like this? This is kind of worrying me...

 

 

[tessierp]

I was able to recover the syslog before I had to force a shutdown.. Now a parity check is underway now.. Not sure if a DEV would like to take a look at this. I've seen others complain of the WEBUI becoming unresponsive and that weren't able to reboot... I think this needs to be looked into.

[Tristankin]

Could you share your diagnostic data and syslog mirrored to flash?

[tessierp]

I can certainly provide the syslog.. As for the Diagnostic data, it would be from right now, not the moment the problem occurred if that is what you mean.

[tessierp]

@Tristankin Here are the files you requested.

 

Thanks

syslog ptr1-nas-1-diagnostics-20211118-1946.zip

[Tristankin]

Looking into other cases of rcu_sched self-detected stalls seem to point to cache corruption. I'm looking to see if there is anything specific for BTRFS checks.

Other posts have also talked about power supply low power control settings in bios.

 

Other things to check are ram overclocking and disabling some of the low power c states.

 

Edited November 19, 2021 by Tristankin

[tessierp]

I have no idea how I could have had cache corruption.

 

As for Global C States, they are all disabled. No RAM overclocking. There is no power supply low power, what I mean is, everything is set to nominal in that regard.

 

Thanks for the help.. Been looking at the logs, I didn't find anything that really helped me, hopefully someone does. But the cache corruption? I really wonder how that could have happened...

Edited November 19, 2021 by tessierp

[Tristankin]

Sorry I couldn't be of further assistance, hopefully someone more knowledgeable chimes in.