For the last month or so my unraid machine has become responsive every few days which I've only been able to resolve via powering off. Looking at "syslog-previous" it looks like an issue occurred at around 23:09 on Dec 18 but my google fu has failed me. Can anyone see what the problem is? 

syslog tower-diagnostics-20241219-1155.zip syslog-previous

Was the VM shutdown initiated by you?

Just now, JorgeB said:

Was the VM shutdown initiated by you?

Not that I recall

But was that the time you tried to power off? A shutdown attempt will start by shutting down the VMs.

no I only realised there was a problem at 04:26 on Dec 19 when I cycled the power.

It's possible I ran a docker update at 23:09 on Dec 18 , could be a red herring though

it's just happened again, just after 18:00. "syslog-previous" doesn't seem very useful, is there some other log that might suggest what is happening?

tower-diagnostics-20241219-1914.zip syslog-previous syslog

You can enable the syslog server to see if it catches anything more.

I'm now getting some read errors and a lot of segfaults. Any ideas?

tower-diagnostics-20241221-1017.zip

Segfaults can mean a hardware issue, like bad RAM, but they appear to be limited to Connect for now, uninstall the Connect plugin and retest.

ok thanks, i've removed the connect plug in, some other errors now:

 

ec 21 13:12:20 Tower kernel: unraid-api[9542]: segfault at 8 ip 00001543e6b7980c sp 00007fff870e60c0 error 4 in ld-2.37.so[1543e6b5c000+27000] likely on CPU 4 (core 4, socket 0)
Dec 21 13:12:20 Tower kernel: Code: 84 7a 06 00 00 48 85 c0 75 e4 48 83 3d 14 67 01 00 00 0f 85 8b 06 00 00 48 8b bd 98 fd ff ff 48 8b 57 68 48 8b 87 80 02 00 00 <48> 8b 5a 08 f6 87 36 03 00 00 20 0f 84 2f 06 00 00 48 03 1f 48 85
Dec 21 13:12:20 Tower kernel: unraid-api[9545]: segfault at 8 ip 0000145c72f2780c sp 00007ffdd1b1f940 error 4 in ld-2.37.so[145c72f0a000+27000] likely on CPU 8 (core 2, socket 0)
Dec 21 13:12:20 Tower kernel: Code: 84 7a 06 00 00 48 85 c0 75 e4 48 83 3d 14 67 01 00 00 0f 85 8b 06 00 00 48 8b bd 98 fd ff ff 48 8b 57 68 48 8b 87 80 02 00 00 <48> 8b 5a 08 f6 87 36 03 00 00 20 0f 84 2f 06 00 00 48 03 1f 48 85
Dec 21 13:12:21 Tower root: plugin: running: anonymous
Dec 21 13:12:21 Tower root: plugin: dynamix.unraid.net.plg removed
Dec 21 13:12:22 Tower nginx: 2024/12/21 13:12:22 [error] 9416#9416: *342981 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:27 Tower nginx: 2024/12/21 13:12:27 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:33 Tower nginx: 2024/12/21 13:12:33 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:38 Tower nginx: 2024/12/21 13:12:38 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:43 Tower nginx: 2024/12/21 13:12:43 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:48 Tower nginx: 2024/12/21 13:12:48 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:53 Tower nginx: 2024/12/21 13:12:53 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:12:58 Tower nginx: 2024/12/21 13:12:58 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:04 Tower nginx: 2024/12/21 13:13:04 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:09 Tower nginx: 2024/12/21 13:13:09 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:15 Tower nginx: 2024/12/21 13:13:15 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:21 Tower nginx: 2024/12/21 13:13:21 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:27 Tower nginx: 2024/12/21 13:13:27 [error] 9416#9416: *342986 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:33 Tower nginx: 2024/12/21 13:13:33 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:39 Tower nginx: 2024/12/21 13:13:39 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"
Dec 21 13:13:45 Tower nginx: 2024/12/21 13:13:45 [error] 9416#9416: *342986 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.my.servers/include/unraid-api.php (No such file or directory)" while reading response header from upstream, client: 192.168.2.12, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "tower", referrer: "http://tower/Plugins"

 

See here>

https://forums.unraid.net/topic/154030-6128-log-spammed-with-references-to-unraid-apiphp/

 

I'm still getting crashes every couple of days. So far I've replaced the motherboard, cpu and RAM so it doesn't seem related to that.

 

Googling for "rcu: INFO: rcu_preempt self-detected stall on CPU" suggests there might be a problem with a docker container or a hardware problem with one of my cache drives. Does that sound likely?

unraid.log

It could be, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers.

the hardware problem appears to be an m.2 drive i'd added as a cache drive. Now that it's removed the machine is stable again