Server Becoming Unavailable and Requires Reboot

[QPlus7]

Hi, all.

 

I've been dealing with an issue for about a month or so but it hasn't been at the top of my priority list as a reboot fixes the issue, but it is becoming more and more prevalent now.

 

From time-to-time by server will go offline and when I check the console I see the errors below:

 

grcu: INFO: rcu preempt detected stalls on CUs/tasks: orcu: oTasks blocked on level-0 rcu_node (CPUs 0-7): P9307/1:b..1 oldetected by 5, t=83224420 jiffies, g=3172721, 4=51962606 ncpus=8)
rcu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: ( P9307 } 83249702 jiffies s: 26941 roat: 0x0/T orcu: blocking rcu node structures (internal RCU debug ):
orcu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: { P9307 } 83315237 jiffies s: 26941 root: 0x0/T orcu: blocking rcu node structures (internal RCU debug):
orcu: INFO: rcu_ preempt detected expedited stalls on CPUs/tasks: € P9307 } 83380773 jiffies s: 26941 root: 0x0/T orcu: blocking rcu _node structures (internal RCU debug ): orcu: INFO: rcu_ preempt detected stalls on CPUs/tasks: orcu: oTasks blocked on level-o rcu_node (CUs 0-7): P9307/1:b..1 lodetected by 4, t=83404433 jiffies, g=3172721, 4=52049432 ncpus=8)
orcu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: € P9307 } 83446315 jiffies s: 26941 root: 0x0/T
orcu: blocking rcu node structures (internal RCU debug ):
calls on CPus/tasks: ( P9307 7 83511846 liffies s: 26941 root: Overt
orCu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: € P9307 ) 83511846 jiffies s: 26941 root: 0x0/T
orcu: blocking rcu node structures (internal RCU debug ):
orcu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: { P9307 } 83577381 jiffies s: 26941 root: 0x0/T orcu: blocking rcu_node structures (internal RCU debug ): rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: orcu: oTasks blocked on level-0 rcu_node (CPUs 0-7): P9307/1:b..1 coldetected by 2, t=83584444 jiffies, g=3172721, 9=52139803 ncpus=B)
orcu: INFO: rcu _preempt detected expedited stalls on CPUs/tasks: ( P9307 } 83642917 jiffies s: 26941 root: 0x0/T orcu: blocking rcu_node structures (internal RCU debug ):
rcu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: { P9307 } 83708453 jiffies s: 26941 root: 0x0/T
rcu: blocking rcu _node structures (internal RCU debug) : rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: Tasks blocked on level-0 rcu_node (CPUs 0-7): P9307/1:b..1 oldetected by 0, t=83764457 jiffies, g=3172721, q-52229960 ncpus=8)
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: € P9307 } 83773989 jiffies s: 26941 root: 0x0/T rcu: blocking rcu_node structures (internal RCU debug ):
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { P9307 } 83839526 jiffies s: 26941 root: 0x0/T rcu: blocking rcu_node structures (internal RCU debug ): rcu :
INFO: rcu preempt detected expedited stalls on CPUs/tasks: { P9307 } 83905061 jiffies s: 26941 root: 0x0/T
rcu: blocking rcu_node structures (internal RCU debug ): rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: Tasks blocked on level-0 rcu_node (CPUs 0-7): P9307/1:b..1
•(detected by 1, t=83944468 jiffies, g=3172721, 4-52319742 ncus=8)
rcu: INFO: rcu preempt detected expedited stalls on CPUs/tasks: { P9307 } 83970598 jiffies s: 26941 root: 0x0/T
rcu: blocking rcu _node structures (internal RCU debug ):
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: ( P9307 7 84036133 jiffies s: 26941 root: 0x0/T rcu: blocking rcu_node structures (internal RCU debug ):

 

I cannot type anything at the physical console and the server is unresponsive on the network. Rebooting the server resolves the issue, but it comes back later.

 

I'm running UnRaid 6.12.3 and diagnostics file is attached for review.

 

Any help is greatly appreciated - thanks in advance!

 

scw-nas01-diagnostics-20240327-0937.zip

[QPlus7]

I didn't realize I was so far out of date. I'm currently updating to 6.12.9 and will see if the issue persists.

[QPlus7]

It just happened again. I had forgotten about this until I got an alert notifying me that UnRaid was offline. 

[JorgeB]

Enable the syslog server and post that after a crash.

[QPlus7]

On 4/8/2024 at 3:38 AM, JorgeB said:

Enable the syslog server and post that after a crash.

Done. Let’s see if it happens again. 
 

Thanks for the suggestion.