Greetings,
I was trying to use plex this morning, and it wouldn't load. I checked unraid, and it was very sluggish, and one of my cores was pegged at 100%. Which is very unusual. I checked the system log, and found numerous entries such as this:
Aug 31 07:08:01 UnRaid kernel: Call Trace:
Aug 31 07:08:01 UnRaid kernel: __schedule+0x45e/0x4a4
Aug 31 07:08:01 UnRaid kernel: schedule+0x73/0x99
Aug 31 07:08:01 UnRaid kernel: schedule_timeout+0xa7/0xe0
Aug 31 07:08:01 UnRaid kernel: ? __next_timer_interrupt+0xe8/0xe8
Aug 31 07:08:01 UnRaid kernel: rcu_gp_kthread+0x417/0x7a9
Aug 31 07:08:01 UnRaid kernel: ? kfree_rcu_work+0x19e/0x19e
Aug 31 07:08:01 UnRaid kernel: kthread+0xe5/0xea
Aug 31 07:08:01 UnRaid kernel: ? __kthread_bind_mask+0x57/0x57
Aug 31 07:08:01 UnRaid kernel: ret_from_fork+0x22/0x30
Aug 31 07:09:01 UnRaid kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 31 07:09:01 UnRaid kernel: rcu: 10-...!: (0 ticks this GP) idle=f4c/0/0x0 softirq=58410444/58410444 fqs=2 (false positive?)
Aug 31 07:09:01 UnRaid kernel: (detected by 22, t=60002 jiffies, g=143930057, q=204228)
Aug 31 07:09:01 UnRaid kernel: Sending NMI from CPU 22 to CPUs 10:
Aug 31 07:09:01 UnRaid kernel: NMI backtrace for cpu 10 skipped: idling at acpi_idle_do_entry+0x31/0x37
Aug 31 07:09:01 UnRaid kernel: rcu: rcu_sched kthread starved for 60003 jiffies! g143930057 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=10
Aug 31 07:09:01 UnRaid kernel: rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
Aug 31 07:09:01 UnRaid kernel: rcu: RCU grace-period kthread stack dump:
Aug 31 07:09:01 UnRaid kernel: task:rcu_sched state:I stack: 0 pid: 10 ppid: 2 flags:0x00004000
I don't know linux good enough to determine what the cause of this could be? I tried to kill the vm that is pinned to cpu 10/22(which is it's physical and logical core) and that did not help. It actually made things worse. I ended up having to hard reset the box.
I've also have come in to work in the morning and plex was down. And checking unraid, the whole box was unresponsive. syslog never showed me anything out of the ordinary. the last few lines were just things like a drive spinning down, or the fans spinning up or down due to drive temps. I think I may have been running in to this issue for a few weeks now, and only now was able to see it happening in real time to see better data? I can grab splunk logs if that would help as I am willing to bet the diagnostic zip won't have this data in it, as I had to reboot to get back in to unraid. I was unable to pull logs prior as the ui was basically unresponsive..
Any help would be greatly appreciated here! Thanks!!!
unraid-diagnostics-20210831-0808.zip