So roughly since I first tried to install the pihole docker, my Unraid Server started locking up every night. I had to force-shotdown and restart every time, server would run normally until sometime in the night and be locked up again the next morning. Initially I thought it had something to do with Pihole, so I removed said docker. no change. The issue started in Unraid 6.11.something and is persistent after the update to current version. all plugins up-to-date.

 

It took me a while to figure out persistent syslogs, now that I have them it seems there's some sort of infinite loop going on. I tried googling every single line, but imma be honest, I'm a Windows guy and don't understand most of what's going on here. So I'm coming to you guys for help.

 

Can you point me in the right direction? Were do I even start with figuring this out further?

 

 

Here's the part of the log that keeps repeating every ~30 seconds give or take.

Aug 15 20:57:16 IanMainServer kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 5-...D } 21498 jiffies s: 10741 root: 0x1/.
Aug 15 20:57:16 IanMainServer kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x20/.
Aug 15 20:57:16 IanMainServer kernel: Sending NMI from CPU 13 to CPUs 5:
Aug 15 20:57:16 IanMainServer kernel: NMI backtrace for cpu 5
Aug 15 20:57:16 IanMainServer kernel: CPU: 5 PID: 12888 Comm: docker-proxy Tainted: P      D W  O       6.1.38-Unraid #2
Aug 15 20:57:16 IanMainServer kernel: Hardware name: System manufacturer System Product Name/PRIME B450-PLUS, BIOS 2202 07/14/2020
Aug 15 20:57:16 IanMainServer kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf
Aug 15 20:57:16 IanMainServer kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 4a 3f ff
Aug 15 20:57:16 IanMainServer kernel: RSP: 0018:ffffc900031bbd28 EFLAGS: 00000002
Aug 15 20:57:16 IanMainServer kernel: RAX: 0000000000000101 RBX: ffff88881e9200c0 RCX: 0000000000000000
Aug 15 20:57:16 IanMainServer kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88881e9200c0
Aug 15 20:57:16 IanMainServer kernel: RBP: ffffc900031bbdd8 R08: 0000000000000000 R09: ffff88881e960120
Aug 15 20:57:16 IanMainServer kernel: R10: 0000000000000020 R11: 0000000000000078 R12: ffffc900031bbd90
Aug 15 20:57:16 IanMainServer kernel: R13: ffffc900031bbee8 R14: 0000000000000283 R15: 00000000005af458
Aug 15 20:57:16 IanMainServer kernel: FS:  000000c000054090(0000) GS:ffff88881e940000(0000) knlGS:0000000000000000
Aug 15 20:57:16 IanMainServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 15 20:57:16 IanMainServer kernel: CR2: 0000290959092000 CR3: 000000015d26e000 CR4: 00000000003506e0
Aug 15 20:57:16 IanMainServer kernel: Call Trace:
Aug 15 20:57:16 IanMainServer kernel: <NMI>
Aug 15 20:57:16 IanMainServer kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 15 20:57:16 IanMainServer kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 15 20:57:16 IanMainServer kernel: ? nmi_handle+0x57/0x131
Aug 15 20:57:16 IanMainServer kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf
Aug 15 20:57:16 IanMainServer kernel: ? default_do_nmi+0x66/0x15b
Aug 15 20:57:16 IanMainServer kernel: ? exc_nmi+0xbf/0x130
Aug 15 20:57:16 IanMainServer kernel: ? end_repeat_nmi+0x16/0x67
Aug 15 20:57:16 IanMainServer kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf
Aug 15 20:57:16 IanMainServer kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf
Aug 15 20:57:16 IanMainServer kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf
Aug 15 20:57:16 IanMainServer kernel: </NMI>
Aug 15 20:57:16 IanMainServer kernel: <TASK>
Aug 15 20:57:16 IanMainServer kernel: do_raw_spin_lock+0x14/0x1a
Aug 15 20:57:16 IanMainServer kernel: _raw_spin_lock_irqsave+0x2c/0x37
Aug 15 20:57:16 IanMainServer kernel: lock_hrtimer_base+0x24/0x49
Aug 15 20:57:16 IanMainServer kernel: hrtimer_try_to_cancel+0x37/0x84
Aug 15 20:57:16 IanMainServer kernel: hrtimer_cancel+0x11/0x1f
Aug 15 20:57:16 IanMainServer kernel: futex_wait+0x14a/0x17e
Aug 15 20:57:16 IanMainServer kernel: ? hrtimer_init_sleeper+0x41/0x41
Aug 15 20:57:16 IanMainServer kernel: do_futex+0xcd/0x143
Aug 15 20:57:16 IanMainServer kernel: __do_sys_futex+0x127/0x150
Aug 15 20:57:16 IanMainServer kernel: do_syscall_64+0x6b/0x81
Aug 15 20:57:16 IanMainServer kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Aug 15 20:57:16 IanMainServer kernel: RIP: 0033:0x463ea3
Aug 15 20:57:16 IanMainServer kernel: Code: 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 24 14 4c 8b 54 24 18 4c 8b 44 24 20 44 8b 4c 24 28 b8 ca 00 00 00 0f 05 <89> 44 24 30 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
Aug 15 20:57:16 IanMainServer kernel: RSP: 002b:000000c000063e78 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
Aug 15 20:57:16 IanMainServer kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000463ea3
Aug 15 20:57:16 IanMainServer kernel: RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00000000005af458
Aug 15 20:57:16 IanMainServer kernel: RBP: 000000c000063ec0 R08: 0000000000000000 R09: 0000000000000000
Aug 15 20:57:16 IanMainServer kernel: R10: 000000c000063eb0 R11: 0000000000000246 R12: 000000c000063eb0
Aug 15 20:57:16 IanMainServer kernel: R13: 000000c000054000 R14: 000000c0000064e0 R15: 000014a00f4ee334
Aug 15 20:57:16 IanMainServer kernel: </TASK>

 

The first appearance is in line 138 in the attached log. Immediately before that were a few error lines I also can't quite pin down, presumably that's related?

syslog_cut_for_privacy.txt

There are constant crashes, suggesting more a hardware problem, start here, then also good to run memtest.

On 8/17/2023 at 6:58 PM, JorgeB said:

There are constant crashes, suggesting more a hardware problem, start here, then also good to run memtest.

 

After having zero crashes/freezes these last few days, I'll mark this one as the solution and give some extra info:

 

- Running Ryzen 1700X with 2133MHz RAM, no manual overclock

- I deleted Pihole docker

- I disabled the C-States completely and also turned off Precision Auto boost, 'cause I realisticly don't need the extra performance for my docker containers.

 

Since then I've run without crashes, so I blame C-States. Still not quite sure what triggered it, it did start happening when I put the Pihole static IP on networt br0. No memtest yet. I'll re-add pihole and see if my system continues to be stable. If I don't report back, assume everything has worked.