EarthYak Posted November 3, 2020 Share Posted November 3, 2020 Unraid around 1-3 times a week at apparently random times is stopping and becoming unresponsive, the syslog shows this before it stops until I pull the power and restart. I restarted it the next day. Does anyone have any ideas, help would be gratefully appreciated. Excerpt from the syslog, full log for the day attached: Nov 2 20:16:13 Hydra kernel: rcu: INFO: rcu_sched self-detected stall on CPU Nov 2 20:16:13 Hydra kernel: rcu: #01110-....: (59998 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=203068959/203068959 fqs=14300 Nov 2 20:16:13 Hydra kernel: rcu: #011 (t=60001 jiffies g=284290801 q=1671571) Nov 2 20:16:13 Hydra kernel: NMI backtrace for cpu 10 Nov 2 20:16:13 Hydra kernel: CPU: 10 PID: 435 Comm: smbd Tainted: G W O 4.19.107-Unraid #1 Nov 2 20:16:13 Hydra kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P3.30 11/04/2019 Nov 2 20:16:13 Hydra kernel: Call Trace: Nov 2 20:16:13 Hydra kernel: <IRQ> Nov 2 20:16:13 Hydra kernel: dump_stack+0x67/0x83 Nov 2 20:16:13 Hydra kernel: nmi_cpu_backtrace+0x71/0x83 Nov 2 20:16:13 Hydra kernel: ? lapic_can_unplug_cpu+0x97/0x97 Nov 2 20:16:13 Hydra kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Nov 2 20:16:13 Hydra kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Nov 2 20:16:13 Hydra kernel: rcu_check_callbacks+0x296/0x5a0 Nov 2 20:16:13 Hydra kernel: update_process_times+0x24/0x47 Nov 2 20:16:13 Hydra kernel: tick_sched_timer+0x36/0x64 Nov 2 20:16:13 Hydra kernel: __hrtimer_run_queues+0xb7/0x10b Nov 2 20:16:13 Hydra kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Nov 2 20:16:13 Hydra kernel: hrtimer_interrupt+0xf4/0x20e Nov 2 20:16:13 Hydra kernel: smp_apic_timer_interrupt+0x7b/0x93 Nov 2 20:16:13 Hydra kernel: apic_timer_interrupt+0xf/0x20 Nov 2 20:16:13 Hydra kernel: </IRQ> Nov 2 21:58:13 Hydra kernel: rcu: INFO: rcu_sched self-detected stall on CPU Nov 2 21:58:13 Hydra kernel: rcu: #01110-....: (6180101 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=203068959/203068959 fqs=1502221 Nov 2 21:58:13 Hydra kernel: rcu: #011 (t=6180104 jiffies g=284290801 q=40981941) Nov 2 21:58:13 Hydra kernel: NMI backtrace for cpu 10 Nov 2 21:58:13 Hydra kernel: CPU: 10 PID: 435 Comm: smbd Tainted: G W O 4.19.107-Unraid #1 Nov 2 21:58:13 Hydra kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P3.30 11/04/2019 Nov 2 21:58:13 Hydra kernel: Call Trace: Nov 2 21:58:13 Hydra kernel: <IRQ> Nov 2 21:58:13 Hydra kernel: dump_stack+0x67/0x83 Nov 2 21:58:13 Hydra kernel: nmi_cpu_backtrace+0x71/0x83 Nov 2 21:58:13 Hydra kernel: ? lapic_can_unplug_cpu+0x97/0x97 Nov 2 21:58:13 Hydra kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Nov 2 21:58:13 Hydra kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Nov 2 21:58:13 Hydra kernel: rcu_check_callbacks+0x296/0x5a0 Nov 2 21:58:13 Hydra kernel: update_process_times+0x24/0x47 Nov 2 21:58:13 Hydra kernel: tick_sched_timer+0x36/0x64 Nov 2 21:58:13 Hydra kernel: __hrtimer_run_queues+0xb7/0x10b Nov 2 21:58:13 Hydra kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Nov 2 21:58:13 Hydra kernel: hrtimer_interrupt+0xf4/0x20e Nov 2 21:58:13 Hydra kernel: smp_apic_timer_interrupt+0x7b/0x93 Nov 2 21:58:13 Hydra kernel: apic_timer_interrupt+0xf/0x20 Nov 2 21:58:13 Hydra kernel: </IRQ> Nov 2 21:58:13 Hydra kernel: RIP: 0010:radix_tree_descend+0x16/0x57 Nov 2 21:58:13 Hydra kernel: Code: 48 8b 42 08 4c 89 4a 08 48 89 57 18 48 89 47 20 4c 89 08 c3 0f b6 0f 48 89 d0 48 d3 e8 83 e0 3f 89 c2 48 8d 54 d7 28 48 8b 12 <48> 89 d1 83 e1 03 48 ff c9 75 32 48 8d 4f 28 48 39 ca 72 29 4c 8d Nov 2 21:58:13 Hydra kernel: RSP: 0018:ffffc9000c657c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Nov 2 21:58:13 Hydra kernel: RAX: 000000000000001f RBX: 000000000001f518 RCX: 000000000000000c Nov 2 21:58:13 Hydra kernel: RDX: ffff888106090001 RSI: ffffc9000c657c78 RDI: ffff888102ff8db0 Nov 2 21:58:13 Hydra kernel: RBP: ffff888151151078 R08: ffff888151151078 R09: ffffc9000c657ca8 Nov 2 21:58:13 Hydra kernel: R10: 0000000000000000 R11: ffff888151151070 R12: 00000000006200ca Nov 2 21:58:13 Hydra kernel: R13: ffff888151151068 R14: 0000000000001000 R15: 000000000001f518 Nov 2 21:58:13 Hydra kernel: __radix_tree_lookup+0x69/0xa2 Nov 2 21:58:13 Hydra kernel: radix_tree_lookup_slot+0x1e/0x41 Nov 2 21:58:13 Hydra kernel: find_get_entry+0x14/0x8f Nov 2 21:58:13 Hydra kernel: pagecache_get_page+0x20/0x1bd Nov 2 21:58:13 Hydra kernel: grab_cache_page_write_begin+0x1a/0x31 Nov 2 21:58:13 Hydra kernel: fuse_perform_write+0x178/0x43a Nov 2 21:58:13 Hydra kernel: ? file_remove_privs+0x55/0xb9 Nov 2 21:58:13 Hydra kernel: fuse_file_write_iter+0x1b6/0x22f Nov 2 21:58:13 Hydra kernel: __vfs_write+0xfc/0x13a Nov 2 21:58:13 Hydra kernel: vfs_write+0xc7/0x166 Nov 2 21:58:13 Hydra kernel: ksys_pwrite64+0x5d/0x79 Nov 2 21:58:13 Hydra kernel: do_syscall_64+0x57/0xf2 Nov 2 21:58:13 Hydra kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 2 21:58:13 Hydra kernel: RIP: 0033:0x14d2885a3ed7 Nov 2 21:58:13 Hydra kernel: Code: 08 89 3c 24 48 89 4c 24 18 e8 05 f3 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 35 f3 ff ff 48 8b Nov 2 21:58:13 Hydra kernel: RSP: 002b:000014d2867e8bf0 EFLAGS: 00000293 ORIG_RAX: 0000000000000012 Nov 2 21:58:13 Hydra kernel: RAX: ffffffffffffffda RBX: 000055f7b41e7890 RCX: 000014d2885a3ed7 Nov 2 21:58:13 Hydra kernel: RDX: 0000000000100000 RSI: 000055f7b4631850 RDI: 000000000000001f Nov 2 21:58:13 Hydra kernel: RBP: 000055f7b3e7fff0 R08: 0000000000000000 R09: 00000000ffffffff Nov 2 21:58:13 Hydra kernel: R10: 000000001f500000 R11: 0000000000000293 R12: 000014d2867e8c70 Nov 2 21:58:13 Hydra kernel: R13: 000055f7b3e80028 R14: 000055f7b3efaf50 R15: 000014d289f850a0 Nov 2 21:58:42 Hydra kernel: rcu: INFO: rcu_bh self-detected stall on CPU Nov 2 21:58:42 Hydra kernel: rcu: #01110-....: (6222689 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=203061910/203068959 fqs=1457949 Nov 2 21:58:42 Hydra kernel: rcu: #011 (t=6000105 jiffies g=19189 q=8) Nov 2 21:58:42 Hydra kernel: NMI backtrace for cpu 10 Nov 2 21:58:42 Hydra kernel: CPU: 10 PID: 435 Comm: smbd Tainted: G W O 4.19.107-Unraid #1 Nov 2 21:58:42 Hydra kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P3.30 11/04/2019 Nov 2 21:58:42 Hydra kernel: Call Trace: Nov 2 21:58:42 Hydra kernel: <IRQ> Nov 2 21:58:42 Hydra kernel: dump_stack+0x67/0x83 Nov 2 21:58:42 Hydra kernel: nmi_cpu_backtrace+0x71/0x83 Nov 2 21:58:42 Hydra kernel: ? lapic_can_unplug_cpu+0x97/0x97 Nov 2 21:58:42 Hydra kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Nov 2 21:58:42 Hydra kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Nov 2 21:58:42 Hydra kernel: rcu_check_callbacks+0x296/0x5a0 Nov 2 21:58:42 Hydra kernel: update_process_times+0x24/0x47 Nov 2 21:58:42 Hydra kernel: tick_sched_timer+0x36/0x64 Nov 2 21:58:42 Hydra kernel: __hrtimer_run_queues+0xb7/0x10b Nov 2 21:58:42 Hydra kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Nov 2 21:58:42 Hydra kernel: hrtimer_interrupt+0xf4/0x20e Nov 2 21:58:42 Hydra kernel: smp_apic_timer_interrupt+0x7b/0x93 Nov 2 21:58:42 Hydra kernel: apic_timer_interrupt+0xf/0x20 Nov 2 21:58:42 Hydra kernel: </IRQ> Nov 2 21:58:42 Hydra kernel: RIP: 0010:radix_tree_descend+0x13/0x57 Nov 2 21:58:42 Hydra kernel: Code: 4c 89 c2 48 8b 42 08 4c 89 4a 08 48 89 57 18 48 89 47 20 4c 89 08 c3 0f b6 0f 48 89 d0 48 d3 e8 83 e0 3f 89 c2 48 8d 54 d7 28 <48> 8b 12 48 89 d1 83 e1 03 48 ff c9 75 32 48 8d 4f 28 48 39 ca 72 Nov 2 21:58:42 Hydra kernel: RSP: 0018:ffffc9000c657c70 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 Nov 2 21:58:42 Hydra kernel: RAX: 0000000000000014 RBX: 000000000001f518 RCX: 0000000000000006 Nov 2 21:58:42 Hydra kernel: RDX: ffff8881060900c8 RSI: ffffc9000c657c78 RDI: ffff888106090000 Nov 2 21:58:42 Hydra kernel: RBP: ffff888151151078 R08: ffff888102ff8ed0 R09: ffffc9000c657ca8 Nov 2 21:58:42 Hydra kernel: R10: 0000000000000000 R11: ffff888151151070 R12: 00000000006200ca Nov 2 21:58:42 Hydra kernel: R13: ffff888151151068 R14: 0000000000001000 R15: 000000000001f518 Nov 2 21:58:42 Hydra kernel: __radix_tree_lookup+0x69/0xa2 Nov 2 21:58:42 Hydra kernel: radix_tree_lookup_slot+0x1e/0x41 Nov 2 21:58:42 Hydra kernel: find_get_entry+0x14/0x8f Nov 2 21:58:42 Hydra kernel: pagecache_get_page+0x20/0x1bd Nov 2 21:58:42 Hydra kernel: grab_cache_page_write_begin+0x1a/0x31 Nov 2 21:58:42 Hydra kernel: fuse_perform_write+0x178/0x43a Nov 2 21:58:42 Hydra kernel: ? file_remove_privs+0x55/0xb9 Nov 2 21:58:42 Hydra kernel: fuse_file_write_iter+0x1b6/0x22f Nov 2 21:58:42 Hydra kernel: __vfs_write+0xfc/0x13a Nov 2 21:58:42 Hydra kernel: vfs_write+0xc7/0x166 Nov 2 21:58:42 Hydra kernel: ksys_pwrite64+0x5d/0x79 Nov 2 21:58:42 Hydra kernel: do_syscall_64+0x57/0xf2 Nov 2 21:58:42 Hydra kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 2 21:58:42 Hydra kernel: RIP: 0033:0x14d2885a3ed7 Nov 2 21:58:42 Hydra kernel: Code: 08 89 3c 24 48 89 4c 24 18 e8 05 f3 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 35 f3 ff ff 48 8b Nov 2 21:58:42 Hydra kernel: RSP: 002b:000014d2867e8bf0 EFLAGS: 00000293 ORIG_RAX: 0000000000000012 Nov 2 21:58:42 Hydra kernel: RAX: ffffffffffffffda RBX: 000055f7b41e7890 RCX: 000014d2885a3ed7 Nov 2 21:58:42 Hydra kernel: RDX: 0000000000100000 RSI: 000055f7b4631850 RDI: 000000000000001f Nov 2 21:58:42 Hydra kernel: RBP: 000055f7b3e7fff0 R08: 0000000000000000 R09: 00000000ffffffff Nov 2 21:58:42 Hydra kernel: R10: 000000001f500000 R11: 0000000000000293 R12: 000014d2867e8c70 Nov 2 21:58:42 Hydra kernel: R13: 000055f7b3e80028 R14: 000055f7b3efaf50 R15: 000014d289f850a0 hydra-diagnostics-20201103-1033.zip syslog 2 nov 2020.log Quote Link to comment
JorgeB Posted November 3, 2020 Share Posted November 3, 2020 Make sure you're using the correct "Power Supply Idle Control", see here. 1 Quote Link to comment
EarthYak Posted November 3, 2020 Author Share Posted November 3, 2020 I have made that change, I will see what happens. Thanks Quote Link to comment
EarthYak Posted December 8, 2020 Author Share Posted December 8, 2020 I have given it a month to assess the change, I have set Power Supply Idle Control to typical and my RAM is at 2400 with 4 sticks so I think from the tables that is fine. I have still had at least one crash each week since then. Can anyone point me where to go next? Quote Link to comment
JorgeB Posted December 8, 2020 Share Posted December 8, 2020 One thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.