Unraid around 1-3 times a week at apparently random times is stopping and becoming unresponsive, the syslog shows this before it stops until I pull the power and restart. I restarted it the next day. Does anyone have any ideas, help would be gratefully appreciated. 

 

Excerpt from the syslog, full log for the day attached:

Nov  2 20:16:13 Hydra kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Nov  2 20:16:13 Hydra kernel: rcu: #01110-....: (59998 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=203068959/203068959 fqs=14300 
Nov  2 20:16:13 Hydra kernel: rcu: #011 (t=60001 jiffies g=284290801 q=1671571)
Nov  2 20:16:13 Hydra kernel: NMI backtrace for cpu 10
Nov  2 20:16:13 Hydra kernel: CPU: 10 PID: 435 Comm: smbd Tainted: G        W  O      4.19.107-Unraid #1
Nov  2 20:16:13 Hydra kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P3.30 11/04/2019
Nov  2 20:16:13 Hydra kernel: Call Trace:
Nov  2 20:16:13 Hydra kernel: <IRQ>
Nov  2 20:16:13 Hydra kernel: dump_stack+0x67/0x83
Nov  2 20:16:13 Hydra kernel: nmi_cpu_backtrace+0x71/0x83
Nov  2 20:16:13 Hydra kernel: ? lapic_can_unplug_cpu+0x97/0x97
Nov  2 20:16:13 Hydra kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Nov  2 20:16:13 Hydra kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Nov  2 20:16:13 Hydra kernel: rcu_check_callbacks+0x296/0x5a0
Nov  2 20:16:13 Hydra kernel: update_process_times+0x24/0x47
Nov  2 20:16:13 Hydra kernel: tick_sched_timer+0x36/0x64
Nov  2 20:16:13 Hydra kernel: __hrtimer_run_queues+0xb7/0x10b
Nov  2 20:16:13 Hydra kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Nov  2 20:16:13 Hydra kernel: hrtimer_interrupt+0xf4/0x20e
Nov  2 20:16:13 Hydra kernel: smp_apic_timer_interrupt+0x7b/0x93
Nov  2 20:16:13 Hydra kernel: apic_timer_interrupt+0xf/0x20
Nov  2 20:16:13 Hydra kernel: </IRQ>

Nov  2 21:58:13 Hydra kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Nov  2 21:58:13 Hydra kernel: rcu: #01110-....: (6180101 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=203068959/203068959 fqs=1502221 
Nov  2 21:58:13 Hydra kernel: rcu: #011 (t=6180104 jiffies g=284290801 q=40981941)
Nov  2 21:58:13 Hydra kernel: NMI backtrace for cpu 10
Nov  2 21:58:13 Hydra kernel: CPU: 10 PID: 435 Comm: smbd Tainted: G        W  O      4.19.107-Unraid #1
Nov  2 21:58:13 Hydra kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P3.30 11/04/2019
Nov  2 21:58:13 Hydra kernel: Call Trace:
Nov  2 21:58:13 Hydra kernel: <IRQ>
Nov  2 21:58:13 Hydra kernel: dump_stack+0x67/0x83
Nov  2 21:58:13 Hydra kernel: nmi_cpu_backtrace+0x71/0x83
Nov  2 21:58:13 Hydra kernel: ? lapic_can_unplug_cpu+0x97/0x97
Nov  2 21:58:13 Hydra kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Nov  2 21:58:13 Hydra kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Nov  2 21:58:13 Hydra kernel: rcu_check_callbacks+0x296/0x5a0
Nov  2 21:58:13 Hydra kernel: update_process_times+0x24/0x47
Nov  2 21:58:13 Hydra kernel: tick_sched_timer+0x36/0x64
Nov  2 21:58:13 Hydra kernel: __hrtimer_run_queues+0xb7/0x10b
Nov  2 21:58:13 Hydra kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Nov  2 21:58:13 Hydra kernel: hrtimer_interrupt+0xf4/0x20e
Nov  2 21:58:13 Hydra kernel: smp_apic_timer_interrupt+0x7b/0x93
Nov  2 21:58:13 Hydra kernel: apic_timer_interrupt+0xf/0x20
Nov  2 21:58:13 Hydra kernel: </IRQ>
Nov  2 21:58:13 Hydra kernel: RIP: 0010:radix_tree_descend+0x16/0x57
Nov  2 21:58:13 Hydra kernel: Code: 48 8b 42 08 4c 89 4a 08 48 89 57 18 48 89 47 20 4c 89 08 c3 0f b6 0f 48 89 d0 48 d3 e8 83 e0 3f 89 c2 48 8d 54 d7 28 48 8b 12 <48> 89 d1 83 e1 03 48 ff c9 75 32 48 8d 4f 28 48 39 ca 72 29 4c 8d
Nov  2 21:58:13 Hydra kernel: RSP: 0018:ffffc9000c657c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Nov  2 21:58:13 Hydra kernel: RAX: 000000000000001f RBX: 000000000001f518 RCX: 000000000000000c
Nov  2 21:58:13 Hydra kernel: RDX: ffff888106090001 RSI: ffffc9000c657c78 RDI: ffff888102ff8db0
Nov  2 21:58:13 Hydra kernel: RBP: ffff888151151078 R08: ffff888151151078 R09: ffffc9000c657ca8
Nov  2 21:58:13 Hydra kernel: R10: 0000000000000000 R11: ffff888151151070 R12: 00000000006200ca
Nov  2 21:58:13 Hydra kernel: R13: ffff888151151068 R14: 0000000000001000 R15: 000000000001f518
Nov  2 21:58:13 Hydra kernel: __radix_tree_lookup+0x69/0xa2
Nov  2 21:58:13 Hydra kernel: radix_tree_lookup_slot+0x1e/0x41
Nov  2 21:58:13 Hydra kernel: find_get_entry+0x14/0x8f
Nov  2 21:58:13 Hydra kernel: pagecache_get_page+0x20/0x1bd
Nov  2 21:58:13 Hydra kernel: grab_cache_page_write_begin+0x1a/0x31
Nov  2 21:58:13 Hydra kernel: fuse_perform_write+0x178/0x43a
Nov  2 21:58:13 Hydra kernel: ? file_remove_privs+0x55/0xb9
Nov  2 21:58:13 Hydra kernel: fuse_file_write_iter+0x1b6/0x22f
Nov  2 21:58:13 Hydra kernel: __vfs_write+0xfc/0x13a
Nov  2 21:58:13 Hydra kernel: vfs_write+0xc7/0x166
Nov  2 21:58:13 Hydra kernel: ksys_pwrite64+0x5d/0x79
Nov  2 21:58:13 Hydra kernel: do_syscall_64+0x57/0xf2
Nov  2 21:58:13 Hydra kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov  2 21:58:13 Hydra kernel: RIP: 0033:0x14d2885a3ed7
Nov  2 21:58:13 Hydra kernel: Code: 08 89 3c 24 48 89 4c 24 18 e8 05 f3 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 35 f3 ff ff 48 8b
Nov  2 21:58:13 Hydra kernel: RSP: 002b:000014d2867e8bf0 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
Nov  2 21:58:13 Hydra kernel: RAX: ffffffffffffffda RBX: 000055f7b41e7890 RCX: 000014d2885a3ed7
Nov  2 21:58:13 Hydra kernel: RDX: 0000000000100000 RSI: 000055f7b4631850 RDI: 000000000000001f
Nov  2 21:58:13 Hydra kernel: RBP: 000055f7b3e7fff0 R08: 0000000000000000 R09: 00000000ffffffff
Nov  2 21:58:13 Hydra kernel: R10: 000000001f500000 R11: 0000000000000293 R12: 000014d2867e8c70
Nov  2 21:58:13 Hydra kernel: R13: 000055f7b3e80028 R14: 000055f7b3efaf50 R15: 000014d289f850a0
Nov  2 21:58:42 Hydra kernel: rcu: INFO: rcu_bh self-detected stall on CPU
Nov  2 21:58:42 Hydra kernel: rcu: #01110-....: (6222689 ticks this GP) idle=6ea/1/0x4000000000000002 softirq=203061910/203068959 fqs=1457949 
Nov  2 21:58:42 Hydra kernel: rcu: #011 (t=6000105 jiffies g=19189 q=8)
Nov  2 21:58:42 Hydra kernel: NMI backtrace for cpu 10
Nov  2 21:58:42 Hydra kernel: CPU: 10 PID: 435 Comm: smbd Tainted: G        W  O      4.19.107-Unraid #1
Nov  2 21:58:42 Hydra kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P3.30 11/04/2019
Nov  2 21:58:42 Hydra kernel: Call Trace:
Nov  2 21:58:42 Hydra kernel: <IRQ>
Nov  2 21:58:42 Hydra kernel: dump_stack+0x67/0x83
Nov  2 21:58:42 Hydra kernel: nmi_cpu_backtrace+0x71/0x83
Nov  2 21:58:42 Hydra kernel: ? lapic_can_unplug_cpu+0x97/0x97
Nov  2 21:58:42 Hydra kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Nov  2 21:58:42 Hydra kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Nov  2 21:58:42 Hydra kernel: rcu_check_callbacks+0x296/0x5a0
Nov  2 21:58:42 Hydra kernel: update_process_times+0x24/0x47
Nov  2 21:58:42 Hydra kernel: tick_sched_timer+0x36/0x64
Nov  2 21:58:42 Hydra kernel: __hrtimer_run_queues+0xb7/0x10b
Nov  2 21:58:42 Hydra kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Nov  2 21:58:42 Hydra kernel: hrtimer_interrupt+0xf4/0x20e
Nov  2 21:58:42 Hydra kernel: smp_apic_timer_interrupt+0x7b/0x93
Nov  2 21:58:42 Hydra kernel: apic_timer_interrupt+0xf/0x20
Nov  2 21:58:42 Hydra kernel: </IRQ>
Nov  2 21:58:42 Hydra kernel: RIP: 0010:radix_tree_descend+0x13/0x57
Nov  2 21:58:42 Hydra kernel: Code: 4c 89 c2 48 8b 42 08 4c 89 4a 08 48 89 57 18 48 89 47 20 4c 89 08 c3 0f b6 0f 48 89 d0 48 d3 e8 83 e0 3f 89 c2 48 8d 54 d7 28 <48> 8b 12 48 89 d1 83 e1 03 48 ff c9 75 32 48 8d 4f 28 48 39 ca 72
Nov  2 21:58:42 Hydra kernel: RSP: 0018:ffffc9000c657c70 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
Nov  2 21:58:42 Hydra kernel: RAX: 0000000000000014 RBX: 000000000001f518 RCX: 0000000000000006
Nov  2 21:58:42 Hydra kernel: RDX: ffff8881060900c8 RSI: ffffc9000c657c78 RDI: ffff888106090000
Nov  2 21:58:42 Hydra kernel: RBP: ffff888151151078 R08: ffff888102ff8ed0 R09: ffffc9000c657ca8
Nov  2 21:58:42 Hydra kernel: R10: 0000000000000000 R11: ffff888151151070 R12: 00000000006200ca
Nov  2 21:58:42 Hydra kernel: R13: ffff888151151068 R14: 0000000000001000 R15: 000000000001f518
Nov  2 21:58:42 Hydra kernel: __radix_tree_lookup+0x69/0xa2
Nov  2 21:58:42 Hydra kernel: radix_tree_lookup_slot+0x1e/0x41
Nov  2 21:58:42 Hydra kernel: find_get_entry+0x14/0x8f
Nov  2 21:58:42 Hydra kernel: pagecache_get_page+0x20/0x1bd
Nov  2 21:58:42 Hydra kernel: grab_cache_page_write_begin+0x1a/0x31
Nov  2 21:58:42 Hydra kernel: fuse_perform_write+0x178/0x43a
Nov  2 21:58:42 Hydra kernel: ? file_remove_privs+0x55/0xb9
Nov  2 21:58:42 Hydra kernel: fuse_file_write_iter+0x1b6/0x22f
Nov  2 21:58:42 Hydra kernel: __vfs_write+0xfc/0x13a
Nov  2 21:58:42 Hydra kernel: vfs_write+0xc7/0x166
Nov  2 21:58:42 Hydra kernel: ksys_pwrite64+0x5d/0x79
Nov  2 21:58:42 Hydra kernel: do_syscall_64+0x57/0xf2
Nov  2 21:58:42 Hydra kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov  2 21:58:42 Hydra kernel: RIP: 0033:0x14d2885a3ed7
Nov  2 21:58:42 Hydra kernel: Code: 08 89 3c 24 48 89 4c 24 18 e8 05 f3 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 35 f3 ff ff 48 8b
Nov  2 21:58:42 Hydra kernel: RSP: 002b:000014d2867e8bf0 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
Nov  2 21:58:42 Hydra kernel: RAX: ffffffffffffffda RBX: 000055f7b41e7890 RCX: 000014d2885a3ed7
Nov  2 21:58:42 Hydra kernel: RDX: 0000000000100000 RSI: 000055f7b4631850 RDI: 000000000000001f
Nov  2 21:58:42 Hydra kernel: RBP: 000055f7b3e7fff0 R08: 0000000000000000 R09: 00000000ffffffff
Nov  2 21:58:42 Hydra kernel: R10: 000000001f500000 R11: 0000000000000293 R12: 000014d2867e8c70
Nov  2 21:58:42 Hydra kernel: R13: 000055f7b3e80028 R14: 000055f7b3efaf50 R15: 000014d289f850a0

hydra-diagnostics-20201103-1033.zip syslog 2 nov 2020.log

Make sure you're using the correct "Power Supply Idle Control", see here.

I have made that change, I will see what happens. Thanks

I have given it a month to assess the change, I have set Power Supply Idle Control to typical and my RAM is at 2400 with 4 sticks so I think from the tables that is fine. I have still had at least one crash each week since then. Can anyone point me where to go next?

One thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.