[6.8.2] All Network Access Lost After A Few Days (rcu_sched self-detected stall on CPU)

[DarkIceq]

Hi All

 

I have Unraid (6.8.2) on a Dell R710 and recently have been experiencing issues with complete loss of network connectivity that require a reboot to get working again.  The times seem to vary between about a 1.5 days and a week or so with no obvious pattern. After the last time it happened (last weekend) I enabled logging syslog to the flash drive to preserve the logs (which I have attached). The lockup starts at line 1607, Oct 21 17:05. Snippet below:

 

Oct 21 17:05:45 NAS03 kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Oct 21 17:05:45 NAS03 kernel: rcu: 	4-....: (60000 ticks this GP) idle=49e/1/0x4000000000000002 softirq=11398145/11398147 fqs=14691 
Oct 21 17:05:45 NAS03 kernel: rcu: 	 (t=60000 jiffies g=64650489 q=55080)
Oct 21 17:05:45 NAS03 kernel: NMI backtrace for cpu 4
Oct 21 17:05:45 NAS03 kernel: CPU: 4 PID: 22170 Comm: kworker/u49:3 Tainted: G        W I       4.19.98-Unraid #1
Oct 21 17:05:45 NAS03 kernel: Hardware name: Dell Inc. PowerEdge R710/0PV9DG, BIOS 6.4.0 07/23/2013
Oct 21 17:05:45 NAS03 kernel: Workqueue: events_power_efficient gc_worker
Oct 21 17:05:45 NAS03 kernel: Call Trace:
Oct 21 17:05:45 NAS03 kernel: <IRQ>
Oct 21 17:05:45 NAS03 kernel: dump_stack+0x67/0x83
Oct 21 17:05:45 NAS03 kernel: nmi_cpu_backtrace+0x71/0x83
Oct 21 17:05:45 NAS03 kernel: ? lapic_can_unplug_cpu+0x97/0x97
Oct 21 17:05:45 NAS03 kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Oct 21 17:05:45 NAS03 kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Oct 21 17:05:45 NAS03 kernel: rcu_check_callbacks+0x296/0x5a0
Oct 21 17:05:45 NAS03 kernel: update_process_times+0x24/0x47
Oct 21 17:05:45 NAS03 kernel: tick_sched_timer+0x36/0x64
Oct 21 17:05:45 NAS03 kernel: __hrtimer_run_queues+0xb7/0x10b
Oct 21 17:05:45 NAS03 kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Oct 21 17:05:45 NAS03 kernel: hrtimer_interrupt+0xf4/0x20e
Oct 21 17:05:45 NAS03 kernel: smp_apic_timer_interrupt+0x7b/0x93
Oct 21 17:05:45 NAS03 kernel: apic_timer_interrupt+0xf/0x20
Oct 21 17:05:45 NAS03 kernel: </IRQ>
Oct 21 17:05:45 NAS03 kernel: RIP: 0010:gc_worker+0xb4/0x270
Oct 21 17:05:45 NAS03 kernel: Code: 00 00 41 0f b6 46 37 49 c7 c0 f0 ff ff ff 41 ff c5 48 6b c0 38 49 29 c0 4f 8d 3c 06 49 8b 97 80 00 00 00 41 8b 87 88 00 00 00 <0f> ba e2 0e 73 2c 48 8b 15 18 e1 88 00 29 d0 3d ff 2d 93 02 0f 8f
Oct 21 17:05:45 NAS03 kernel: RSP: 0018:ffffc90007013e60 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
Oct 21 17:05:45 NAS03 kernel: RAX: 00000000337b24ae RBX: 0000000000000000 RCX: 0000000000010000
Oct 21 17:05:45 NAS03 kernel: RDX: 0000000000000188 RSI: 0000000000000166 RDI: ffff889216c41980
Oct 21 17:05:45 NAS03 kernel: RBP: 0000000000004b67 R08: ffffffffffffffb8 R09: ffffffff81574800
Oct 21 17:05:45 NAS03 kernel: R10: ffffea001dcb9680 R11: ffff889216c41901 R12: ffffffff822aa760
Oct 21 17:05:45 NAS03 kernel: R13: 000000001595f22c R14: ffff8891179f2e08 R15: ffff8891179f2dc0
Oct 21 17:05:45 NAS03 kernel: ? nf_ct_tmpl_alloc+0x13/0x52
Oct 21 17:05:45 NAS03 kernel: process_one_work+0x16e/0x24f
Oct 21 17:05:45 NAS03 kernel: worker_thread+0x1e2/0x2b8
Oct 21 17:05:45 NAS03 kernel: ? rescuer_thread+0x2a7/0x2a7
Oct 21 17:05:45 NAS03 kernel: kthread+0x10c/0x114
Oct 21 17:05:45 NAS03 kernel: ? kthread_park+0x89/0x89
Oct 21 17:05:45 NAS03 kernel: ret_from_fork+0x35/0x40

Does anyone have any ideas what might be causing this? Is it likely to be software or hardware? If any more information would be useful, I'm happy to provide it. 

Thanks in advance. 

 

syslog

Edited October 21, 2020 by DarkIceq

[trurl]

6 minutes ago, DarkIceq said:

more information

Go to Tools-Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread 

[DarkIceq]

nas03-diagnostics-20201021-2227.zip

 

As requested.

[zgo]

I have the same exact problem with unraid 6.8.3 ! here is my diagnostics

borg-diagnostics-20210217-2328.zip syslog-1613151690

[zgo]

@DarkIceqhave you found the problem?

[DarkIceq]

Wish I could be some help but I upgraded to 6.8.3 and the issue went away.

[Michiel]

i have exactly the same problem. Every day it stalls now.

I try now to switch to a different nic. And writing the syslog to a temp directory.

Edited February 19, 2021 by Michiel

[zgo]

@Michielwhat hardware do you have ? I have an ASRock RACK X570D4U-2L2T ryzen7 and my router had been flashed with ddwrt.