I'm seeing repeated call traces during a parity check:

Nov  4 04:35:08 Mandaue kernel: INFO: rcu_sched self-detected stall on CPU
Nov  4 04:35:08 Mandaue kernel: 	0-....: (59998 ticks this GP) idle=ec6/1/4611686018427387906 softirq=500578/500578 fqs=14897 
Nov  4 04:35:08 Mandaue kernel: 	 (t=60000 jiffies g=166573 c=166572 q=27992)
Nov  4 04:35:08 Mandaue kernel: NMI backtrace for cpu 0
Nov  4 04:35:08 Mandaue kernel: CPU: 0 PID: 8868 Comm: unraidd Not tainted 4.18.15-unRAID #1
Nov  4 04:35:08 Mandaue kernel: Hardware name: System manufacturer System Product Name/A88X-PLUS, BIOS 3003 03/10/2016
Nov  4 04:35:08 Mandaue kernel: Call Trace:
Nov  4 04:35:08 Mandaue kernel: 
Nov  4 04:35:08 Mandaue kernel: dump_stack+0x5d/0x79
Nov  4 04:35:08 Mandaue kernel: nmi_cpu_backtrace+0x71/0x83
Nov  4 04:35:08 Mandaue kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
Nov  4 04:35:08 Mandaue kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7
Nov  4 04:35:08 Mandaue kernel: rcu_dump_cpu_stacks+0x91/0xbb
Nov  4 04:35:08 Mandaue kernel: rcu_check_callbacks+0x23f/0x5ca
Nov  4 04:35:08 Mandaue kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
Nov  4 04:35:08 Mandaue kernel: update_process_times+0x23/0x45
Nov  4 04:35:08 Mandaue kernel: tick_sched_timer+0x36/0x64
Nov  4 04:35:08 Mandaue kernel: __hrtimer_run_queues+0xb1/0x105
Nov  4 04:35:08 Mandaue kernel: hrtimer_interrupt+0xf4/0x20d
Nov  4 04:35:08 Mandaue kernel: smp_apic_timer_interrupt+0x79/0x89
Nov  4 04:35:08 Mandaue kernel: apic_timer_interrupt+0xf/0x20
Nov  4 04:35:08 Mandaue kernel: 
Nov  4 04:35:08 Mandaue kernel: RIP: 0010:raid6_sse24_gen_syndrome+0xe5/0x1b3
Nov  4 04:35:08 Mandaue kernel: Code: e4 66 45 0f fc f6 66 0f db e8 66 0f db f8 66 44 0f db e8 66 44 0f db f8 66 0f ef e5 66 0f ef f7 66 45 0f ef e5 66 45 0f ef f7 <48> 8b 0a 66 0f 6f 2c 01 66 42 0f 6f 3c 11 66 46 0f 6f 2c 01 66 46 
Nov  4 04:35:08 Mandaue kernel: RSP: 0018:ffffc90003177c80 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
Nov  4 04:35:08 Mandaue kernel: RAX: 0000000000000380 RBX: 0000000000000008 RCX: ffff8803c10fb000
Nov  4 04:35:08 Mandaue kernel: RDX: ffff8803c10875d0 RSI: 0000000000000380 RDI: 0000000000000006
Nov  4 04:35:08 Mandaue kernel: RBP: ffff880401e38000 R08: 00000000000003a0 R09: ffff8803c10875e0
Nov  4 04:35:08 Mandaue kernel: R10: 0000000000000390 R11: 00000000000003b0 R12: ffff880401e39000
Nov  4 04:35:08 Mandaue kernel: R13: 0000000000001000 R14: ffff8803c10875a0 R15: 0000000000000008
Nov  4 04:35:08 Mandaue kernel: check_parity+0x202/0x349 [md_mod]
Nov  4 04:35:08 Mandaue kernel: ? autoremove_wake_function+0x9/0x2a
Nov  4 04:35:08 Mandaue kernel: ? __wake_up_common+0xa5/0x121
Nov  4 04:35:08 Mandaue kernel: handle_stripe+0xe8a/0x1226 [md_mod]
Nov  4 04:35:08 Mandaue kernel: unraidd+0xbc/0x123 [md_mod]
Nov  4 04:35:08 Mandaue kernel: ? md_open+0x2c/0x2c [md_mod]
Nov  4 04:35:08 Mandaue kernel: md_thread+0xcc/0xf1 [md_mod]
Nov  4 04:35:08 Mandaue kernel: ? wait_woken+0x68/0x68
Nov  4 04:35:08 Mandaue kernel: kthread+0x10b/0x113
Nov  4 04:35:08 Mandaue kernel: ? kthread_flush_work_fn+0x9/0x9
Nov  4 04:35:08 Mandaue kernel: ret_from_fork+0x22/0x40

I've looked for a newer BIOS but there isn't one. Maybe a different kernel will help.

 

Diagnostics: mandaue-diagnostics-20181105-0349.zip

It happens to some users on recent releases, can be fixed by lowering the tunables, just md_sync_thresh is enough according to one recent case, keep trying slightly lower values until the call traces stop.

Thanks Johnnie, I'll give that a try.

My original tunables settings were as previously recommended for the Dell H310 HBA, namely

• nr_requests = 128
• md_num_stripes = 4096
• md_sync_window = 2048
• md_sync_thresh = 2000

I set md_sync_thresh to the default value of 192 while a parity check was underway and the call traces stopped. So, reasoning that my new optimal value must be somewhere between 192 and 2000, I tried 1000 and the call traces started again within a couple of minutes of making the change. I eventually narrowed it down: 700 was bad, 500 was good. So for the time being I've settled on a value of 600. I haven't changed any of the other settings. If this parity check finishes without any more call traces I'll reboot and run another parity check (non-correcting, of course), with the aim of not getting any call traces at all.

 

What I find interesting is that I'm also running a parity check on another server that is very similar in terms of hardware (same socket FM2+, same AMD X88 chipset on Asus motherboards - one ATX, one mATX, same Dell H310 HBA, similar Bulldozer processors, same 16 GiB RAM) and running identical versions of Unraid but that one works fine with md_sync_thresh = 2000.

15 minutes ago, John_M said:

What I find interesting is that I'm also running a parity check on another server that is very similar in terms of hardware (same socket FM2+, same AMD X88 chipset on Asus motherboards - one ATX, one mATX, same Dell H310 HBA, similar Bulldozer processors, same 16 GiB RAM) and running identical versions of Unraid but that one works fine with md_sync_thresh = 2000.

Yep, only some particular servers are affected, couldn't find anything in common for now.