[6.9.2] rcu_sched detected stalls on CPUs/tasks


VlarpNL

Recommended Posts

Hi guys,

 

My Unraid server hangs every few days. I grabbed the below info from syslog showing the trace. I'm not a Linux guru at all so I have no clue what to look at that shows what happened. Any insights would be appreciated. Thanks!

 

2021-08-11 10:20:23	Kernel.Error	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
2021-08-11 10:20:23	Kernel.Error	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: rcu: 	1-...0: (5 ticks this GP) idle=586/1/0x4000000000000000 softirq=41567442/41567443 fqs=14976 
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: 	(detected by 0, t=60002 jiffies, g=65020929, q=2400)
2021-08-11 10:20:23	Kernel.Info	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Sending NMI from CPU 0 to CPUs 1:
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: NMI backtrace for cpu 1
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: CPU: 1 PID: 17600 Comm: Tdarr_Node Tainted: P           O      5.10.28-Unraid #1
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Hardware name: HPE ProLiant MicroServer Gen10/ProLiant MicroServer Gen10, BIOS 5.12 06/26/2018
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x18a
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Code: c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 47 01 00 eb 14 85 c0 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c0 00 30 02 00 65 48 03 05 f0 8e f8
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RSP: 0018:ffffc9000015ce68 EFLAGS: 00000002
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RAX: 0000000000040101 RBX: ffffc9000015ce90 RCX: 0000000000000000
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881000a9d40
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RBP: 8000015d00006c0d R08: ffffffff82013888 R09: ffffffff82013580
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: R10: 0001370722c9b2fd R11: 0000972caf555a8f R12: 00008044bfcf4231
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: R13: 0000000000000001 R14: 000000003b9aca00 R15: ffff8887ef49f040
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: FS:  000014f8eece3780(0000) GS:ffff8887ef480000(0000) knlGS:0000000000000000
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: CR2: 000017d29dc3b000 CR3: 0000000217226000 CR4: 00000000001506e0
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Call Trace:
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: <IRQ>
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: queued_spin_lock_slowpath+0x7/0xa
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: nr_blockdev_pages+0x13/0x64
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: si_meminfo+0x3a/0x57
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Sys_MemInfo+0x20/0x9b [corefreqk]
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? _raw_spin_unlock_irqrestore+0xd/0xe
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? try_to_wake_up+0x1b0/0x1e5
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? timekeeping_get_ns+0x19/0x2f
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? Sys_DumpTask+0xe9/0xf1 [corefreqk]
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Cycle_AMD_Family_15h+0x265/0x3e6 [corefreqk]
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: __hrtimer_run_queues+0xb7/0x10b
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? Core_AMD_Family_15h_Temp+0x8b/0x8b [corefreqk]
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: hrtimer_interrupt+0x8d/0x15b
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: __sysvec_apic_timer_interrupt+0x5d/0x68
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: asm_call_irq_on_stack+0x12/0x20
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: </IRQ>
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: sysvec_apic_timer_interrupt+0x71/0x95
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RIP: 0010:nr_blockdev_pages+0x4c/0x64
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Code: 00 00 48 8d 97 48 05 00 00 48 2d 10 01 00 00 48 8d 88 10 01 00 00 48 39 d1 74 17 48 8b 48 30 48 8b 80 10 01 00 00 4c 03 41 58 <48> 2d 10 01 00 00 eb dd 48 81 c7 40 05 00 00 e8 fa de ff ff 4c 89
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RSP: 0018:ffffc90001adfcd8 EFLAGS: 00000202
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RAX: ffff888100444dc8 RBX: ffffc90001adfd20 RCX: ffff888100445128
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RDX: ffff8881000a9d48 RSI: 0000000000000001 RDI: ffff8881000a9800
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RBP: ffff888100a8e4b0 R08: 000000000000005b R09: ffff8883f6308400
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: R10: 0000000000001008 R11: 0000000000000001 R12: ffffc90001adfe80
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: R13: 0000000000000001 R14: ffff888100a8e4d8 R15: ffff888100a8e4b0
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? nr_blockdev_pages+0x13/0x64
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: si_meminfo+0x3a/0x57
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: meminfo_proc_show+0x33/0x540
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? __mod_memcg_state+0x5/0x9d
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? mod_objcg_state+0x2a/0x30
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? slab_post_alloc_hook+0x10c/0x14a
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ? __kmalloc_node+0x144/0x16d
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: seq_read_iter+0x15e/0x339
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: proc_reg_read_iter+0x4d/0x5f
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: new_sync_read+0x77/0xaa
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: vfs_read+0xbe/0xff
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: ksys_read+0x71/0xba
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: do_syscall_64+0x5d/0x6a
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RIP: 0033:0x14f8ee95636c
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 89 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 30 44 89 c7 48 89 44 24 08 e8 bf fc ff ff 48
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RSP: 002b:00007ffc2983d500 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RAX: ffffffffffffffda RBX: 0000000000000017 RCX: 000014f8ee95636c
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RDX: 0000000000000fff RSI: 00007ffc2983d540 RDI: 0000000000000017
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: RBP: 00007ffc2983e560 R08: 0000000000000000 R09: 0000339d65e80471
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000017d29dc3b299
2021-08-11 10:20:23	Kernel.Warning	192.168.1.4	Aug 11 10:20:22 VLARPSERV kernel: R13: 0000000001ff3a48 R14: 000000000435ee88 R15: 00000000042d5658

 

Link to comment
43 minutes ago, JorgeB said:

You should always post the complete diagnostics, are you using the corefreq plugin? If yes try without it.

Yeah, I was just going through the logs and didn't think to post the full diagnostics, sorry about that. I had the corefreq plugin installed but I removed it. I'm not completely sure if I removed it before the hang or not.

 

I attached the diagnostics but I just checked and it doesn't seem to contain the moment the hang/crash occured. The only thing that contains that is the log saved by my syslog server. I attached that log as well.

vlarpserv-diagnostics-20210811-2054.zip Syslog.txt

Link to comment
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.