Diagnosing Random Lockup


Recommended Posts

In general things have been pretty solid, but recently I've been having the system lock up every couple days, where the only apparent way to break through is to hard reset. (Or at least, the web UI will not respond... since the video card is setup for passthrough I don't know whether it spit a message on-screen.)

 

I can't think of anything that would've coincided with this starting to happen... I enabled the option to persist the syslog through reboots, attached is the syslog I captured today after having to hard reset again...

 

I see that from 2:45 PM until I reboot there is a trace message from the Linux RCU thread, about every three minutes...

Aug 14 14:51:36 Raiden kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug 14 14:51:36 Raiden kernel: rcu: 	1-....: (420005 ticks this GP) idle=1be/1/0x4000000000000002 softirq=1700269094/1700269094 fqs=102699 
Aug 14 14:51:36 Raiden kernel: rcu: 	 (t=420006 jiffies g=217067417 q=25007350)
Aug 14 14:51:36 Raiden kernel: NMI backtrace for cpu 1
Aug 14 14:51:36 Raiden kernel: CPU: 1 PID: 19660 Comm: kworker/u64:2 Tainted: G        W         4.19.56-Unraid #1
Aug 14 14:51:36 Raiden kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450M Pro4, BIOS P3.10 03/07/2019
Aug 14 14:51:36 Raiden kernel: Workqueue: events_power_efficient gc_worker
Aug 14 14:51:36 Raiden kernel: Call Trace:
Aug 14 14:51:36 Raiden kernel: <IRQ>
Aug 14 14:51:36 Raiden kernel: dump_stack+0x5d/0x79
Aug 14 14:51:36 Raiden kernel: nmi_cpu_backtrace+0x71/0x83
Aug 14 14:51:36 Raiden kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
Aug 14 14:51:36 Raiden kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7
Aug 14 14:51:36 Raiden kernel: rcu_dump_cpu_stacks+0x91/0xbb
Aug 14 14:51:36 Raiden kernel: rcu_check_callbacks+0x28f/0x58e
Aug 14 14:51:36 Raiden kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
Aug 14 14:51:36 Raiden kernel: update_process_times+0x23/0x45
Aug 14 14:51:36 Raiden kernel: tick_sched_timer+0x36/0x64
Aug 14 14:51:36 Raiden kernel: __hrtimer_run_queues+0xb1/0x105
Aug 14 14:51:36 Raiden kernel: hrtimer_interrupt+0xf4/0x20d
Aug 14 14:51:36 Raiden kernel: smp_apic_timer_interrupt+0x79/0x91
Aug 14 14:51:36 Raiden kernel: apic_timer_interrupt+0xf/0x20
Aug 14 14:51:36 Raiden kernel: </IRQ>
Aug 14 14:51:36 Raiden kernel: RIP: 0010:gc_worker+0xf6/0x258
Aug 14 14:51:36 Raiden kernel: Code: 02 0f 8f ec 00 00 00 48 8b 05 00 ea 8b 00 05 00 5c 26 05 41 89 87 88 00 00 00 e9 d4 00 00 00 48 8b 15 e8 e9 8b 00 29 d0 85 c0 <7f> 11 4c 89 ff e8 da e9 ff ff ff 44 24 08 e9 b6 00 00 00 85 ed 0f
Aug 14 14:51:36 Raiden kernel: RSP: 0018:ffffc90008af7e68 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
Aug 14 14:51:36 Raiden kernel: RAX: 000000001742ae36 RBX: ffffffff8228e440 RCX: ffffffffffffffb8
Aug 14 14:51:36 Raiden kernel: RDX: 00000001174f929c RSI: 0000000000000135 RDI: ffffffff8228e440
Aug 14 14:51:36 Raiden kernel: RBP: 0000000000000000 R08: 0000000000000018 R09: 0000746e65696369
Aug 14 14:51:36 Raiden kernel: R10: 8080808080808080 R11: fefefefefefefeff R12: 0000000000002f36
Aug 14 14:51:36 Raiden kernel: R13: 0000000064ac2ff9 R14: ffff8882e3b18f48 R15: ffff8882e3b18f00
Aug 14 14:51:36 Raiden kernel: ? gc_worker+0x1cc/0x258
Aug 14 14:51:36 Raiden kernel: process_one_work+0x16e/0x24f
Aug 14 14:51:36 Raiden kernel: ? pwq_unbound_release_workfn+0xb7/0xb7
Aug 14 14:51:36 Raiden kernel: worker_thread+0x1dc/0x2ac
Aug 14 14:51:36 Raiden kernel: kthread+0x10b/0x113
Aug 14 14:51:36 Raiden kernel: ? kthread_park+0x71/0x71
Aug 14 14:51:36 Raiden kernel: ret_from_fork+0x22/0x40
Aug 14 14:54:36 Raiden kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug 14 14:54:36 Raiden kernel: rcu: 	1-....: (600008 ticks this GP) idle=1be/1/0x4000000000000002 softirq=1700269094/1700269094 fqs=146764 
Aug 14 14:54:36 Raiden kernel: rcu: 	 (t=600009 jiffies g=217067417 q=35280857)
Aug 14 14:54:36 Raiden kernel: NMI backtrace for cpu 1
Aug 14 14:54:36 Raiden kernel: CPU: 1 PID: 19660 Comm: kworker/u64:2 Tainted: G        W         4.19.56-Unraid #1
Aug 14 14:54:36 Raiden kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450M Pro4, BIOS P3.10 03/07/2019
Aug 14 14:54:36 Raiden kernel: Workqueue: events_power_efficient gc_worker
Aug 14 14:54:36 Raiden kernel: Call Trace:
Aug 14 14:54:36 Raiden kernel: <IRQ>
Aug 14 14:54:36 Raiden kernel: dump_stack+0x5d/0x79
Aug 14 14:54:36 Raiden kernel: nmi_cpu_backtrace+0x71/0x83
Aug 14 14:54:36 Raiden kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
Aug 14 14:54:36 Raiden kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7
Aug 14 14:54:36 Raiden kernel: rcu_dump_cpu_stacks+0x91/0xbb
Aug 14 14:54:36 Raiden kernel: rcu_check_callbacks+0x28f/0x58e
Aug 14 14:54:36 Raiden kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
Aug 14 14:54:36 Raiden kernel: update_process_times+0x23/0x45
Aug 14 14:54:36 Raiden kernel: tick_sched_timer+0x36/0x64
Aug 14 14:54:36 Raiden kernel: __hrtimer_run_queues+0xb1/0x105
Aug 14 14:54:36 Raiden kernel: hrtimer_interrupt+0xf4/0x20d
Aug 14 14:54:36 Raiden kernel: smp_apic_timer_interrupt+0x79/0x91
Aug 14 14:54:36 Raiden kernel: apic_timer_interrupt+0xf/0x20
Aug 14 14:54:36 Raiden kernel: </IRQ>
Aug 14 14:54:36 Raiden kernel: RIP: 0010:gc_worker+0xa7/0x258
Aug 14 14:54:36 Raiden kernel: Code: 44 89 e0 48 8d 04 c2 4c 8b 30 41 f6 c6 01 0f 85 36 01 00 00 41 0f b6 46 37 48 c7 c1 f0 ff ff ff 41 ff c5 48 6b c0 38 48 29 c1 <4d> 8d 3c 0e 49 8b 97 80 00 00 00 41 8b 87 88 00 00 00 0f ba e2 0e
Aug 14 14:54:36 Raiden kernel: RSP: 0018:ffffc90008af7e68 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
Aug 14 14:54:36 Raiden kernel: RAX: 0000000000000038 RBX: ffffffff8228e440 RCX: ffffffffffffffb8
Aug 14 14:54:36 Raiden kernel: RDX: 00000001175251bf RSI: 0000000000000135 RDI: ffffffff8228e440
Aug 14 14:54:36 Raiden kernel: RBP: 0000000000000000 R08: 0000000000000018 R09: 0000746e65696369
Aug 14 14:54:36 Raiden kernel: R10: 8080808080808080 R11: fefefefefefefeff R12: 0000000000002f36
Aug 14 14:54:36 Raiden kernel: R13: 00000000dc4f32b4 R14: ffff8882e3b18f48 R15: ffff8882e3b18f00
Aug 14 14:54:36 Raiden kernel: ? gc_worker+0x1cc/0x258
Aug 14 14:54:36 Raiden kernel: process_one_work+0x16e/0x24f
Aug 14 14:54:36 Raiden kernel: ? pwq_unbound_release_workfn+0xb7/0xb7
Aug 14 14:54:36 Raiden kernel: worker_thread+0x1dc/0x2ac
Aug 14 14:54:36 Raiden kernel: kthread+0x10b/0x113
Aug 14 14:54:36 Raiden kernel: ? kthread_park+0x71/0x71
Aug 14 14:54:36 Raiden kernel: ret_from_fork+0x22/0x40

does anyone understand how to tell what it is saying is stalled and/or how I can work backward to find the culprit?

syslog

Link to comment
  • 1 year later...

I recently starting seeing similar and only changes made were various docker updates.

============================================================================

[151178.007884] rcu: INFO: rcu_sched self-detected stall on CPU
[151178.007889] rcu:    8-....: (4740077 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13368129/13368129 fqs=1174979
[151178.007889] rcu:     (t=4740078 jiffies g=37631889 q=17400053)
[151178.007891] NMI backtrace for cpu 8
[151178.007893] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151178.007894] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151178.007900] Workqueue: events_power_efficient gc_worker
[151178.007900] Call Trace:
[151178.007903]  <IRQ>
[151178.007905]  dump_stack+0x67/0x83
[151178.007906]  nmi_cpu_backtrace+0x71/0x83
[151178.007909]  ? lapic_can_unplug_cpu+0x97/0x97
[151178.007910]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151178.007913]  rcu_dump_cpu_stacks+0x8b/0xb4
[151178.007914]  rcu_check_callbacks+0x296/0x5a0
[151178.007915]  update_process_times+0x24/0x47
[151178.007917]  tick_sched_timer+0x36/0x64
[151178.007918]  __hrtimer_run_queues+0xb7/0x10b
[151178.007919]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151178.007920]  hrtimer_interrupt+0xf4/0x20e
[151178.007922]  smp_apic_timer_interrupt+0x7b/0x93
[151178.007923]  apic_timer_interrupt+0xf/0x20
[151178.007924]  </IRQ>
[151178.007925] RIP: 0010:gc_worker+0xf1/0x270
[151178.007926] Code: 02 0f 8f 00 01 00 00 48 8b 05 ba dc 88 00 05 00 5c 26 05 41 89 87 88 00 00 00 e9 e8 00 00 00 48 8b 15 a2 dc 88 00 29 d0 85 c0 <7f> 11 4c 89 ff e8 e1 e8 ff ff ff 44 24 08 e9 ca 00 00 00 85 db 0f
[151178.007926] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[151178.007927] RAX: 00000000086d964b RBX: 0000000000000000 RCX: 0000000000010000
[151178.007928] RDX: 0000000108fe35c3 RSI: 00000000000001f4 RDI: ffffffff822aa760
[151178.007929] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151178.007929] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151178.007930] R13: 00000000e55d5b1f R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151178.007933]  process_one_work+0x16e/0x24f
[151178.007934]  worker_thread+0x1e2/0x2b8
[151178.007935]  ? rescuer_thread+0x2a7/0x2a7
[151178.007937]  kthread+0x10c/0x114
[151178.007938]  ? kthread_park+0x89/0x89
[151178.007939]  ret_from_fork+0x22/0x40
[151284.033663] rcu: INFO: rcu_bh self-detected stall on CPU
[151284.033670] rcu:    8-....: (4848680 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13366809/13368129 fqs=1175347
[151284.033670] rcu:     (t=4740078 jiffies g=1601 q=20)
[151284.033673] NMI backtrace for cpu 8
[151284.033676] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151284.033677] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151284.033683] Workqueue: events_power_efficient gc_worker
[151284.033684] Call Trace:
[151284.033686]  <IRQ>
[151284.033690]  dump_stack+0x67/0x83
[151284.033692]  nmi_cpu_backtrace+0x71/0x83
[151284.033696]  ? lapic_can_unplug_cpu+0x97/0x97
[151284.033697]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151284.033700]  rcu_dump_cpu_stacks+0x8b/0xb4
[151284.033702]  rcu_check_callbacks+0x296/0x5a0
[151284.033704]  update_process_times+0x24/0x47
[151284.033706]  tick_sched_timer+0x36/0x64
[151284.033708]  __hrtimer_run_queues+0xb7/0x10b
[151284.033709]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151284.033711]  hrtimer_interrupt+0xf4/0x20e
[151284.033713]  smp_apic_timer_interrupt+0x7b/0x93
[151284.033714]  apic_timer_interrupt+0xf/0x20
[151284.033715]  </IRQ>
[151284.033717] RIP: 0010:gc_worker+0x1d1/0x270
[151284.033718] Code: 84 c0 75 ce 4d 85 ff 75 10 48 c7 c7 79 fc d2 81 e8 e1 6f b1 ff 0f 0b eb 0e f0 41 ff 0f 75 08 4c 89 ff e8 2f 6d ff ff 4d 8b 36 <e9> ac fe ff ff e8 a8 0b 11 00 ff 44 24 04 8b 74 24 04 39 74 24 0c
[151284.033719] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[151284.033721] RAX: 00000000086bf821 RBX: 0000000000000000 RCX: 0000000000010000
[151284.033722] RDX: 0000000108ffd3ed RSI: 00000000000001f4 RDI: ffffffff822aa760
[151284.033723] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151284.033724] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151284.033724] R13: 00000000f2afb653 R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151284.033728]  process_one_work+0x16e/0x24f
[151284.033730]  worker_thread+0x1e2/0x2b8
[151284.033732]  ? rescuer_thread+0x2a7/0x2a7
[151284.033734]  kthread+0x10c/0x114
[151284.033735]  ? kthread_park+0x89/0x89
[151284.033737]  ret_from_fork+0x22/0x40
[151358.010515] rcu: INFO: rcu_sched self-detected stall on CPU
[151358.010519] rcu:    8-....: (4920080 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13368129/13368129 fqs=1219827
[151358.010519] rcu:     (t=4920081 jiffies g=37631889 q=18113057)
[151358.010522] NMI backtrace for cpu 8
[151358.010523] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151358.010524] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151358.010529] Workqueue: events_power_efficient gc_worker
[151358.010530] Call Trace:
[151358.010532]  <IRQ>
[151358.010534]  dump_stack+0x67/0x83
[151358.010536]  nmi_cpu_backtrace+0x71/0x83
[151358.010539]  ? lapic_can_unplug_cpu+0x97/0x97
[151358.010540]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151358.010542]  rcu_dump_cpu_stacks+0x8b/0xb4
[151358.010544]  rcu_check_callbacks+0x296/0x5a0
[151358.010546]  update_process_times+0x24/0x47
[151358.010547]  tick_sched_timer+0x36/0x64
[151358.010548]  __hrtimer_run_queues+0xb7/0x10b
[151358.010549]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151358.010550]  hrtimer_interrupt+0xf4/0x20e
[151358.010552]  smp_apic_timer_interrupt+0x7b/0x93
[151358.010553]  apic_timer_interrupt+0xf/0x20
[151358.010554]  </IRQ>
[151358.010555] RIP: 0010:gc_worker+0xa2/0x270
[151358.010557] Code: ed 89 e8 48 8d 04 c2 4c 8b 30 41 f6 c6 01 0f 85 4a 01 00 00 41 0f b6 46 37 49 c7 c0 f0 ff ff ff 41 ff c5 48 6b c0 38 49 29 c0 <4f> 8d 3c 06 49 8b 97 80 00 00 00 41 8b 87 88 00 00 00 0f ba e2 0e
[151358.010557] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[151358.010559] RAX: 0000000000000038 RBX: 0000000000000000 RCX: 0000000000010000
[151358.010560] RDX: 000000010900f4e6 RSI: 00000000000001f4 RDI: ffffffff822aa760
[151358.010560] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151358.010561] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151358.010562] R13: 000000002aae350f R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151358.010565]  process_one_work+0x16e/0x24f
[151358.010566]  worker_thread+0x1e2/0x2b8
[151358.010567]  ? rescuer_thread+0x2a7/0x2a7
[151358.010568]  kthread+0x10c/0x114
[151358.010569]  ? kthread_park+0x89/0x89
[151358.010571]  ret_from_fork+0x22/0x40
[151464.036313] rcu: INFO: rcu_bh self-detected stall on CPU
[151464.036320] rcu:    8-....: (5028683 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13366809/13368129 fqs=1220148
[151464.036321] rcu:     (t=4920081 jiffies g=1601 q=20)
[151464.036325] NMI backtrace for cpu 8
[151464.036327] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151464.036328] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151464.036334] Workqueue: events_power_efficient gc_worker
[151464.036335] Call Trace:
[151464.036338]  <IRQ>
[151464.036342]  dump_stack+0x67/0x83
[151464.036344]  nmi_cpu_backtrace+0x71/0x83
[151464.036348]  ? lapic_can_unplug_cpu+0x97/0x97
[151464.036350]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151464.036352]  rcu_dump_cpu_stacks+0x8b/0xb4
[151464.036354]  rcu_check_callbacks+0x296/0x5a0
[151464.036356]  update_process_times+0x24/0x47
[151464.036358]  tick_sched_timer+0x36/0x64
[151464.036359]  __hrtimer_run_queues+0xb7/0x10b
[151464.036361]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151464.036362]  hrtimer_interrupt+0xf4/0x20e
[151464.036365]  smp_apic_timer_interrupt+0x7b/0x93
[151464.036366]  apic_timer_interrupt+0xf/0x20
[151464.036367]  </IRQ>
[151464.036369] RIP: 0010:gc_worker+0xad/0x270
[151464.036371] Code: f6 c6 01 0f 85 4a 01 00 00 41 0f b6 46 37 49 c7 c0 f0 ff ff ff 41 ff c5 48 6b c0 38 49 29 c0 4f 8d 3c 06 49 8b 97 80 00 00 00 <41> 8b 87 88 00 00 00 0f ba e2 0e 73 2c 48 8b 15 ce dc 88 00 29 d0
[151464.036372] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[151464.036373] RAX: 0000000000000038 RBX: 0000000000000000 RCX: 0000000000010000
[151464.036374] RDX: 0000000000000188 RSI: 00000000000001f4 RDI: ffffffff822aa760
[151464.036375] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151464.036376] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151464.036377] R13: 000000002a30fc3e R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151464.036381]  process_one_work+0x16e/0x24f
[151464.036382]  worker_thread+0x1e2/0x2b8
[151464.036384]  ? rescuer_thread+0x2a7/0x2a7
[151464.036386]  kthread+0x10c/0x114
[151464.036387]  ? kthread_park+0x89/0x89
[151464.036389]  ret_from_fork+0x22/0x40
[151538.013172] rcu: INFO: rcu_sched self-detected stall on CPU
[151538.013179] rcu:    8-....: (5100083 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13368129/13368129 fqs=1264604
[151538.013180] rcu:     (t=5100084 jiffies g=37631889 q=18817447)
[151538.013184] NMI backtrace for cpu 8
[151538.013187] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151538.013188] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151538.013194] Workqueue: events_power_efficient gc_worker
[151538.013195] Call Trace:
[151538.013197]  <IRQ>
[151538.013201]  dump_stack+0x67/0x83
[151538.013204]  nmi_cpu_backtrace+0x71/0x83
[151538.013208]  ? lapic_can_unplug_cpu+0x97/0x97
[151538.013210]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151538.013212]  rcu_dump_cpu_stacks+0x8b/0xb4
[151538.013214]  rcu_check_callbacks+0x296/0x5a0
[151538.013216]  update_process_times+0x24/0x47
[151538.013218]  tick_sched_timer+0x36/0x64
[151538.013220]  __hrtimer_run_queues+0xb7/0x10b
[151538.013222]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151538.013223]  hrtimer_interrupt+0xf4/0x20e
[151538.013226]  smp_apic_timer_interrupt+0x7b/0x93
[151538.013227]  apic_timer_interrupt+0xf/0x20
[151538.013229]  </IRQ>
[151538.013231] RIP: 0010:gc_worker+0xad/0x270
[151538.013232] Code: f6 c6 01 0f 85 4a 01 00 00 41 0f b6 46 37 49 c7 c0 f0 ff ff ff 41 ff c5 48 6b c0 38 49 29 c0 4f 8d 3c 06 49 8b 97 80 00 00 00 <41> 8b 87 88 00 00 00 0f ba e2 0e 73 2c 48 8b 15 ce dc 88 00 29 d0
[151538.013233] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13
[151538.013235] RAX: 0000000000000038 RBX: 0000000000000000 RCX: 0000000000010000
[151538.013236] RDX: 0000000000000188 RSI: 00000000000001f4 RDI: ffffffff822aa760
[151538.013237] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151538.013238] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151538.013238] R13: 00000000570d0e9f R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151538.013242]  process_one_work+0x16e/0x24f
[151538.013244]  worker_thread+0x1e2/0x2b8
[151538.013246]  ? rescuer_thread+0x2a7/0x2a7
[151538.013248]  kthread+0x10c/0x114
[151538.013250]  ? kthread_park+0x89/0x89
[151538.013252]  ret_from_fork+0x22/0x40
[151644.038968] rcu: INFO: rcu_bh self-detected stall on CPU
[151644.038974] rcu:    8-....: (5208686 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13366809/13368129 fqs=1264908
[151644.038974] rcu:     (t=5100084 jiffies g=1601 q=20)
[151644.038977] NMI backtrace for cpu 8
[151644.038980] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151644.038981] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151644.038987] Workqueue: events_power_efficient gc_worker
[151644.038988] Call Trace:
[151644.038991]  <IRQ>
[151644.038995]  dump_stack+0x67/0x83
[151644.038997]  nmi_cpu_backtrace+0x71/0x83
[151644.039001]  ? lapic_can_unplug_cpu+0x97/0x97
[151644.039003]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151644.039005]  rcu_dump_cpu_stacks+0x8b/0xb4
[151644.039007]  rcu_check_callbacks+0x296/0x5a0
[151644.039009]  update_process_times+0x24/0x47
[151644.039011]  tick_sched_timer+0x36/0x64
[151644.039013]  __hrtimer_run_queues+0xb7/0x10b
[151644.039014]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151644.039016]  hrtimer_interrupt+0xf4/0x20e
[151644.039019]  smp_apic_timer_interrupt+0x7b/0x93
[151644.039020]  apic_timer_interrupt+0xf/0x20
[151644.039021]  </IRQ>
[151644.039023] RIP: 0010:gc_worker+0x106/0x270
[151644.039025] Code: 87 88 00 00 00 e9 e8 00 00 00 48 8b 15 a2 dc 88 00 29 d0 85 c0 7f 11 4c 89 ff e8 e1 e8 ff ff ff 44 24 08 e9 ca 00 00 00 85 db <0f> 84 c2 00 00 00 49 8b 87 80 00 00 00 a8 08 0f 84 b3 00 00 00 49
[151644.039026] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[151644.039027] RAX: 00000000086679db RBX: 0000000000000000 RCX: 0000000000010000
[151644.039028] RDX: 0000000109055233 RSI: 00000000000001f4 RDI: ffffffff822aa760
[151644.039029] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151644.039030] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151644.039031] R13: 0000000054d121f6 R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151644.039035]  process_one_work+0x16e/0x24f
[151644.039037]  worker_thread+0x1e2/0x2b8
[151644.039039]  ? rescuer_thread+0x2a7/0x2a7
[151644.039040]  kthread+0x10c/0x114
[151644.039042]  ? kthread_park+0x89/0x89
[151644.039044]  ret_from_fork+0x22/0x40
[151718.015826] rcu: INFO: rcu_sched self-detected stall on CPU
[151718.015831] rcu:    8-....: (5280086 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13368129/13368129 fqs=1309359
[151718.015832] rcu:     (t=5280087 jiffies g=37631889 q=19518613)
[151718.015835] NMI backtrace for cpu 8
[151718.015837] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151718.015838] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151718.015844] Workqueue: events_power_efficient gc_worker
[151718.015845] Call Trace:
[151718.015848]  <IRQ>
[151718.015851]  dump_stack+0x67/0x83
[151718.015853]  nmi_cpu_backtrace+0x71/0x83
[151718.015857]  ? lapic_can_unplug_cpu+0x97/0x97
[151718.015858]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151718.015861]  rcu_dump_cpu_stacks+0x8b/0xb4
[151718.015864]  rcu_check_callbacks+0x296/0x5a0
[151718.015866]  update_process_times+0x24/0x47
[151718.015867]  tick_sched_timer+0x36/0x64
[151718.015869]  __hrtimer_run_queues+0xb7/0x10b
[151718.015871]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151718.015872]  hrtimer_interrupt+0xf4/0x20e
[151718.015875]  smp_apic_timer_interrupt+0x7b/0x93
[151718.015876]  apic_timer_interrupt+0xf/0x20
[151718.015877]  </IRQ>
[151718.015880] RIP: 0010:gc_worker+0x1d1/0x270
[151718.015881] Code: 84 c0 75 ce 4d 85 ff 75 10 48 c7 c7 79 fc d2 81 e8 e1 6f b1 ff 0f 0b eb 0e f0 41 ff 0f 75 08 4c 89 ff e8 2f 6d ff ff 4d 8b 36 <e9> ac fe ff ff e8 a8 0b 11 00 ff 44 24 04 8b 74 24 04 39 74 24 0c
[151718.015882] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[151718.015884] RAX: 00000000086558e2 RBX: 0000000000000000 RCX: 0000000000010000
[151718.015885] RDX: 000000010906732c RSI: 00000000000001f4 RDI: ffffffff822aa760
[151718.015886] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151718.015886] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151718.015887] R13: 00000000815a8e88 R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151718.015891]  process_one_work+0x16e/0x24f
[151718.015893]  worker_thread+0x1e2/0x2b8
[151718.015895]  ? rescuer_thread+0x2a7/0x2a7
[151718.015897]  kthread+0x10c/0x114
[151718.015899]  ? kthread_park+0x89/0x89
[151718.015901]  ret_from_fork+0x22/0x40
[151824.041622] rcu: INFO: rcu_bh self-detected stall on CPU
[151824.041628] rcu:    8-....: (5388689 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13366809/13368129 fqs=1309605
[151824.041629] rcu:     (t=5280087 jiffies g=1601 q=20)
[151824.041631] NMI backtrace for cpu 8
[151824.041634] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151824.041635] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151824.041641] Workqueue: events_power_efficient gc_worker
[151824.041642] Call Trace:
[151824.041645]  <IRQ>
[151824.041647]  dump_stack+0x67/0x83
[151824.041650]  nmi_cpu_backtrace+0x71/0x83
[151824.041654]  ? lapic_can_unplug_cpu+0x97/0x97
[151824.041655]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151824.041658]  rcu_dump_cpu_stacks+0x8b/0xb4
[151824.041660]  rcu_check_callbacks+0x296/0x5a0
[151824.041662]  update_process_times+0x24/0x47
[151824.041664]  tick_sched_timer+0x36/0x64
[151824.041667]  __hrtimer_run_queues+0xb7/0x10b
[151824.041668]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151824.041670]  hrtimer_interrupt+0xf4/0x20e
[151824.041672]  smp_apic_timer_interrupt+0x7b/0x93
[151824.041674]  apic_timer_interrupt+0xf/0x20
[151824.041675]  </IRQ>
[151824.041677] RIP: 0010:gc_worker+0xf1/0x270
[151824.041679] Code: 02 0f 8f 00 01 00 00 48 8b 05 ba dc 88 00 05 00 5c 26 05 41 89 87 88 00 00 00 e9 e8 00 00 00 48 8b 15 a2 dc 88 00 29 d0 85 c0 <7f> 11 4c 89 ff e8 e1 e8 ff ff ff 44 24 08 e9 ca 00 00 00 85 db 0f
[151824.041680] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[151824.041681] RAX: 000000000863bab8 RBX: 0000000000000000 RCX: 0000000000010000
[151824.041682] RDX: 0000000109081156 RSI: 00000000000001f4 RDI: ffffffff822aa760
[151824.041683] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151824.041684] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151824.041685] R13: 000000007b44b7d9 R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151824.041689]  process_one_work+0x16e/0x24f
[151824.041690]  worker_thread+0x1e2/0x2b8
[151824.041692]  ? rescuer_thread+0x2a7/0x2a7
[151824.041694]  kthread+0x10c/0x114
[151824.041696]  ? kthread_park+0x89/0x89
[151824.041698]  ret_from_fork+0x22/0x40
[151898.018479] rcu: INFO: rcu_sched self-detected stall on CPU
[151898.018485] rcu:    8-....: (5460089 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13368129/13368129 fqs=1354093
[151898.018485] rcu:     (t=5460090 jiffies g=37631889 q=20224901)
[151898.018488] NMI backtrace for cpu 8
[151898.018491] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[151898.018492] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[151898.018498] Workqueue: events_power_efficient gc_worker
[151898.018499] Call Trace:
[151898.018501]  <IRQ>
[151898.018504]  dump_stack+0x67/0x83
[151898.018507]  nmi_cpu_backtrace+0x71/0x83
[151898.018511]  ? lapic_can_unplug_cpu+0x97/0x97
[151898.018513]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[151898.018515]  rcu_dump_cpu_stacks+0x8b/0xb4
[151898.018518]  rcu_check_callbacks+0x296/0x5a0
[151898.018519]  update_process_times+0x24/0x47
[151898.018521]  tick_sched_timer+0x36/0x64
[151898.018523]  __hrtimer_run_queues+0xb7/0x10b
[151898.018525]  ? tick_sched_handle.isra.0+0x2f/0x2f
[151898.018526]  hrtimer_interrupt+0xf4/0x20e
[151898.018528]  smp_apic_timer_interrupt+0x7b/0x93
[151898.018530]  apic_timer_interrupt+0xf/0x20
[151898.018531]  </IRQ>
[151898.018533] RIP: 0010:gc_worker+0x1d1/0x270
[151898.018535] Code: 84 c0 75 ce 4d 85 ff 75 10 48 c7 c7 79 fc d2 81 e8 e1 6f b1 ff 0f 0b eb 0e f0 41 ff 0f 75 08 4c 89 ff e8 2f 6d ff ff 4d 8b 36 <e9> ac fe ff ff e8 a8 0b 11 00 ff 44 24 04 8b 74 24 04 39 74 24 0c
[151898.018536] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[151898.018537] RAX: 00000000086299bf RBX: 0000000000000000 RCX: 0000000000010000
[151898.018538] RDX: 000000010909324f RSI: 00000000000001f4 RDI: ffffffff822aa760
[151898.018539] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[151898.018540] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[151898.018541] R13: 00000000a82c71f9 R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[151898.018545]  process_one_work+0x16e/0x24f
[151898.018547]  worker_thread+0x1e2/0x2b8
[151898.018548]  ? rescuer_thread+0x2a7/0x2a7
[151898.018550]  kthread+0x10c/0x114
[151898.018552]  ? kthread_park+0x89/0x89
[151898.018554]  ret_from_fork+0x22/0x40
[152004.044275] rcu: INFO: rcu_bh self-detected stall on CPU
[152004.044282] rcu:    8-....: (5568692 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13366809/13368129 fqs=1354357
[152004.044282] rcu:     (t=5460090 jiffies g=1601 q=20)
[152004.044285] NMI backtrace for cpu 8
[152004.044288] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[152004.044290] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[152004.044296] Workqueue: events_power_efficient gc_worker
[152004.044297] Call Trace:
[152004.044299]  <IRQ>
[152004.044303]  dump_stack+0x67/0x83
[152004.044305]  nmi_cpu_backtrace+0x71/0x83
[152004.044309]  ? lapic_can_unplug_cpu+0x97/0x97
[152004.044310]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[152004.044313]  rcu_dump_cpu_stacks+0x8b/0xb4
[152004.044315]  rcu_check_callbacks+0x296/0x5a0
[152004.044317]  update_process_times+0x24/0x47
[152004.044320]  tick_sched_timer+0x36/0x64
[152004.044321]  __hrtimer_run_queues+0xb7/0x10b
[152004.044323]  ? tick_sched_handle.isra.0+0x2f/0x2f
[152004.044324]  hrtimer_interrupt+0xf4/0x20e
[152004.044327]  smp_apic_timer_interrupt+0x7b/0x93
[152004.044328]  apic_timer_interrupt+0xf/0x20
[152004.044329]  </IRQ>
[152004.044331] RIP: 0010:gc_worker+0x86/0x270
[152004.044333] Code: eb f2 8b 0d d7 9a 93 00 48 8b 15 e4 9a 93 00 3b 05 c2 9a 93 00 75 dd 39 cd 72 02 31 ed 89 e8 48 8d 04 c2 4c 8b 30 41 f6 c6 01 <0f> 85 4a 01 00 00 41 0f b6 46 37 49 c7 c0 f0 ff ff ff 41 ff c5 48
[152004.044334] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[152004.044335] RAX: 000000000860fb95 RBX: 0000000000000000 RCX: 0000000000010000
[152004.044336] RDX: 00000001090ad079 RSI: 00000000000001f4 RDI: ffffffff822aa760
[152004.044337] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[152004.044338] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[152004.044339] R13: 00000000a6263806 R14: ffff888ce9496c08 R15: ffff888ce9496bc0
[152004.044343]  process_one_work+0x16e/0x24f
[152004.044344]  worker_thread+0x1e2/0x2b8
[152004.044346]  ? rescuer_thread+0x2a7/0x2a7
[152004.044348]  kthread+0x10c/0x114
[152004.044349]  ? kthread_park+0x89/0x89
[152004.044352]  ret_from_fork+0x22/0x40
[152078.021132] rcu: INFO: rcu_sched self-detected stall on CPU
[152078.021138] rcu:    8-....: (5640092 ticks this GP) idle=04a/1/0x4000000000000002 softirq=13368129/13368129 fqs=1398865
[152078.021138] rcu:     (t=5640093 jiffies g=37631889 q=20934676)
[152078.021141] NMI backtrace for cpu 8
[152078.021143] CPU: 8 PID: 18088 Comm: kworker/u64:1 Tainted: P        W  O      4.19.107-Unraid #1
[152078.021144] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4, BIOS P2.20 11/25/2019
[152078.021150] Workqueue: events_power_efficient gc_worker
[152078.021151] Call Trace:
[152078.021153]  <IRQ>
[152078.021156]  dump_stack+0x67/0x83
[152078.021159]  nmi_cpu_backtrace+0x71/0x83
[152078.021163]  ? lapic_can_unplug_cpu+0x97/0x97
[152078.021165]  nmi_trigger_cpumask_backtrace+0x57/0xd4
[152078.021168]  rcu_dump_cpu_stacks+0x8b/0xb4
[152078.021170]  rcu_check_callbacks+0x296/0x5a0
[152078.021172]  update_process_times+0x24/0x47
[152078.021173]  tick_sched_timer+0x36/0x64
[152078.021175]  __hrtimer_run_queues+0xb7/0x10b
[152078.021177]  ? tick_sched_handle.isra.0+0x2f/0x2f
[152078.021178]  hrtimer_interrupt+0xf4/0x20e
[152078.021181]  smp_apic_timer_interrupt+0x7b/0x93
[152078.021183]  apic_timer_interrupt+0xf/0x20
[152078.021184]  </IRQ>
[152078.021186] RIP: 0010:gc_worker+0x1d1/0x270
[152078.021188] Code: 84 c0 75 ce 4d 85 ff 75 10 48 c7 c7 79 fc d2 81 e8 e1 6f b1 ff 0f 0b eb 0e f0 41 ff 0f 75 08 4c 89 ff e8 2f 6d ff ff 4d 8b 36 <e9> ac fe ff ff e8 a8 0b 11 00 ff 44 24 04 8b 74 24 04 39 74 24 0c
[152078.021189] RSP: 0018:ffffc900068e3e60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[152078.021190] RAX: 00000000085fda9c RBX: 0000000000000000 RCX: 0000000000010000
[152078.021191] RDX: 00000001090bf172 RSI: 00000000000001f4 RDI: ffffffff822aa760
[152078.021192] RBP: 00000000000083f5 R08: ffffffffffffffb8 R09: 0000746e65696369
[152078.021193] R10: 8080808080808080 R11: fefefefefefefeff R12: ffffffff822aa760
[152078.021194] R13: 00000000d3b58faa R14: ffff888ce9496c08 R15: ffff888ce9496bc0
 

 

Can anyone assist to prevent unRaid from randomly locking up?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.