Hey guys,

 

Having a new error with very high cpu load:

 

Oct 5 09:27:32 Tower kernel: <IRQ>
Oct 5 09:27:32 Tower kernel: dump_stack+0x67/0x83
Oct 5 09:27:32 Tower kernel: nmi_cpu_backtrace+0x71/0x83
Oct 5 09:27:32 Tower kernel: ? lapic_can_unplug_cpu+0x97/0x97
Oct 5 09:27:32 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Oct 5 09:27:32 Tower kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Oct 5 09:27:32 Tower kernel: rcu_check_callbacks+0x296/0x5a0
Oct 5 09:27:32 Tower kernel: update_process_times+0x24/0x47
Oct 5 09:27:32 Tower kernel: tick_sched_timer+0x36/0x64
Oct 5 09:27:32 Tower kernel: __hrtimer_run_queues+0xb7/0x10b
Oct 5 09:27:32 Tower kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Oct 5 09:27:32 Tower kernel: hrtimer_interrupt+0xf4/0x20e
Oct 5 09:27:32 Tower kernel: smp_apic_timer_interrupt+0x7b/0x93
Oct 5 09:27:32 Tower kernel: apic_timer_interrupt+0xf/0x20
Oct 5 09:27:32 Tower kernel: </IRQ>
Oct 5 09:27:32 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x69/0x6b
Oct 5 09:27:32 Tower kernel: Code: 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 8b 03 3d 00 01 00 00 74 ec f3 90 <eb> f3 55 53 48 89 fb 65 8b 35 19 bf f8 7e 81 e6 00 ff 1f 00 74 0c
Oct 5 09:27:32 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
Oct 5 09:27:32 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000
Oct 5 09:27:32 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060
Oct 5 09:27:32 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000
Oct 5 09:27:32 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
Oct 5 09:27:32 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230
Oct 5 09:27:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55
Oct 5 09:27:32 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab
Oct 5 09:27:32 Tower kernel: btree_write_cache_pages+0x28c/0x2ce
Oct 5 09:27:32 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad
Oct 5 09:27:32 Tower kernel: do_writepages+0x28/0x51
Oct 5 09:27:32 Tower kernel: __writeback_single_inode+0x36/0x15a
Oct 5 09:27:32 Tower kernel: writeback_sb_inodes+0x1e7/0x373
Oct 5 09:27:32 Tower kernel: __writeback_inodes_wb+0x63/0x9a
Oct 5 09:27:32 Tower kernel: wb_writeback+0x11f/0x1c3
Oct 5 09:27:32 Tower kernel: wb_workfn+0x1d1/0x253
Oct 5 09:27:32 Tower kernel: process_one_work+0x16e/0x24f
Oct 5 09:27:32 Tower kernel: worker_thread+0x1e2/0x2b8
Oct 5 09:27:32 Tower kernel: ? rescuer_thread+0x2a7/0x2a7
Oct 5 09:27:32 Tower kernel: kthread+0x10c/0x114
Oct 5 09:27:32 Tower kernel: ? kthread_park+0x89/0x89
Oct 5 09:27:32 Tower kernel: ret_from_fork+0x1f/0x40

some more log entries:

 

Oct 5 09:39:32 Tower kernel: kthread+0x10c/0x114
Oct 5 09:39:32 Tower kernel: ? kthread_park+0x89/0x89
Oct 5 09:39:32 Tower kernel: ret_from_fork+0x1f/0x40
Oct 5 09:39:32 Tower kernel: Sending NMI from CPU 12 to CPUs 17:
Oct 5 09:39:32 Tower kernel: NMI backtrace for cpu 17
Oct 5 09:39:32 Tower kernel: CPU: 17 PID: 26425 Comm: kworker/u48:18 Tainted: G D O 4.19.107-Unraid #1
Oct 5 09:39:32 Tower kernel: Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2 01/16/2015
Oct 5 09:39:32 Tower kernel: Workqueue: writeback wb_workfn (flush-btrfs-2)
Oct 5 09:39:32 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x67/0x6b
Oct 5 09:39:32 Tower kernel: Code: 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 8b 03 3d 00 01 00 00 74 ec <f3> 90 eb f3 55 53 48 89 fb 65 8b 35 19 bf f8 7e 81 e6 00 ff 1f 00
Oct 5 09:39:32 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206
Oct 5 09:39:32 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000
Oct 5 09:39:32 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060
Oct 5 09:39:32 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000
Oct 5 09:39:32 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
Oct 5 09:39:32 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230
Oct 5 09:39:32 Tower kernel: FS: 0000000000000000(0000) GS:ffff88903fa40000(0000) knlGS:0000000000000000
Oct 5 09:39:32 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 5 09:39:32 Tower kernel: CR2: 000000649dcfdfd4 CR3: 0000000001e0a003 CR4: 00000000001626e0
Oct 5 09:39:32 Tower kernel: DR0: 0000000002a259fb DR1: 0000000000000000 DR2: 0000000000000000
Oct 5 09:39:32 Tower kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 5 09:39:32 Tower kernel: Call Trace:
Oct 5 09:39:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55
Oct 5 09:39:32 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab
Oct 5 09:39:32 Tower kernel: btree_write_cache_pages+0x28c/0x2ce
Oct 5 09:39:32 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad
Oct 5 09:39:32 Tower kernel: do_writepages+0x28/0x51
Oct 5 09:39:32 Tower kernel: __writeback_single_inode+0x36/0x15a
Oct 5 09:39:32 Tower kernel: writeback_sb_inodes+0x1e7/0x373
Oct 5 09:39:32 Tower kernel: __writeback_inodes_wb+0x63/0x9a
Oct 5 09:39:32 Tower kernel: wb_writeback+0x11f/0x1c3
Oct 5 09:39:32 Tower kernel: wb_workfn+0x1d1/0x253
Oct 5 09:39:32 Tower kernel: process_one_work+0x16e/0x24f
Oct 5 09:39:32 Tower kernel: worker_thread+0x1e2/0x2b8
Oct 5 09:39:32 Tower kernel: ? rescuer_thread+0x2a7/0x2a7
Oct 5 09:39:32 Tower kernel: kthread+0x10c/0x114
Oct 5 09:39:32 Tower kernel: ? kthread_park+0x89/0x89
Oct 5 09:39:32 Tower kernel: ret_from_fork+0x1f/0x40
Oct 5 09:39:40 Tower kernel: md: recovery thread: PQ corrected, sector=628430296

top output

wasn't able to download diagnostics file

some more

gui is irresponsive at this stage

this happened earlier in the morning a couple hours ago I did a hard reset

 

Oct 5 09:47:45 Tower kernel: dump_stack+0x67/0x83
Oct 5 09:47:45 Tower kernel: nmi_cpu_backtrace+0x71/0x83
Oct 5 09:47:45 Tower kernel: ? lapic_can_unplug_cpu+0x97/0x97
Oct 5 09:47:45 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Oct 5 09:47:45 Tower kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Oct 5 09:47:45 Tower kernel: rcu_check_callbacks+0x296/0x5a0
Oct 5 09:47:45 Tower kernel: update_process_times+0x24/0x47
Oct 5 09:47:45 Tower kernel: tick_sched_timer+0x36/0x64
Oct 5 09:47:45 Tower kernel: __hrtimer_run_queues+0xb7/0x10b
Oct 5 09:47:45 Tower kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Oct 5 09:47:45 Tower kernel: hrtimer_interrupt+0xf4/0x20e
Oct 5 09:47:45 Tower kernel: smp_apic_timer_interrupt+0x7b/0x93
Oct 5 09:47:45 Tower kernel: apic_timer_interrupt+0xf/0x20
Oct 5 09:47:45 Tower kernel: </IRQ>
Oct 5 09:47:45 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x5e/0x6b
Oct 5 09:47:45 Tower kernel: Code: 00 00 00 eb 25 ba ff 00 00 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 <8b> 03 3d 00 01 00 00 74 ec f3 90 eb f3 55 53 48 89 fb 65 8b 35 19
Oct 5 09:47:45 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
Oct 5 09:47:45 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000
Oct 5 09:47:45 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060
Oct 5 09:47:45 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000
Oct 5 09:47:45 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
Oct 5 09:47:45 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230
Oct 5 09:47:45 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55
Oct 5 09:47:45 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab
Oct 5 09:47:45 Tower kernel: btree_write_cache_pages+0x28c/0x2ce
Oct 5 09:47:45 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad
Oct 5 09:47:45 Tower kernel: do_writepages+0x28/0x51
Oct 5 09:47:45 Tower kernel: __writeback_single_inode+0x36/0x15a
Oct 5 09:47:45 Tower kernel: writeback_sb_inodes+0x1e7/0x373
Oct 5 09:47:45 Tower kernel: __writeback_inodes_wb+0x63/0x9a
Oct 5 09:47:45 Tower kernel: wb_writeback+0x11f/0x1c3
Oct 5 09:47:45 Tower kernel: wb_workfn+0x1d1/0x253
Oct 5 09:47:45 Tower kernel: process_one_work+0x16e/0x24f
Oct 5 09:47:45 Tower kernel: worker_thread+0x1e2/0x2b8
Oct 5 09:47:45 Tower kernel: ? rescuer_thread+0x2a7/0x2a7
Oct 5 09:47:45 Tower kernel: kthread+0x10c/0x114
Oct 5 09:47:45 Tower kernel: ? kthread_park+0x89/0x89
Oct 5 09:47:45 Tower kernel: ret_from_fork+0x1f/0x40
Oct 5 09:48:28 Tower nginx: 2020/10/05 09:48:28 [error] 8051#8051: *17680 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "GET /plugins/ipmi/include/ipmi_temp.php?unit=C&dot=. HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard"
Oct 5 09:48:28 Tower nginx: 2020/10/05 09:48:28 [error] 8051#8051: *17697 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard"
Oct 5 09:48:28 Tower php-fpm[8025]: [WARNING] [pool www] server reached max_children setting (50), consider raising it
Oct 5 09:48:28 Tower nginx: 2020/10/05 09:48:28 [error] 8051#8051: *17695 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "POST /webGui/include/DashboardApps.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard"
Oct 5 09:48:32 Tower kernel: rcu: INFO: rcu_bh self-detected stall on CPU
Oct 5 09:48:32 Tower kernel: rcu: 12-....: (1563708 ticks this GP) idle=b6e/1/0x4000000000000002 softirq=939787/947039 fqs=354696
Oct 5 09:48:32 Tower kernel: rcu: (t=1500026 jiffies g=-947 q=36)
Oct 5 09:48:32 Tower kernel: NMI backtrace for cpu 12
Oct 5 09:48:32 Tower kernel: CPU: 12 PID: 11089 Comm: btrfs-transacti Tainted: G D O 4.19.107-Unraid #1
Oct 5 09:48:32 Tower kernel: Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2 01/16/2015
Oct 5 09:48:32 Tower kernel: Call Trace:
Oct 5 09:48:32 Tower kernel: <IRQ>
Oct 5 09:48:32 Tower kernel: dump_stack+0x67/0x83
Oct 5 09:48:32 Tower kernel: nmi_cpu_backtrace+0x71/0x83
Oct 5 09:48:32 Tower kernel: ? lapic_can_unplug_cpu+0x97/0x97
Oct 5 09:48:32 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Oct 5 09:48:32 Tower kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Oct 5 09:48:32 Tower kernel: rcu_check_callbacks+0x296/0x5a0
Oct 5 09:48:32 Tower kernel: update_process_times+0x24/0x47
Oct 5 09:48:32 Tower kernel: tick_sched_timer+0x36/0x64
Oct 5 09:48:32 Tower kernel: __hrtimer_run_queues+0xb7/0x10b
Oct 5 09:48:32 Tower kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Oct 5 09:48:32 Tower kernel: hrtimer_interrupt+0xf4/0x20e
Oct 5 09:48:32 Tower kernel: smp_apic_timer_interrupt+0x7b/0x93
Oct 5 09:48:32 Tower kernel: apic_timer_interrupt+0xf/0x20
Oct 5 09:48:32 Tower kernel: </IRQ>
Oct 5 09:48:32 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x63/0x171
Oct 5 09:48:32 Tower kernel: Code: 2f 08 b8 00 01 00 00 0f 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a <8b> 07 84 c0 74 04 f3 90 eb f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02
Oct 5 09:48:32 Tower kernel: RSP: 0018:ffffc9000998fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Oct 5 09:48:32 Tower kernel: RAX: 0000000000000101 RBX: ffff888820d96060 RCX: 0000000000000000
Oct 5 09:48:32 Tower kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff888820d96064
Oct 5 09:48:32 Tower kernel: RBP: ffff888820d96064 R08: ffff8890330a6000 R09: 0000000000000001
Oct 5 09:48:32 Tower kernel: R10: 0000000000000000 R11: ffff8890330a7130 R12: 0000000000000001
Oct 5 09:48:32 Tower kernel: R13: ffff888ff596f000 R14: 0000000000000001 R15: ffff88867eba0938
Oct 5 09:48:32 Tower kernel: queued_write_lock_slowpath+0x23/0x6b
Oct 5 09:48:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55
Oct 5 09:48:32 Tower kernel: btrfs_search_slot+0x7a9/0x84a
Oct 5 09:48:32 Tower kernel: lookup_inline_extent_backref+0x118/0x58d
Oct 5 09:48:32 Tower kernel: __btrfs_free_extent+0xf2/0x90f
Oct 5 09:48:32 Tower kernel: __btrfs_run_delayed_refs+0xa72/0xbf4
Oct 5 09:48:32 Tower kernel: ? pick_next_task_fair+0x3eb/0x480
Oct 5 09:48:32 Tower kernel: ? put_prev_entity+0x21/0x415
Oct 5 09:48:32 Tower kernel: btrfs_run_delayed_refs+0x5d/0x16d
Oct 5 09:48:32 Tower kernel: btrfs_commit_transaction+0x54/0x79c
Oct 5 09:48:32 Tower kernel: ? start_transaction+0x293/0x333
Oct 5 09:48:32 Tower kernel: transaction_kthread+0xd7/0x144
Oct 5 09:48:32 Tower kernel: ? btrfs_cleanup_transaction+0x492/0x492
Oct 5 09:48:32 Tower kernel: kthread+0x10c/0x114
Oct 5 09:48:32 Tower kernel: ? kthread_park+0x89/0x89
Oct 5 09:48:32 Tower kernel: ret_from_fork+0x1f/0x40
Oct 5 09:48:32 Tower kernel: Sending NMI from CPU 12 to CPUs 17:
Oct 5 09:48:32 Tower kernel: NMI backtrace for cpu 17
Oct 5 09:48:32 Tower kernel: CPU: 17 PID: 26425 Comm: kworker/u48:18 Tainted: G D O 4.19.107-Unraid #1
Oct 5 09:48:32 Tower kernel: Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2 01/16/2015
Oct 5 09:48:32 Tower kernel: Workqueue: writeback wb_workfn (flush-btrfs-2)
Oct 5 09:48:32 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x60/0x6b
Oct 5 09:48:32 Tower kernel: Code: 00 eb 25 ba ff 00 00 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 8b 03 <3d> 00 01 00 00 74 ec f3 90 eb f3 55 53 48 89 fb 65 8b 35 19 bf f8
Oct 5 09:48:32 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206
Oct 5 09:48:32 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000
Oct 5 09:48:32 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060
Oct 5 09:48:32 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000
Oct 5 09:48:32 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
Oct 5 09:48:32 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230
Oct 5 09:48:32 Tower kernel: FS: 0000000000000000(0000) GS:ffff88903fa40000(0000) knlGS:0000000000000000
Oct 5 09:48:32 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 5 09:48:32 Tower kernel: CR2: 000000649dcfdfd4 CR3: 0000000001e0a003 CR4: 00000000001626e0
Oct 5 09:48:32 Tower kernel: DR0: 0000000002a259fb DR1: 0000000000000000 DR2: 0000000000000000
Oct 5 09:48:32 Tower kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 5 09:48:32 Tower kernel: Call Trace:
Oct 5 09:48:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55
Oct 5 09:48:32 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab
Oct 5 09:48:32 Tower kernel: btree_write_cache_pages+0x28c/0x2ce
Oct 5 09:48:32 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad
Oct 5 09:48:32 Tower kernel: do_writepages+0x28/0x51
Oct 5 09:48:32 Tower kernel: __writeback_single_inode+0x36/0x15a
Oct 5 09:48:32 Tower kernel: writeback_sb_inodes+0x1e7/0x373
Oct 5 09:48:32 Tower kernel: __writeback_inodes_wb+0x63/0x9a
Oct 5 09:48:32 Tower kernel: wb_writeback+0x11f/0x1c3
Oct 5 09:48:32 Tower kernel: wb_workfn+0x1d1/0x253
Oct 5 09:48:32 Tower kernel: process_one_work+0x16e/0x24f
Oct 5 09:48:32 Tower kernel: worker_thread+0x1e2/0x2b8
Oct 5 09:48:32 Tower kernel: ? rescuer_thread+0x2a7/0x2a7
Oct 5 09:48:32 Tower kernel: kthread+0x10c/0x114
Oct 5 09:48:32 Tower kernel: ? kthread_park+0x89/0x89
Oct 5 09:48:32 Tower kernel: ret_from_fork+0x1f/0x40
Oct 5 09:48:43 Tower nginx: 2020/10/05 09:48:43 [error] 8051#8051: *17855 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard"