DjSamLb Posted October 5, 2020 Share Posted October 5, 2020 Hey guys, Having a new error with very high cpu load: Oct 5 09:27:32 Tower kernel: <IRQ> Oct 5 09:27:32 Tower kernel: dump_stack+0x67/0x83 Oct 5 09:27:32 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 5 09:27:32 Tower kernel: ? lapic_can_unplug_cpu+0x97/0x97 Oct 5 09:27:32 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Oct 5 09:27:32 Tower kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Oct 5 09:27:32 Tower kernel: rcu_check_callbacks+0x296/0x5a0 Oct 5 09:27:32 Tower kernel: update_process_times+0x24/0x47 Oct 5 09:27:32 Tower kernel: tick_sched_timer+0x36/0x64 Oct 5 09:27:32 Tower kernel: __hrtimer_run_queues+0xb7/0x10b Oct 5 09:27:32 Tower kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Oct 5 09:27:32 Tower kernel: hrtimer_interrupt+0xf4/0x20e Oct 5 09:27:32 Tower kernel: smp_apic_timer_interrupt+0x7b/0x93 Oct 5 09:27:32 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 5 09:27:32 Tower kernel: </IRQ> Oct 5 09:27:32 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x69/0x6b Oct 5 09:27:32 Tower kernel: Code: 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 8b 03 3d 00 01 00 00 74 ec f3 90 <eb> f3 55 53 48 89 fb 65 8b 35 19 bf f8 7e 81 e6 00 ff 1f 00 74 0c Oct 5 09:27:32 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 Oct 5 09:27:32 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000 Oct 5 09:27:32 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060 Oct 5 09:27:32 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000 Oct 5 09:27:32 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 Oct 5 09:27:32 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230 Oct 5 09:27:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55 Oct 5 09:27:32 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab Oct 5 09:27:32 Tower kernel: btree_write_cache_pages+0x28c/0x2ce Oct 5 09:27:32 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad Oct 5 09:27:32 Tower kernel: do_writepages+0x28/0x51 Oct 5 09:27:32 Tower kernel: __writeback_single_inode+0x36/0x15a Oct 5 09:27:32 Tower kernel: writeback_sb_inodes+0x1e7/0x373 Oct 5 09:27:32 Tower kernel: __writeback_inodes_wb+0x63/0x9a Oct 5 09:27:32 Tower kernel: wb_writeback+0x11f/0x1c3 Oct 5 09:27:32 Tower kernel: wb_workfn+0x1d1/0x253 Oct 5 09:27:32 Tower kernel: process_one_work+0x16e/0x24f Oct 5 09:27:32 Tower kernel: worker_thread+0x1e2/0x2b8 Oct 5 09:27:32 Tower kernel: ? rescuer_thread+0x2a7/0x2a7 Oct 5 09:27:32 Tower kernel: kthread+0x10c/0x114 Oct 5 09:27:32 Tower kernel: ? kthread_park+0x89/0x89 Oct 5 09:27:32 Tower kernel: ret_from_fork+0x1f/0x40 Quote Link to comment
DjSamLb Posted October 5, 2020 Author Share Posted October 5, 2020 some more log entries: Oct 5 09:39:32 Tower kernel: kthread+0x10c/0x114 Oct 5 09:39:32 Tower kernel: ? kthread_park+0x89/0x89 Oct 5 09:39:32 Tower kernel: ret_from_fork+0x1f/0x40 Oct 5 09:39:32 Tower kernel: Sending NMI from CPU 12 to CPUs 17: Oct 5 09:39:32 Tower kernel: NMI backtrace for cpu 17 Oct 5 09:39:32 Tower kernel: CPU: 17 PID: 26425 Comm: kworker/u48:18 Tainted: G D O 4.19.107-Unraid #1 Oct 5 09:39:32 Tower kernel: Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2 01/16/2015 Oct 5 09:39:32 Tower kernel: Workqueue: writeback wb_workfn (flush-btrfs-2) Oct 5 09:39:32 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x67/0x6b Oct 5 09:39:32 Tower kernel: Code: 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 8b 03 3d 00 01 00 00 74 ec <f3> 90 eb f3 55 53 48 89 fb 65 8b 35 19 bf f8 7e 81 e6 00 ff 1f 00 Oct 5 09:39:32 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206 Oct 5 09:39:32 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000 Oct 5 09:39:32 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060 Oct 5 09:39:32 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000 Oct 5 09:39:32 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 Oct 5 09:39:32 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230 Oct 5 09:39:32 Tower kernel: FS: 0000000000000000(0000) GS:ffff88903fa40000(0000) knlGS:0000000000000000 Oct 5 09:39:32 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 5 09:39:32 Tower kernel: CR2: 000000649dcfdfd4 CR3: 0000000001e0a003 CR4: 00000000001626e0 Oct 5 09:39:32 Tower kernel: DR0: 0000000002a259fb DR1: 0000000000000000 DR2: 0000000000000000 Oct 5 09:39:32 Tower kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 5 09:39:32 Tower kernel: Call Trace: Oct 5 09:39:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55 Oct 5 09:39:32 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab Oct 5 09:39:32 Tower kernel: btree_write_cache_pages+0x28c/0x2ce Oct 5 09:39:32 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad Oct 5 09:39:32 Tower kernel: do_writepages+0x28/0x51 Oct 5 09:39:32 Tower kernel: __writeback_single_inode+0x36/0x15a Oct 5 09:39:32 Tower kernel: writeback_sb_inodes+0x1e7/0x373 Oct 5 09:39:32 Tower kernel: __writeback_inodes_wb+0x63/0x9a Oct 5 09:39:32 Tower kernel: wb_writeback+0x11f/0x1c3 Oct 5 09:39:32 Tower kernel: wb_workfn+0x1d1/0x253 Oct 5 09:39:32 Tower kernel: process_one_work+0x16e/0x24f Oct 5 09:39:32 Tower kernel: worker_thread+0x1e2/0x2b8 Oct 5 09:39:32 Tower kernel: ? rescuer_thread+0x2a7/0x2a7 Oct 5 09:39:32 Tower kernel: kthread+0x10c/0x114 Oct 5 09:39:32 Tower kernel: ? kthread_park+0x89/0x89 Oct 5 09:39:32 Tower kernel: ret_from_fork+0x1f/0x40 Oct 5 09:39:40 Tower kernel: md: recovery thread: PQ corrected, sector=628430296 Quote Link to comment
DjSamLb Posted October 5, 2020 Author Share Posted October 5, 2020 top output wasn't able to download diagnostics file Quote Link to comment
DjSamLb Posted October 5, 2020 Author Share Posted October 5, 2020 some more gui is irresponsive at this stage this happened earlier in the morning a couple hours ago I did a hard reset Oct 5 09:47:45 Tower kernel: dump_stack+0x67/0x83 Oct 5 09:47:45 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 5 09:47:45 Tower kernel: ? lapic_can_unplug_cpu+0x97/0x97 Oct 5 09:47:45 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Oct 5 09:47:45 Tower kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Oct 5 09:47:45 Tower kernel: rcu_check_callbacks+0x296/0x5a0 Oct 5 09:47:45 Tower kernel: update_process_times+0x24/0x47 Oct 5 09:47:45 Tower kernel: tick_sched_timer+0x36/0x64 Oct 5 09:47:45 Tower kernel: __hrtimer_run_queues+0xb7/0x10b Oct 5 09:47:45 Tower kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Oct 5 09:47:45 Tower kernel: hrtimer_interrupt+0xf4/0x20e Oct 5 09:47:45 Tower kernel: smp_apic_timer_interrupt+0x7b/0x93 Oct 5 09:47:45 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 5 09:47:45 Tower kernel: </IRQ> Oct 5 09:47:45 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x5e/0x6b Oct 5 09:47:45 Tower kernel: Code: 00 00 00 eb 25 ba ff 00 00 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 <8b> 03 3d 00 01 00 00 74 ec f3 90 eb f3 55 53 48 89 fb 65 8b 35 19 Oct 5 09:47:45 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 Oct 5 09:47:45 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000 Oct 5 09:47:45 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060 Oct 5 09:47:45 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000 Oct 5 09:47:45 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 Oct 5 09:47:45 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230 Oct 5 09:47:45 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55 Oct 5 09:47:45 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab Oct 5 09:47:45 Tower kernel: btree_write_cache_pages+0x28c/0x2ce Oct 5 09:47:45 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad Oct 5 09:47:45 Tower kernel: do_writepages+0x28/0x51 Oct 5 09:47:45 Tower kernel: __writeback_single_inode+0x36/0x15a Oct 5 09:47:45 Tower kernel: writeback_sb_inodes+0x1e7/0x373 Oct 5 09:47:45 Tower kernel: __writeback_inodes_wb+0x63/0x9a Oct 5 09:47:45 Tower kernel: wb_writeback+0x11f/0x1c3 Oct 5 09:47:45 Tower kernel: wb_workfn+0x1d1/0x253 Oct 5 09:47:45 Tower kernel: process_one_work+0x16e/0x24f Oct 5 09:47:45 Tower kernel: worker_thread+0x1e2/0x2b8 Oct 5 09:47:45 Tower kernel: ? rescuer_thread+0x2a7/0x2a7 Oct 5 09:47:45 Tower kernel: kthread+0x10c/0x114 Oct 5 09:47:45 Tower kernel: ? kthread_park+0x89/0x89 Oct 5 09:47:45 Tower kernel: ret_from_fork+0x1f/0x40 Oct 5 09:48:28 Tower nginx: 2020/10/05 09:48:28 [error] 8051#8051: *17680 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "GET /plugins/ipmi/include/ipmi_temp.php?unit=C&dot=. HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard" Oct 5 09:48:28 Tower nginx: 2020/10/05 09:48:28 [error] 8051#8051: *17697 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard" Oct 5 09:48:28 Tower php-fpm[8025]: [WARNING] [pool www] server reached max_children setting (50), consider raising it Oct 5 09:48:28 Tower nginx: 2020/10/05 09:48:28 [error] 8051#8051: *17695 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "POST /webGui/include/DashboardApps.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard" Oct 5 09:48:32 Tower kernel: rcu: INFO: rcu_bh self-detected stall on CPU Oct 5 09:48:32 Tower kernel: rcu: 12-....: (1563708 ticks this GP) idle=b6e/1/0x4000000000000002 softirq=939787/947039 fqs=354696 Oct 5 09:48:32 Tower kernel: rcu: (t=1500026 jiffies g=-947 q=36) Oct 5 09:48:32 Tower kernel: NMI backtrace for cpu 12 Oct 5 09:48:32 Tower kernel: CPU: 12 PID: 11089 Comm: btrfs-transacti Tainted: G D O 4.19.107-Unraid #1 Oct 5 09:48:32 Tower kernel: Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2 01/16/2015 Oct 5 09:48:32 Tower kernel: Call Trace: Oct 5 09:48:32 Tower kernel: <IRQ> Oct 5 09:48:32 Tower kernel: dump_stack+0x67/0x83 Oct 5 09:48:32 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 5 09:48:32 Tower kernel: ? lapic_can_unplug_cpu+0x97/0x97 Oct 5 09:48:32 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Oct 5 09:48:32 Tower kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Oct 5 09:48:32 Tower kernel: rcu_check_callbacks+0x296/0x5a0 Oct 5 09:48:32 Tower kernel: update_process_times+0x24/0x47 Oct 5 09:48:32 Tower kernel: tick_sched_timer+0x36/0x64 Oct 5 09:48:32 Tower kernel: __hrtimer_run_queues+0xb7/0x10b Oct 5 09:48:32 Tower kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Oct 5 09:48:32 Tower kernel: hrtimer_interrupt+0xf4/0x20e Oct 5 09:48:32 Tower kernel: smp_apic_timer_interrupt+0x7b/0x93 Oct 5 09:48:32 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 5 09:48:32 Tower kernel: </IRQ> Oct 5 09:48:32 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x63/0x171 Oct 5 09:48:32 Tower kernel: Code: 2f 08 b8 00 01 00 00 0f 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a <8b> 07 84 c0 74 04 f3 90 eb f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02 Oct 5 09:48:32 Tower kernel: RSP: 0018:ffffc9000998fae0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Oct 5 09:48:32 Tower kernel: RAX: 0000000000000101 RBX: ffff888820d96060 RCX: 0000000000000000 Oct 5 09:48:32 Tower kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff888820d96064 Oct 5 09:48:32 Tower kernel: RBP: ffff888820d96064 R08: ffff8890330a6000 R09: 0000000000000001 Oct 5 09:48:32 Tower kernel: R10: 0000000000000000 R11: ffff8890330a7130 R12: 0000000000000001 Oct 5 09:48:32 Tower kernel: R13: ffff888ff596f000 R14: 0000000000000001 R15: ffff88867eba0938 Oct 5 09:48:32 Tower kernel: queued_write_lock_slowpath+0x23/0x6b Oct 5 09:48:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55 Oct 5 09:48:32 Tower kernel: btrfs_search_slot+0x7a9/0x84a Oct 5 09:48:32 Tower kernel: lookup_inline_extent_backref+0x118/0x58d Oct 5 09:48:32 Tower kernel: __btrfs_free_extent+0xf2/0x90f Oct 5 09:48:32 Tower kernel: __btrfs_run_delayed_refs+0xa72/0xbf4 Oct 5 09:48:32 Tower kernel: ? pick_next_task_fair+0x3eb/0x480 Oct 5 09:48:32 Tower kernel: ? put_prev_entity+0x21/0x415 Oct 5 09:48:32 Tower kernel: btrfs_run_delayed_refs+0x5d/0x16d Oct 5 09:48:32 Tower kernel: btrfs_commit_transaction+0x54/0x79c Oct 5 09:48:32 Tower kernel: ? start_transaction+0x293/0x333 Oct 5 09:48:32 Tower kernel: transaction_kthread+0xd7/0x144 Oct 5 09:48:32 Tower kernel: ? btrfs_cleanup_transaction+0x492/0x492 Oct 5 09:48:32 Tower kernel: kthread+0x10c/0x114 Oct 5 09:48:32 Tower kernel: ? kthread_park+0x89/0x89 Oct 5 09:48:32 Tower kernel: ret_from_fork+0x1f/0x40 Oct 5 09:48:32 Tower kernel: Sending NMI from CPU 12 to CPUs 17: Oct 5 09:48:32 Tower kernel: NMI backtrace for cpu 17 Oct 5 09:48:32 Tower kernel: CPU: 17 PID: 26425 Comm: kworker/u48:18 Tainted: G D O 4.19.107-Unraid #1 Oct 5 09:48:32 Tower kernel: Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2 01/16/2015 Oct 5 09:48:32 Tower kernel: Workqueue: writeback wb_workfn (flush-btrfs-2) Oct 5 09:48:32 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x60/0x6b Oct 5 09:48:32 Tower kernel: Code: 00 eb 25 ba ff 00 00 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 0f 1f 40 00 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 8b 03 <3d> 00 01 00 00 74 ec f3 90 eb f3 55 53 48 89 fb 65 8b 35 19 bf f8 Oct 5 09:48:32 Tower kernel: RSP: 0018:ffffc9000ee9fac0 EFLAGS: 00000206 Oct 5 09:48:32 Tower kernel: RAX: 00000000000001ff RBX: ffff888820d96060 RCX: 0000000000000000 Oct 5 09:48:32 Tower kernel: RDX: 00000000000000ff RSI: ffff8890330a6000 RDI: ffff888820d96060 Oct 5 09:48:32 Tower kernel: RBP: ffff888820d96064 R08: 000000000000000b R09: ffff88867d7a8000 Oct 5 09:48:32 Tower kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 Oct 5 09:48:32 Tower kernel: R13: 0000000000000000 R14: ffffc9000ee9fb60 R15: ffff888820d96230 Oct 5 09:48:32 Tower kernel: FS: 0000000000000000(0000) GS:ffff88903fa40000(0000) knlGS:0000000000000000 Oct 5 09:48:32 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 5 09:48:32 Tower kernel: CR2: 000000649dcfdfd4 CR3: 0000000001e0a003 CR4: 00000000001626e0 Oct 5 09:48:32 Tower kernel: DR0: 0000000002a259fb DR1: 0000000000000000 DR2: 0000000000000000 Oct 5 09:48:32 Tower kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 5 09:48:32 Tower kernel: Call Trace: Oct 5 09:48:32 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55 Oct 5 09:48:32 Tower kernel: lock_extent_buffer_for_io+0x1c/0x1ab Oct 5 09:48:32 Tower kernel: btree_write_cache_pages+0x28c/0x2ce Oct 5 09:48:32 Tower kernel: ? btrfs_wq_submit_bio+0x9d/0xad Oct 5 09:48:32 Tower kernel: do_writepages+0x28/0x51 Oct 5 09:48:32 Tower kernel: __writeback_single_inode+0x36/0x15a Oct 5 09:48:32 Tower kernel: writeback_sb_inodes+0x1e7/0x373 Oct 5 09:48:32 Tower kernel: __writeback_inodes_wb+0x63/0x9a Oct 5 09:48:32 Tower kernel: wb_writeback+0x11f/0x1c3 Oct 5 09:48:32 Tower kernel: wb_workfn+0x1d1/0x253 Oct 5 09:48:32 Tower kernel: process_one_work+0x16e/0x24f Oct 5 09:48:32 Tower kernel: worker_thread+0x1e2/0x2b8 Oct 5 09:48:32 Tower kernel: ? rescuer_thread+0x2a7/0x2a7 Oct 5 09:48:32 Tower kernel: kthread+0x10c/0x114 Oct 5 09:48:32 Tower kernel: ? kthread_park+0x89/0x89 Oct 5 09:48:32 Tower kernel: ret_from_fork+0x1f/0x40 Oct 5 09:48:43 Tower nginx: 2020/10/05 09:48:43 [error] 8051#8051: *17855 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.101.141, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "tower:801", referrer: "http://tower:801/Dashboard" Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.