• [6.10.3] Kernel panic?/crash all system, webui not reachble - unstable


    b0n3v
    • Urgent

    A little background, when I was on 6.9.2 I had no problems, uptime was 90+days no issues, i decide to upgrade to 6.10.1 and the problems start, after this downgrade to 6.9.2 but sometime the server just was unreachable and only manual restart do the thing. Now with 6.10.3, after 6+ days he do the same thing with attached log:

    First im not sure what cause the problem, i see the erros in the log and asap click on diagnostic and system hang... But I copy actual syslog to the clipbord and will attach here. Any suggestion? After this the system hang/crash and not respond on SMB/WEB and etc.

    Spoiler

    Jul 16 14:00:38 Server emhttpd: spinning down /dev/sdb
    Jul 16 14:01:38 Server emhttpd: spinning down /dev/sdc
    Jul 16 18:53:33 Server kernel: BUG: unable to handle page fault for address: fffff8efc7a616c8
    Jul 16 18:53:33 Server kernel: #PF: supervisor read access in kernel mode
    Jul 16 18:53:33 Server kernel: #PF: error_code(0x0000) - not-present page
    Jul 16 18:53:33 Server kernel: PGD 0 P4D 0 
    Jul 16 18:53:33 Server kernel: Oops: 0000 [#2] SMP NOPTI
    Jul 16 18:53:33 Server kernel: CPU: 1 PID: 19622 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 18:53:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 18:53:33 Server kernel: RIP: 0010:_compound_head+0x0/0x11
    Jul 16 18:53:33 Server kernel: Code: 54 f2 ff 5b 5d 41 5c 41 5d c3 8b 44 24 10 48 89 44 24 10 8b 44 24 08 48 89 44 24 08 e9 3a d4 f3 ff 89 d2 89 f6 e9 1b d1 f3 ff <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff
    Jul 16 18:53:33 Server kernel: RSP: 0000:ffffc900061dfd68 EFLAGS: 00010246
    Jul 16 18:53:33 Server kernel: RAX: ffffea0000000000 RBX: fff0000000000818 RCX: 0000000000000000
    Jul 16 18:53:33 Server kernel: RDX: 0000003bbf1e985b RSI: ffff8881db73f9a0 RDI: fffff8efc7a616c0
    Jul 16 18:53:33 Server kernel: RBP: ffffea00076dcfe8 R08: ffff888154884228 R09: 0000000000000000
    Jul 16 18:53:33 Server kernel: R10: 0000000000000000 R11: 0000000000000000 R12: fffff8efc7a616c0
    Jul 16 18:53:33 Server kernel: R13: 0000000000000000 R14: ffff8883cb4facc0 R15: 7c00003bbf1e985b
    Jul 16 18:53:33 Server kernel: FS:  000014bd20144b30(0000) GS:ffff888ffe840000(0000) knlGS:0000000000000000
    Jul 16 18:53:33 Server kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jul 16 18:53:33 Server kernel: CR2: fffff8efc7a616c8 CR3: 00000001ceb6a000 CR4: 0000000000350ee0
    Jul 16 18:53:33 Server kernel: Call Trace:
    Jul 16 18:53:33 Server kernel: <TASK>
    Jul 16 18:53:33 Server kernel: pfn_swap_entry_to_page+0x27/0x3c
    Jul 16 18:53:33 Server kernel: __migration_entry_wait+0x48/0x82
    Jul 16 18:53:33 Server kernel: do_swap_page+0x57/0x534
    Jul 16 18:53:33 Server kernel: __handle_mm_fault+0x470/0xc5c
    Jul 16 18:53:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:53:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 18:53:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 18:53:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 18:53:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:53:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 18:53:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 18:53:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 18:53:33 Server kernel: RSP: 002b:000014bd20143960 EFLAGS: 00010202
    Jul 16 18:53:33 Server kernel: RAX: 000014bd200d4920 RBX: 000014c5948901f8 RCX: 0000000000002000
    Jul 16 18:53:33 Server kernel: RDX: 000014bd200d8920 RSI: 000014bf08b34000 RDI: 000014bd200d6920
    Jul 16 18:53:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 18:53:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 18:53:33 Server kernel: R13: 000014c5948cf100 R14: 000014bd20143b40 R15: 000014bd20143ae0
    Jul 16 18:53:33 Server kernel: </TASK>
    Jul 16 18:53:33 Server kernel: Modules linked in: xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nvidia(PO) drm backlight nct6775 hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding r8169 realtek e1000e i2c_piix4 input_leds ahci edac_mce_amd wmi_bmof kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl i2c_core ccp tpm_crb k10temp libahci led_class tpm_tis nvme tpm_tis_core tpm nvme_core wmi button acpi_cpufreq [last unloaded: realtek]
    Jul 16 18:53:33 Server kernel: CR2: fffff8efc7a616c8
    Jul 16 18:53:33 Server kernel: ---[ end trace f56b516ddac5dab9 ]---
    Jul 16 18:53:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 18:53:33 Server kernel: RSP: 002b:000014bd2018b960 EFLAGS: 00010202
    Jul 16 18:53:33 Server kernel: RAX: 000014bd200bdff0 RBX: 000014c594890590 RCX: 0000000000002000
    Jul 16 18:53:33 Server kernel: RDX: 000014bd200c1ff0 RSI: 000014bf08b34000 RDI: 000014bd200bfff0
    Jul 16 18:53:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 18:53:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 18:53:33 Server kernel: R13: 000014c5948cf0a0 R14: 000014bd2018bb40 R15: 000014bd2018bae0
    Jul 16 18:53:33 Server kernel: FS:  000014bd20144b30(0000) GS:ffff888ffe840000(0000) knlGS:0000000000000000
    Jul 16 18:53:33 Server kernel: CS:  0033 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jul 16 18:53:33 Server kernel: CR2: fffff8efc7a616c8 CR3: 00000001ceb6a000 CR4: 0000000000350ee0
    Jul 16 18:54:33 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 18:54:33 Server kernel: rcu:     9-....: (60000 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=13981 
    Jul 16 18:54:33 Server kernel:     (t=60001 jiffies g=98202457 q=150527)
    Jul 16 18:54:33 Server kernel: NMI backtrace for cpu 9
    Jul 16 18:54:33 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 18:54:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 18:54:33 Server kernel: Call Trace:
    Jul 16 18:54:33 Server kernel: <IRQ>
    Jul 16 18:54:33 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 18:54:33 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 18:54:33 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 18:54:33 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 18:54:33 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 18:54:33 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 18:54:33 Server kernel: ? trigger_load_balance+0x7a/0x292
    Jul 16 18:54:33 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 18:54:33 Server kernel: update_process_times+0x8c/0xab
    Jul 16 18:54:33 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 18:54:33 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 18:54:33 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 18:54:33 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 18:54:33 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 18:54:33 Server kernel: </IRQ>
    Jul 16 18:54:33 Server kernel: <TASK>
    Jul 16 18:54:33 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 18:54:33 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 18:54:33 Server kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
    Jul 16 18:54:33 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 18:54:33 Server kernel: RAX: 0000000000080101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 18:54:33 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 18:54:33 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 18:54:33 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 18:54:33 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 18:54:33 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 18:54:33 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 18:54:33 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 18:54:33 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 18:54:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:54:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 18:54:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 18:54:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 18:54:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:54:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 18:54:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 18:54:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 18:54:33 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 18:54:33 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 18:54:33 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 18:54:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 18:54:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 18:54:33 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 18:54:33 Server kernel: </TASK>
    Jul 16 18:55:13 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 63759 jiffies s: 45489 root: 0x1/.
    Jul 16 18:55:13 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 18:55:13 Server kernel: Task dump for CPU 1:
    Jul 16 18:55:13 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 18:55:13 Server kernel: Call Trace:
    Jul 16 18:55:13 Server kernel: <TASK>
    Jul 16 18:55:13 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 18:55:13 Server kernel: ? update_curr+0x115/0x140
    Jul 16 18:55:13 Server kernel: ? update_curr+0x26/0x140
    Jul 16 18:55:13 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 18:55:13 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 18:55:13 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 18:55:13 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 18:55:13 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 18:55:13 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 18:55:13 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 18:55:13 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:55:13 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 18:55:13 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 18:55:13 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 18:55:13 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:55:13 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 18:55:13 Server kernel: </TASK>
    Jul 16 18:55:13 Server kernel: Task dump for CPU 9:
    Jul 16 18:55:13 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 18:55:13 Server kernel: Call Trace:
    Jul 16 18:55:13 Server kernel: <TASK>
    Jul 16 18:55:13 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 18:55:13 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 18:55:13 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 18:55:13 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 18:55:13 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 18:55:13 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 18:55:13 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 18:55:13 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 18:55:13 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 18:55:13 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 18:55:13 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 18:55:13 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 18:55:13 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 18:55:13 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:55:13 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 18:55:13 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 18:55:13 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 18:55:13 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:55:13 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 18:55:13 Server kernel: </TASK>
    Jul 16 18:57:33 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 18:57:33 Server kernel: rcu:     9-....: (240004 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=55359 
    Jul 16 18:57:33 Server kernel:     (t=240005 jiffies g=98202457 q=1868348)
    Jul 16 18:57:33 Server kernel: NMI backtrace for cpu 9
    Jul 16 18:57:33 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 18:57:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 18:57:33 Server kernel: Call Trace:
    Jul 16 18:57:33 Server kernel: <IRQ>
    Jul 16 18:57:33 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 18:57:33 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 18:57:33 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 18:57:33 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 18:57:33 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 18:57:33 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 18:57:33 Server kernel: ? trigger_load_balance+0x7a/0x292
    Jul 16 18:57:33 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 18:57:33 Server kernel: update_process_times+0x8c/0xab
    Jul 16 18:57:33 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 18:57:33 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 18:57:33 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 18:57:33 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 18:57:33 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 18:57:33 Server kernel: </IRQ>
    Jul 16 18:57:33 Server kernel: <TASK>
    Jul 16 18:57:33 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 18:57:33 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x78/0x18f
    Jul 16 18:57:33 Server kernel: Code: 2a 08 8b 02 0f 92 c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 <84> c0 74 04 f3 90 eb f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65
    Jul 16 18:57:33 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 18:57:33 Server kernel: RAX: 00000000000c0101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 18:57:33 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 18:57:33 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 18:57:33 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 18:57:33 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 18:57:33 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 18:57:33 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 18:57:33 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 18:57:33 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 18:57:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:57:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 18:57:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 18:57:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 18:57:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:57:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 18:57:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 18:57:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 18:57:33 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 18:57:33 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 18:57:33 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 18:57:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 18:57:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 18:57:33 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 18:57:33 Server kernel: </TASK>
    Jul 16 18:58:13 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 243983 jiffies s: 45489 root: 0x1/.
    Jul 16 18:58:13 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 18:58:13 Server kernel: Task dump for CPU 1:
    Jul 16 18:58:13 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 18:58:13 Server kernel: Call Trace:
    Jul 16 18:58:13 Server kernel: <TASK>
    Jul 16 18:58:13 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 18:58:13 Server kernel: ? update_curr+0x115/0x140
    Jul 16 18:58:13 Server kernel: ? update_curr+0x26/0x140
    Jul 16 18:58:13 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 18:58:13 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 18:58:13 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 18:58:13 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 18:58:13 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 18:58:13 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 18:58:13 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 18:58:13 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:58:13 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 18:58:13 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 18:58:13 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 18:58:13 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:58:13 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 18:58:13 Server kernel: </TASK>
    Jul 16 18:58:13 Server kernel: Task dump for CPU 9:
    Jul 16 18:58:13 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 18:58:13 Server kernel: Call Trace:
    Jul 16 18:58:13 Server kernel: <TASK>
    Jul 16 18:58:13 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 18:58:13 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 18:58:13 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 18:58:13 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 18:58:13 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 18:58:13 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 18:58:13 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 18:58:13 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 18:58:13 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 18:58:13 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 18:58:13 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 18:58:13 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 18:58:13 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 18:58:13 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 18:58:13 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 18:58:13 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 18:58:13 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 18:58:13 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 18:58:13 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 18:58:13 Server kernel: </TASK>
    Jul 16 19:00:33 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 19:00:33 Server kernel: rcu:     9-....: (420008 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=96012 
    Jul 16 19:00:33 Server kernel:     (t=420009 jiffies g=98202457 q=2612320)
    Jul 16 19:00:33 Server kernel: NMI backtrace for cpu 9
    Jul 16 19:00:33 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 19:00:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 19:00:33 Server kernel: Call Trace:
    Jul 16 19:00:33 Server kernel: <IRQ>
    Jul 16 19:00:33 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 19:00:33 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 19:00:33 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 19:00:33 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 19:00:33 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 19:00:33 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 19:00:33 Server kernel: ? trigger_load_balance+0x7a/0x292
    Jul 16 19:00:33 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 19:00:33 Server kernel: update_process_times+0x8c/0xab
    Jul 16 19:00:33 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 19:00:33 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 19:00:33 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 19:00:33 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 19:00:33 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 19:00:33 Server kernel: </IRQ>
    Jul 16 19:00:33 Server kernel: <TASK>
    Jul 16 19:00:33 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:00:33 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:00:33 Server kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
    Jul 16 19:00:33 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 19:00:33 Server kernel: RAX: 00000000001c0101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 19:00:33 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 19:00:33 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 19:00:33 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 19:00:33 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 19:00:33 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:00:33 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 19:00:33 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:00:33 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:00:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:00:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 19:00:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 19:00:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 19:00:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:00:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 19:00:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 19:00:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 19:00:33 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 19:00:33 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 19:00:33 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 19:00:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 19:00:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 19:00:33 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 19:00:33 Server kernel: </TASK>
    Jul 16 19:01:13 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 424207 jiffies s: 45489 root: 0x1/.
    Jul 16 19:01:13 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 19:01:13 Server kernel: Task dump for CPU 1:
    Jul 16 19:01:13 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 19:01:13 Server kernel: Call Trace:
    Jul 16 19:01:13 Server kernel: <TASK>
    Jul 16 19:01:13 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 19:01:13 Server kernel: ? update_curr+0x115/0x140
    Jul 16 19:01:13 Server kernel: ? update_curr+0x26/0x140
    Jul 16 19:01:13 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:01:13 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:01:13 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 19:01:13 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:01:13 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:01:13 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:01:13 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:01:13 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:01:13 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:01:13 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:01:13 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:01:13 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:01:13 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:01:13 Server kernel: </TASK>
    Jul 16 19:01:13 Server kernel: Task dump for CPU 9:
    Jul 16 19:01:13 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 19:01:13 Server kernel: Call Trace:
    Jul 16 19:01:13 Server kernel: <TASK>
    Jul 16 19:01:13 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 19:01:13 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 19:01:13 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 19:01:13 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 19:01:13 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 19:01:13 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 19:01:13 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:01:13 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:01:13 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:01:13 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:01:13 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:01:13 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:01:13 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:01:13 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:01:13 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:01:13 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:01:13 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:01:13 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:01:13 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:01:13 Server kernel: </TASK>
    Jul 16 19:03:33 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 19:03:33 Server kernel: rcu:     9-....: (600012 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=135106 
    Jul 16 19:03:33 Server kernel:     (t=600013 jiffies g=98202457 q=3051579)
    Jul 16 19:03:33 Server kernel: NMI backtrace for cpu 9
    Jul 16 19:03:33 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 19:03:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 19:03:33 Server kernel: Call Trace:
    Jul 16 19:03:33 Server kernel: <IRQ>
    Jul 16 19:03:33 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 19:03:33 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 19:03:33 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 19:03:33 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 19:03:33 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 19:03:33 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 19:03:33 Server kernel: ? trigger_load_balance+0x7a/0x292
    Jul 16 19:03:33 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 19:03:33 Server kernel: update_process_times+0x8c/0xab
    Jul 16 19:03:33 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 19:03:33 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 19:03:33 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 19:03:33 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 19:03:33 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 19:03:33 Server kernel: </IRQ>
    Jul 16 19:03:33 Server kernel: <TASK>
    Jul 16 19:03:33 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:03:33 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:03:33 Server kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
    Jul 16 19:03:33 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 19:03:33 Server kernel: RAX: 00000000001c0101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 19:03:33 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 19:03:33 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 19:03:33 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 19:03:33 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 19:03:33 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:03:33 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 19:03:33 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:03:33 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:03:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:03:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 19:03:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 19:03:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 19:03:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:03:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 19:03:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 19:03:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 19:03:33 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 19:03:33 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 19:03:33 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 19:03:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 19:03:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 19:03:33 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 19:03:33 Server kernel: </TASK>
    Jul 16 19:04:14 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 604432 jiffies s: 45489 root: 0x1/.
    Jul 16 19:04:14 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 19:04:14 Server kernel: Task dump for CPU 1:
    Jul 16 19:04:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 19:04:14 Server kernel: Call Trace:
    Jul 16 19:04:14 Server kernel: <TASK>
    Jul 16 19:04:14 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 19:04:14 Server kernel: ? update_curr+0x115/0x140
    Jul 16 19:04:14 Server kernel: ? update_curr+0x26/0x140
    Jul 16 19:04:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:04:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:04:14 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 19:04:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:04:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:04:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:04:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:04:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:04:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:04:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:04:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:04:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:04:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:04:14 Server kernel: </TASK>
    Jul 16 19:04:14 Server kernel: Task dump for CPU 9:
    Jul 16 19:04:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 19:04:14 Server kernel: Call Trace:
    Jul 16 19:04:14 Server kernel: <TASK>
    Jul 16 19:04:14 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 19:04:14 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 19:04:14 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 19:04:14 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 19:04:14 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 19:04:14 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 19:04:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:04:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:04:14 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:04:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:04:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:04:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:04:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:04:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:04:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:04:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:04:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:04:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:04:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:04:14 Server kernel: </TASK>
    Jul 16 19:06:33 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 19:06:33 Server kernel: rcu:     9-....: (780016 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=174037 
    Jul 16 19:06:33 Server kernel:     (t=780017 jiffies g=98202457 q=3508115)
    Jul 16 19:06:33 Server kernel: NMI backtrace for cpu 9
    Jul 16 19:06:33 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 19:06:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 19:06:33 Server kernel: Call Trace:
    Jul 16 19:06:33 Server kernel: <IRQ>
    Jul 16 19:06:33 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 19:06:33 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 19:06:33 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 19:06:33 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 19:06:33 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 19:06:33 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 19:06:33 Server kernel: ? trigger_load_balance+0x7a/0x292
    Jul 16 19:06:33 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 19:06:33 Server kernel: update_process_times+0x8c/0xab
    Jul 16 19:06:33 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 19:06:33 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 19:06:33 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 19:06:33 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 19:06:33 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 19:06:33 Server kernel: </IRQ>
    Jul 16 19:06:33 Server kernel: <TASK>
    Jul 16 19:06:33 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:06:33 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:06:33 Server kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
    Jul 16 19:06:33 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 19:06:33 Server kernel: RAX: 00000000001c0101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 19:06:33 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 19:06:33 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 19:06:33 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 19:06:33 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 19:06:33 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:06:33 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 19:06:33 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:06:33 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:06:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:06:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 19:06:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 19:06:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 19:06:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:06:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 19:06:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 19:06:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 19:06:33 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 19:06:33 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 19:06:33 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 19:06:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 19:06:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 19:06:33 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 19:06:33 Server kernel: </TASK>
    Jul 16 19:07:14 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 784655 jiffies s: 45489 root: 0x1/.
    Jul 16 19:07:14 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 19:07:14 Server kernel: Task dump for CPU 1:
    Jul 16 19:07:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 19:07:14 Server kernel: Call Trace:
    Jul 16 19:07:14 Server kernel: <TASK>
    Jul 16 19:07:14 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 19:07:14 Server kernel: ? update_curr+0x115/0x140
    Jul 16 19:07:14 Server kernel: ? update_curr+0x26/0x140
    Jul 16 19:07:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:07:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:07:14 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 19:07:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:07:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:07:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:07:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:07:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:07:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:07:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:07:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:07:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:07:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:07:14 Server kernel: </TASK>
    Jul 16 19:07:14 Server kernel: Task dump for CPU 9:
    Jul 16 19:07:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 19:07:14 Server kernel: Call Trace:
    Jul 16 19:07:14 Server kernel: <TASK>
    Jul 16 19:07:14 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 19:07:14 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 19:07:14 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 19:07:14 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 19:07:14 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 19:07:14 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 19:07:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:07:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:07:14 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:07:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:07:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:07:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:07:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:07:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:07:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:07:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:07:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:07:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:07:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:07:14 Server kernel: </TASK>
    Jul 16 19:09:33 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 19:09:33 Server kernel: rcu:     9-....: (960020 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=212928 
    Jul 16 19:09:33 Server kernel:     (t=960021 jiffies g=98202457 q=4018337)
    Jul 16 19:09:33 Server kernel: NMI backtrace for cpu 9
    Jul 16 19:09:33 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 19:09:33 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 19:09:33 Server kernel: Call Trace:
    Jul 16 19:09:33 Server kernel: <IRQ>
    Jul 16 19:09:33 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 19:09:33 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 19:09:33 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 19:09:33 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 19:09:33 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 19:09:33 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 19:09:33 Server kernel: ? trigger_load_balance+0x7a/0x292
    Jul 16 19:09:33 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 19:09:33 Server kernel: update_process_times+0x8c/0xab
    Jul 16 19:09:33 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 19:09:33 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 19:09:33 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 19:09:33 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 19:09:33 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 19:09:33 Server kernel: </IRQ>
    Jul 16 19:09:33 Server kernel: <TASK>
    Jul 16 19:09:33 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:09:33 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:09:33 Server kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
    Jul 16 19:09:33 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 19:09:33 Server kernel: RAX: 00000000001c0101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 19:09:33 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 19:09:33 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 19:09:33 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 19:09:33 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 19:09:33 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:09:33 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 19:09:33 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:09:33 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:09:33 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:09:33 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 19:09:33 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 19:09:33 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 19:09:33 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:09:33 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 19:09:33 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 19:09:33 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 19:09:33 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 19:09:33 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 19:09:33 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 19:09:33 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 19:09:33 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 19:09:33 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 19:09:33 Server kernel: </TASK>
    Jul 16 19:10:14 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 964880 jiffies s: 45489 root: 0x1/.
    Jul 16 19:10:14 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 19:10:14 Server kernel: Task dump for CPU 1:
    Jul 16 19:10:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 19:10:14 Server kernel: Call Trace:
    Jul 16 19:10:14 Server kernel: <TASK>
    Jul 16 19:10:14 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 19:10:14 Server kernel: ? update_curr+0x115/0x140
    Jul 16 19:10:14 Server kernel: ? update_curr+0x26/0x140
    Jul 16 19:10:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:10:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:10:14 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 19:10:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:10:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:10:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:10:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:10:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:10:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:10:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:10:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:10:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:10:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:10:14 Server kernel: </TASK>
    Jul 16 19:10:14 Server kernel: Task dump for CPU 9:
    Jul 16 19:10:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 19:10:14 Server kernel: Call Trace:
    Jul 16 19:10:14 Server kernel: <TASK>
    Jul 16 19:10:14 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 19:10:14 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 19:10:14 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 19:10:14 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 19:10:14 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 19:10:14 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 19:10:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:10:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:10:14 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:10:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:10:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:10:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:10:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:10:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:10:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:10:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:10:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:10:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:10:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:10:14 Server kernel: </TASK>
    Jul 16 19:10:47 Server emhttpd: read SMART /dev/sdb
    Jul 16 19:12:34 Server kernel: rcu: INFO: rcu_sched self-detected stall on CPU
    Jul 16 19:12:34 Server kernel: rcu:     9-....: (1140024 ticks this GP) idle=dcd/1/0x4000000000000000 softirq=25971577/25971577 fqs=252229 
    Jul 16 19:12:34 Server kernel:     (t=1140025 jiffies g=98202457 q=4456089)
    Jul 16 19:12:34 Server kernel: NMI backtrace for cpu 9
    Jul 16 19:12:34 Server kernel: CPU: 9 PID: 31129 Comm: Disk Tainted: P    B D W  O      5.15.46-Unraid #1
    Jul 16 19:12:34 Server kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS (MS-7B79), BIOS A.K0 05/23/2022
    Jul 16 19:12:34 Server kernel: Call Trace:
    Jul 16 19:12:34 Server kernel: <IRQ>
    Jul 16 19:12:34 Server kernel: dump_stack_lvl+0x46/0x5a
    Jul 16 19:12:34 Server kernel: nmi_cpu_backtrace+0xae/0xd2
    Jul 16 19:12:34 Server kernel: ? lapic_can_unplug_cpu+0x93/0x93
    Jul 16 19:12:34 Server kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
    Jul 16 19:12:34 Server kernel: rcu_dump_cpu_stacks+0xc3/0xea
    Jul 16 19:12:34 Server kernel: rcu_sched_clock_irq+0x22d/0x631
    Jul 16 19:12:34 Server kernel: ? tick_sched_do_timer+0x3e/0x3e
    Jul 16 19:12:34 Server kernel: update_process_times+0x8c/0xab
    Jul 16 19:12:34 Server kernel: tick_sched_timer+0x38/0x65
    Jul 16 19:12:34 Server kernel: __hrtimer_run_queues+0xfa/0x18a
    Jul 16 19:12:34 Server kernel: hrtimer_interrupt+0x92/0x160
    Jul 16 19:12:34 Server kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
    Jul 16 19:12:34 Server kernel: sysvec_apic_timer_interrupt+0x61/0x7d
    Jul 16 19:12:34 Server kernel: </IRQ>
    Jul 16 19:12:34 Server kernel: <TASK>
    Jul 16 19:12:34 Server kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:12:34 Server kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:12:34 Server kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
    Jul 16 19:12:34 Server kernel: RSP: 0000:ffffc90006447d10 EFLAGS: 00000202
    Jul 16 19:12:34 Server kernel: RAX: 00000000001c0101 RBX: ffffc90006447e00 RCX: 0000000000000000
    Jul 16 19:12:34 Server kernel: RDX: ffffea00076dcfe8 RSI: 0000000000000000 RDI: ffffea00076dcfe8
    Jul 16 19:12:34 Server kernel: RBP: ffffea002711d440 R08: ffffea00076dcfe8 R09: ffffc90006447d58
    Jul 16 19:12:34 Server kernel: R10: ffffc90006447d58 R11: ffffc90006447d58 R12: 000014bf08b60000
    Jul 16 19:12:34 Server kernel: R13: 00000001db73f067 R14: 000ffffffffff000 R15: 0000000000000b00
    Jul 16 19:12:34 Server kernel: queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:12:34 Server kernel: filemap_map_pages+0x329/0x44e
    Jul 16 19:12:34 Server kernel: xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:12:34 Server kernel: __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:12:34 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:12:34 Server kernel: handle_mm_fault+0x11c/0x1e2
    Jul 16 19:12:34 Server kernel: do_user_addr_fault+0x342/0x50b
    Jul 16 19:12:34 Server kernel: exc_page_fault+0xe2/0x101
    Jul 16 19:12:34 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:12:34 Server kernel: asm_exc_page_fault+0x1e/0x30
    Jul 16 19:12:34 Server kernel: RIP: 0033:0x14c5969dd6a7
    Jul 16 19:12:34 Server kernel: Code: 8b 32 48 8b 48 08 48 8b 00 48 39 c6 76 09 48 8d 14 08 48 39 d6 72 0e 48 39 c6 73 0b 48 8d 14 0e 48 39 d0 73 02 0f 0b 48 89 c7 <f3> a4 48 8d 7c 24 18 e8 63 49 16 00 48 8b 84 24 e8 00 00 00 64 48
    Jul 16 19:12:34 Server kernel: RSP: 002b:000014bd201cd960 EFLAGS: 00010202
    Jul 16 19:12:34 Server kernel: RAX: 000014c5947333b0 RBX: 000014c59482fc70 RCX: 0000000000004000
    Jul 16 19:12:34 Server kernel: RDX: 000014bf08b72000 RSI: 000014bf08b6e000 RDI: 000014c5947333b0
    Jul 16 19:12:34 Server kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000014c5951f0d40
    Jul 16 19:12:34 Server kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
    Jul 16 19:12:34 Server kernel: R13: 000014c5947a7120 R14: 000014bd201cdb40 R15: 000014bd201cdae0
    Jul 16 19:12:34 Server kernel: </TASK>
    Jul 16 19:13:08 Server sshd[7915]: Connection from 11.11.11.2 port 1624 on 9.9.9.2 port 22 rdomain ""
    Jul 16 19:13:11 Server sshd[7915]: Failed publickey for root from 11.11.11.2 port 1624 ssh2: RSA SHA256:NNyCa0+OHvxvD7WHO9JHrEw0ntI64ey7RTp4F7ekCRs
    Jul 16 19:13:11 Server sshd[7915]: Postponed keyboard-interactive for root from 11.11.11.2 port 1624 ssh2 [preauth]
    Jul 16 19:13:14 Server kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 9-... } 1145104 jiffies s: 45489 root: 0x1/.
    Jul 16 19:13:14 Server kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x202/.
    Jul 16 19:13:14 Server kernel: Task dump for CPU 1:
    Jul 16 19:13:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31666 ppid: 30467 flags:0x00000228
    Jul 16 19:13:14 Server kernel: Call Trace:
    Jul 16 19:13:14 Server kernel: <TASK>
    Jul 16 19:13:14 Server kernel: ? update_cfs_rq_load_avg+0x138/0x146
    Jul 16 19:13:14 Server kernel: ? update_curr+0x115/0x140
    Jul 16 19:13:14 Server kernel: ? update_curr+0x26/0x140
    Jul 16 19:13:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:13:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:13:14 Server kernel: ? native_queued_spin_lock_slowpath+0x14f/0x18f
    Jul 16 19:13:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:13:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:13:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:13:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:13:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:13:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:13:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:13:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:13:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:13:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:13:14 Server kernel: </TASK>
    Jul 16 19:13:14 Server kernel: Task dump for CPU 9:
    Jul 16 19:13:14 Server kernel: task:Disk            state:R  running task     stack:    0 pid:31129 ppid: 30467 flags:0x00000228
    Jul 16 19:13:14 Server kernel: Call Trace:
    Jul 16 19:13:14 Server kernel: <TASK>
    Jul 16 19:13:14 Server kernel: ? dbs_update_util_handler+0x16/0x79
    Jul 16 19:13:14 Server kernel: ? update_load_avg+0x2aa/0x2ce
    Jul 16 19:13:14 Server kernel: ? throttled_hierarchy+0x9/0x21
    Jul 16 19:13:14 Server kernel: ? reweight_entity+0xa9/0xd1
    Jul 16 19:13:14 Server kernel: ? enqueue_entity+0x145/0x19f
    Jul 16 19:13:14 Server kernel: ? common_interrupt+0xf/0x8b
    Jul 16 19:13:14 Server kernel: ? sysvec_apic_timer_interrupt+0xb/0x7d
    Jul 16 19:13:14 Server kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
    Jul 16 19:13:14 Server kernel: ? native_queued_spin_lock_slowpath+0x7e/0x18f
    Jul 16 19:13:14 Server kernel: ? queued_spin_lock_slowpath+0x7/0xa
    Jul 16 19:13:14 Server kernel: ? filemap_map_pages+0x329/0x44e
    Jul 16 19:13:14 Server kernel: ? xfs_filemap_map_pages+0x46/0x60 [xfs]
    Jul 16 19:13:14 Server kernel: ? __handle_mm_fault+0xa6e/0xc5c
    Jul 16 19:13:14 Server kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
    Jul 16 19:13:14 Server kernel: ? handle_mm_fault+0x11c/0x1e2
    Jul 16 19:13:14 Server kernel: ? do_user_addr_fault+0x342/0x50b
    Jul 16 19:13:14 Server kernel: ? exc_page_fault+0xe2/0x101
    Jul 16 19:13:14 Server kernel: ? asm_exc_page_fault+0x8/0x30
    Jul 16 19:13:14 Server kernel: ? asm_exc_page_fault+0x1e/0x30
    Jul 16 19:13:14 Server kernel: </TASK>

     




    User Feedback

    Recommended Comments

    Thanks @ChatNoir i change from mcvlan to ipvlan and after some time, 5-8h the webUI was not reacheble, smb was not reachable but dockers was reachable. I try to SSH from the router(he is the gataway/same VLAN) directly to the Unraid and have connection. I did diagnostics. The problem is appeared for first time when i update from 6.9.2 to 6.10.1 after this i have constantly problems with Unraid, before the update uptime was 90+days aways. Now not more than 2-3 days. Diagnostics attached, when the server was unrachable and only SSH access was accesable. After the reboot, server is not reachable again only SSH from the same VLAN.

    JFYI I have LAG + VLANS with working config since 6.9.2 without updates/changes. (to be sure i disable all drop/reject rules and write accept rule from this network where is the server, no difference).

    EDIT: After long time of search the root problem, memtest, cpu stress and etc, i switched from docker .img to folder(i want to recreate all from beginning) and all problems disappear. Weird..

     

    Edited by b0n3v
    Link to comment
    Quote

    after this downgrade to 6.9.2 but sometime the server just was unreachable and only manual restart do the thing.

    If the server now also crashes with v6.9.2, a previous known good release with it, it suggests something changed or there could be a hardware problem, enable the syslog server and post that after a crash to see if it catches something.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.