Server crash - Help needed to understand syslog

Opawesome

By Opawesome
in General Support

Recommended Posts

Opawesome Contributor

Opawesome

Posted
Posted (edited)

Hi all,

 

My server crashed today. Below is the content of the syslog I was able to retrieve. Unfortunately, I could not get the diagnostics before the crash. I am attaching the diagnostics obtained after a forced reboot.

 

I also had another crash a few days ago. The content of the syslog right before that other crash is also below.

 

I am a bit lost and unsure as to how troubleshoot the issue because the logs look quite cryptic to me.

 

Does anybody have an idea on how to pin point the issue?

 

Many thanks,

OP

 

 

LATEST SYSLOG RIGHT BEFORE CRASH:

  

Aug  8 20:19:40 MOZART kernel: ------------[ cut here ]------------
Aug  8 20:19:40 MOZART kernel: NETDEV WATCHDOG: eth0 (igb): transmit queue 3 timed out
Aug  8 20:19:40 MOZART kernel: WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:477 dev_watchdog+0x115/0x180
Aug  8 20:19:40 MOZART kernel: Modules linked in: ccp veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha macvlan xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_multiport xfs dm_crypt dm_mod dax md_mod iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 tun ip6table_filter ip6_tables iptable_filter ip_tables x_tables igb x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel ipmi_ssif crypto_simd cryptd i915 rapl intel_cstate ast intel_uncore drm_vram_helper iosf_mbi drm_ttm_helper ttm drm_kms_helper i2c_i801 nvme i2c_smbus ahci drm nvme_core i2c_algo_bit acpi_ipmi libahci intel_pch_thermal video input_leds intel_gtt led_class mpt3sas ipmi_si backlight agpgart
Aug  8 20:19:40 MOZART kernel: thermal fan i2c_core acpi_power_meter syscopyarea button raid_class sysfillrect acpi_pad sysimgblt scsi_transport_sas fb_sys_fops [last unloaded: igb]
Aug  8 20:19:40 MOZART kernel: CPU: 6 PID: 0 Comm: swapper/6 Tainted: G        W I       5.15.46-Unraid #1
Aug  8 20:19:40 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Aug  8 20:19:40 MOZART kernel: RIP: 0010:dev_watchdog+0x115/0x180
Aug  8 20:19:40 MOZART kernel: Code: 25 cb 00 00 75 36 48 89 ef c6 05 7d 25 cb 00 01 e8 d0 8e fb ff 44 89 e1 48 89 ee 48 c7 c7 54 09 15 82 48 89 c2 e8 ae 30 12 00 <0f> 0b eb 0e 41 ff c4 48 05 40 01 00 00 e9 65 ff ff ff 48 8b 83 48
Aug  8 20:19:40 MOZART kernel: RSP: 0018:ffffc90000208ec8 EFLAGS: 00010282
Aug  8 20:19:40 MOZART kernel: RAX: 0000000000000000 RBX: ffff888106890480 RCX: 0000000000000027
Aug  8 20:19:40 MOZART kernel: RDX: 0000000000000003 RSI: ffffc90000208d50 RDI: ffff88904539c510
Aug  8 20:19:40 MOZART kernel: RBP: ffff888106890000 R08: ffffffff822b4e68 R09: ffffffff82244e48
Aug  8 20:19:40 MOZART kernel: R10: 0000000000000000 R11: ffffffff8284ecaf R12: 0000000000000003
Aug  8 20:19:40 MOZART kernel: R13: 0000000147683e00 R14: ffffc90000208f10 R15: ffffffff816df274
Aug  8 20:19:40 MOZART kernel: FS:  0000000000000000(0000) GS:ffff889045380000(0000) knlGS:0000000000000000
Aug  8 20:19:40 MOZART kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  8 20:19:40 MOZART kernel: CR2: 0000237c9ac6d068 CR3: 000000000220a003 CR4: 00000000003726e0
Aug  8 20:19:40 MOZART kernel: Call Trace:
Aug  8 20:19:40 MOZART kernel: <IRQ>
Aug  8 20:19:40 MOZART kernel: ? psched_ppscfg_precompute+0x40/0x40
Aug  8 20:19:40 MOZART kernel: call_timer_fn+0x59/0xde
Aug  8 20:19:40 MOZART kernel: __run_timers+0x146/0x184
Aug  8 20:19:40 MOZART kernel: ? enqueue_hrtimer+0x62/0x69
Aug  8 20:19:40 MOZART kernel: ? recalibrate_cpu_khz+0x1/0x1
Aug  8 20:19:40 MOZART kernel: run_timer_softirq+0x19/0x2d
Aug  8 20:19:40 MOZART kernel: __do_softirq+0xef/0x218
Aug  8 20:19:40 MOZART kernel: __irq_exit_rcu+0x4d/0x88
Aug  8 20:19:40 MOZART kernel: sysvec_apic_timer_interrupt+0x66/0x7d
Aug  8 20:19:40 MOZART kernel: </IRQ>
Aug  8 20:19:40 MOZART kernel: <TASK>
Aug  8 20:19:40 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug  8 20:19:40 MOZART kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Aug  8 20:19:40 MOZART kernel: Code: a2 bc 1c 00 85 db 48 89 e8 79 03 48 63 c3 5b 5d 41 5c c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 0f 1f 44 00 00 55 49 89 d3 48 81 c7 b0 00 00 00 48 83 c6 70 53
Aug  8 20:19:40 MOZART kernel: RSP: 0018:ffffc900000efea0 EFLAGS: 00000246
Aug  8 20:19:40 MOZART kernel: RAX: ffff8890453abb40 RBX: 0000000000000006 RCX: 000000000000001f
Aug  8 20:19:40 MOZART kernel: RDX: 0000000000000006 RSI: 0000000000000006 RDI: 0000000000000000
Aug  8 20:19:40 MOZART kernel: RBP: ffff8890453b4c68 R08: 00000000ffffffff R09: 071c71c71c71c71c
Aug  8 20:19:40 MOZART kernel: R10: 0000000000000020 R11: 0000000000000354 R12: ffffffff82311d20
Aug  8 20:19:40 MOZART kernel: R13: 0000000000000006 R14: 000441d8f8574715 R15: 0000000000000000
Aug  8 20:19:40 MOZART kernel: cpuidle_enter_state+0x117/0x1db
Aug  8 20:19:40 MOZART kernel: cpuidle_enter+0x2a/0x36
Aug  8 20:19:40 MOZART kernel: do_idle+0x1b7/0x225
Aug  8 20:19:40 MOZART kernel: cpu_startup_entry+0x1d/0x1f
Aug  8 20:19:40 MOZART kernel: secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:19:40 MOZART kernel: </TASK>
Aug  8 20:19:40 MOZART kernel: ---[ end trace 5d4004415d5f3338 ]---
Aug  8 20:19:40 MOZART kernel: igb 0000:0a:00.0 eth0: Reset adapter
Aug  8 20:19:41 MOZART kernel: igb 0000:0a:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Aug  8 20:20:33 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug  8 20:20:33 MOZART kernel: rcu: #0112-....: (60006 ticks this GP) idle=17f/1/0x4000000000000004 softirq=257932014/257932015 fqs=14988 
Aug  8 20:20:33 MOZART kernel: #011(t=60000 jiffies g=483494137 q=339738)
Aug  8 20:20:33 MOZART kernel: NMI backtrace for cpu 2
Aug  8 20:20:33 MOZART kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W I       5.15.46-Unraid #1
Aug  8 20:20:33 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Aug  8 20:20:33 MOZART kernel: Call Trace:
Aug  8 20:20:33 MOZART kernel: <IRQ>
Aug  8 20:20:33 MOZART kernel: dump_stack_lvl+0x46/0x5a
Aug  8 20:20:33 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Aug  8 20:20:33 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Aug  8 20:20:33 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Aug  8 20:20:33 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Aug  8 20:20:33 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Aug  8 20:20:33 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Aug  8 20:20:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:20:33 MOZART kernel: update_process_times+0x8c/0xab
Aug  8 20:20:33 MOZART kernel: tick_sched_timer+0x38/0x65
Aug  8 20:20:33 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Aug  8 20:20:33 MOZART kernel: hrtimer_interrupt+0x92/0x160
Aug  8 20:20:33 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Aug  8 20:20:33 MOZART kernel: sysvec_apic_timer_interrupt+0x37/0x7d
Aug  8 20:20:33 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug  8 20:20:33 MOZART kernel: RIP: 0010:nf_ct_tuple_equal+0x32/0x47 [nf_conntrack]
Aug  8 20:20:33 MOZART kernel: Code: 48 8b 4f 08 48 33 4e 08 48 09 ca 75 2e 8b 4e 10 39 4f 10 75 26 48 8b 57 14 48 8b 4f 1c 48 33 56 14 48 33 4e 1c 48 09 ca 75 11 <8b> 47 24 33 46 24 a9 ff ff ff 00 0f 94 c0 0f b6 c0 83 e0 01 c3 0f
Aug  8 20:20:33 MOZART kernel: RSP: 0018:ffffc90000158908 EFLAGS: 00000246
Aug  8 20:20:33 MOZART kernel: RAX: 0000000000000000 RBX: ffff8881e0fa28c0 RCX: 0000000000000000
Aug  8 20:20:33 MOZART kernel: RDX: 0000000000000000 RSI: ffff8881e0fa2520 RDI: ffff8881e0fa28e0
Aug  8 20:20:33 MOZART kernel: RBP: 00000000000123b5 R08: 0000000000000003 R09: 7f5ab9ce7bc1419e
Aug  8 20:20:33 MOZART kernel: R10: a0cf3fad5ce73de0 R11: 45ecf4d24635c54c R12: ffff8881e0fa2548
Aug  8 20:20:33 MOZART kernel: R13: ffffffff828e1500 R14: ffffc90000158970 R15: ffff8881e0fa2500
Aug  8 20:20:33 MOZART kernel: nf_conntrack_tuple_taken+0xdb/0x156 [nf_conntrack]
Aug  8 20:20:33 MOZART kernel: nf_nat_used_tuple+0x33/0x4e [nf_nat]
Aug  8 20:20:33 MOZART kernel: nf_nat_setup_info+0x3f5/0x773 [nf_nat]
Aug  8 20:20:33 MOZART kernel: ? addrtype_mt_v1+0xc2/0x100 [xt_addrtype]
Aug  8 20:20:33 MOZART kernel: __nf_nat_alloc_null_binding+0x68/0x7f [nf_nat]
Aug  8 20:20:33 MOZART kernel: nf_nat_inet_fn+0xa6/0x187 [nf_nat]
Aug  8 20:20:33 MOZART kernel: nf_nat_ipv4_pre_routing+0x23/0x4e [nf_nat]
Aug  8 20:20:33 MOZART kernel: nf_hook_slow+0x3e/0x93
Aug  8 20:20:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:20:33 MOZART kernel: NF_HOOK.isra.0+0xbd/0x119 [br_netfilter]
Aug  8 20:20:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:20:33 MOZART kernel: br_nf_pre_routing+0x226/0x236 [br_netfilter]
Aug  8 20:20:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:20:33 MOZART kernel: br_handle_frame+0x292/0x2da
Aug  8 20:20:33 MOZART kernel: ? tcp_v4_destroy_sock+0x50/0x18c
Aug  8 20:20:33 MOZART kernel: ? br_handle_local_finish+0x13/0x13
Aug  8 20:20:33 MOZART kernel: __netif_receive_skb_core.constprop.0+0x4b6/0x685
Aug  8 20:20:33 MOZART kernel: __netif_receive_skb_list_core+0x81/0x112
Aug  8 20:20:33 MOZART kernel: netif_receive_skb_list_internal+0x1c7/0x1fc
Aug  8 20:20:33 MOZART kernel: gro_normal_list+0x22/0x3e
Aug  8 20:20:33 MOZART kernel: napi_complete_done+0x7f/0x118
Aug  8 20:20:33 MOZART kernel: igb_poll+0xd69/0xf96 [igb]
Aug  8 20:20:33 MOZART kernel: ? update_cfs_rq_load_avg+0x138/0x146
Aug  8 20:20:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:20:33 MOZART kernel: __napi_poll.constprop.0+0x2a/0x114
Aug  8 20:20:33 MOZART kernel: net_rx_action+0xe8/0x1f2
Aug  8 20:20:33 MOZART kernel: __do_softirq+0xef/0x218
Aug  8 20:20:33 MOZART kernel: __irq_exit_rcu+0x4d/0x88
Aug  8 20:20:33 MOZART kernel: common_interrupt+0x73/0x8b
Aug  8 20:20:33 MOZART kernel: </IRQ>
Aug  8 20:20:33 MOZART kernel: <TASK>
Aug  8 20:20:33 MOZART kernel: asm_common_interrupt+0x1e/0x40
Aug  8 20:20:33 MOZART kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Aug  8 20:20:33 MOZART kernel: Code: a2 bc 1c 00 85 db 48 89 e8 79 03 48 63 c3 5b 5d 41 5c c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 0f 1f 44 00 00 55 49 89 d3 48 81 c7 b0 00 00 00 48 83 c6 70 53
Aug  8 20:20:33 MOZART kernel: RSP: 0018:ffffc900000cfea0 EFLAGS: 00000246
Aug  8 20:20:33 MOZART kernel: RAX: ffff8890452abb40 RBX: 0000000000000004 RCX: 000000000000001f
Aug  8 20:20:33 MOZART kernel: RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000000
Aug  8 20:20:33 MOZART kernel: RBP: ffff8890452b4c68 R08: 00000000ffffffff R09: 071c71c71c71c71c
Aug  8 20:20:33 MOZART kernel: R10: 0000000000000020 R11: 000000000000006a R12: ffffffff82311d20
Aug  8 20:20:33 MOZART kernel: R13: 0000000000000004 R14: 000441d7297f9002 R15: 0000000000000000
Aug  8 20:20:33 MOZART kernel: cpuidle_enter_state+0x117/0x1db
Aug  8 20:20:33 MOZART kernel: cpuidle_enter+0x2a/0x36
Aug  8 20:20:33 MOZART kernel: do_idle+0x1b7/0x225
Aug  8 20:20:33 MOZART kernel: cpu_startup_entry+0x1d/0x1f
Aug  8 20:20:33 MOZART kernel: secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:20:33 MOZART kernel: </TASK>
Aug  8 20:20:38 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 2-... } 63702 jiffies s: 2989065 root: 0x4/.
Aug  8 20:20:38 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug  8 20:20:38 MOZART kernel: Task dump for CPU 2:
Aug  8 20:20:38 MOZART kernel: task:swapper/2       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00004808
Aug  8 20:20:38 MOZART kernel: Call Trace:
Aug  8 20:20:38 MOZART kernel: <TASK>
Aug  8 20:20:38 MOZART kernel: ? arch_local_irq_enable+0x7/0x8
Aug  8 20:20:38 MOZART kernel: ? cpuidle_enter_state+0x117/0x1db
Aug  8 20:20:38 MOZART kernel: ? cpuidle_enter+0x2a/0x36
Aug  8 20:20:38 MOZART kernel: ? do_idle+0x1b7/0x225
Aug  8 20:20:38 MOZART kernel: ? cpu_startup_entry+0x1d/0x1f
Aug  8 20:20:38 MOZART kernel: ? secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:20:38 MOZART kernel: </TASK>
Aug  8 20:23:24 MOZART emhttpd: spinning down /dev/sdb
Aug  8 20:23:33 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug  8 20:23:33 MOZART kernel: rcu: #0112-....: (240009 ticks this GP) idle=17f/1/0x4000000000000004 softirq=257932014/257932015 fqs=59974 
Aug  8 20:23:33 MOZART kernel: #011(t=240003 jiffies g=483494137 q=1231599)
Aug  8 20:23:33 MOZART kernel: NMI backtrace for cpu 2
Aug  8 20:23:33 MOZART kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W I       5.15.46-Unraid #1
Aug  8 20:23:33 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Aug  8 20:23:33 MOZART kernel: Call Trace:
Aug  8 20:23:33 MOZART kernel: <IRQ>
Aug  8 20:23:33 MOZART kernel: dump_stack_lvl+0x46/0x5a
Aug  8 20:23:33 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Aug  8 20:23:33 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Aug  8 20:23:33 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Aug  8 20:23:33 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Aug  8 20:23:33 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Aug  8 20:23:33 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Aug  8 20:23:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:23:33 MOZART kernel: update_process_times+0x8c/0xab
Aug  8 20:23:33 MOZART kernel: tick_sched_timer+0x38/0x65
Aug  8 20:23:33 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Aug  8 20:23:33 MOZART kernel: hrtimer_interrupt+0x92/0x160
Aug  8 20:23:33 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Aug  8 20:23:33 MOZART kernel: sysvec_apic_timer_interrupt+0x37/0x7d
Aug  8 20:23:33 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug  8 20:23:33 MOZART kernel: RIP: 0010:nf_ct_key_equal+0x48/0x70 [nf_conntrack]
Aug  8 20:23:33 MOZART kernel: Code: ff ff 84 c0 74 3d 48 89 ef e8 6c ef ff ff 0f b6 4d 37 41 8a 55 03 48 89 c7 48 89 c3 66 41 8b 75 00 e8 91 ff ff ff 84 c0 74 19 <48> 8b 83 80 00 00 00 a8 08 74 0e 45 31 e4 4c 39 b3 90 00 00 00 41
Aug  8 20:23:33 MOZART kernel: RSP: 0018:ffffc900001588e0 EFLAGS: 00000202
Aug  8 20:23:33 MOZART kernel: RAX: 0000000000000001 RBX: ffff8881e0fa2500 RCX: 0000000000000000
Aug  8 20:23:33 MOZART kernel: RDX: 0000000000000003 RSI: 00000000e0fa0000 RDI: ffff8881e0fa2500
Aug  8 20:23:33 MOZART kernel: RBP: ffff8881e0fa2548 R08: 0000000000000003 R09: 7f5ab9ce7bc1419e
Aug  8 20:23:33 MOZART kernel: R10: a0cf3fad5ce73de0 R11: 45ecf4d24635c54c R12: 0000000000000000
Aug  8 20:23:33 MOZART kernel: R13: ffff8881e0fa28cc R14: ffffffff828e1500 R15: ffff8881e0fa2500
Aug  8 20:23:33 MOZART kernel: nf_conntrack_tuple_taken+0xc9/0x156 [nf_conntrack]
Aug  8 20:23:33 MOZART kernel: nf_nat_used_tuple+0x33/0x4e [nf_nat]
Aug  8 20:23:33 MOZART kernel: nf_nat_setup_info+0x3f5/0x773 [nf_nat]
Aug  8 20:23:33 MOZART kernel: ? addrtype_mt_v1+0xc2/0x100 [xt_addrtype]
Aug  8 20:23:33 MOZART kernel: __nf_nat_alloc_null_binding+0x68/0x7f [nf_nat]
Aug  8 20:23:33 MOZART kernel: nf_nat_inet_fn+0xa6/0x187 [nf_nat]
Aug  8 20:23:33 MOZART kernel: nf_nat_ipv4_pre_routing+0x23/0x4e [nf_nat]
Aug  8 20:23:33 MOZART kernel: nf_hook_slow+0x3e/0x93
Aug  8 20:23:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:23:33 MOZART kernel: NF_HOOK.isra.0+0xbd/0x119 [br_netfilter]
Aug  8 20:23:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:23:33 MOZART kernel: br_nf_pre_routing+0x226/0x236 [br_netfilter]
Aug  8 20:23:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:23:33 MOZART kernel: br_handle_frame+0x292/0x2da
Aug  8 20:23:33 MOZART kernel: ? tcp_v4_destroy_sock+0x50/0x18c
Aug  8 20:23:33 MOZART kernel: ? br_handle_local_finish+0x13/0x13
Aug  8 20:23:33 MOZART kernel: __netif_receive_skb_core.constprop.0+0x4b6/0x685
Aug  8 20:23:33 MOZART kernel: __netif_receive_skb_list_core+0x81/0x112
Aug  8 20:23:33 MOZART kernel: netif_receive_skb_list_internal+0x1c7/0x1fc
Aug  8 20:23:33 MOZART kernel: gro_normal_list+0x22/0x3e
Aug  8 20:23:33 MOZART kernel: napi_complete_done+0x7f/0x118
Aug  8 20:23:33 MOZART kernel: igb_poll+0xd69/0xf96 [igb]
Aug  8 20:23:33 MOZART kernel: ? update_cfs_rq_load_avg+0x138/0x146
Aug  8 20:23:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:23:33 MOZART kernel: __napi_poll.constprop.0+0x2a/0x114
Aug  8 20:23:33 MOZART kernel: net_rx_action+0xe8/0x1f2
Aug  8 20:23:33 MOZART kernel: __do_softirq+0xef/0x218
Aug  8 20:23:33 MOZART kernel: __irq_exit_rcu+0x4d/0x88
Aug  8 20:23:33 MOZART kernel: common_interrupt+0x73/0x8b
Aug  8 20:23:33 MOZART kernel: </IRQ>
Aug  8 20:23:33 MOZART kernel: <TASK>
Aug  8 20:23:33 MOZART kernel: asm_common_interrupt+0x1e/0x40
Aug  8 20:23:33 MOZART kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Aug  8 20:23:33 MOZART kernel: Code: a2 bc 1c 00 85 db 48 89 e8 79 03 48 63 c3 5b 5d 41 5c c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 0f 1f 44 00 00 55 49 89 d3 48 81 c7 b0 00 00 00 48 83 c6 70 53
Aug  8 20:23:33 MOZART kernel: RSP: 0018:ffffc900000cfea0 EFLAGS: 00000246
Aug  8 20:23:33 MOZART kernel: RAX: ffff8890452abb40 RBX: 0000000000000004 RCX: 000000000000001f
Aug  8 20:23:33 MOZART kernel: RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000000
Aug  8 20:23:33 MOZART kernel: RBP: ffff8890452b4c68 R08: 00000000ffffffff R09: 071c71c71c71c71c
Aug  8 20:23:33 MOZART kernel: R10: 0000000000000020 R11: 000000000000006a R12: ffffffff82311d20
Aug  8 20:23:33 MOZART kernel: R13: 0000000000000004 R14: 000441d7297f9002 R15: 0000000000000000
Aug  8 20:23:33 MOZART kernel: cpuidle_enter_state+0x117/0x1db
Aug  8 20:23:33 MOZART kernel: cpuidle_enter+0x2a/0x36
Aug  8 20:23:33 MOZART kernel: do_idle+0x1b7/0x225
Aug  8 20:23:33 MOZART kernel: cpu_startup_entry+0x1d/0x1f
Aug  8 20:23:33 MOZART kernel: secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:23:33 MOZART kernel: </TASK>
Aug  8 20:23:38 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 2-... } 243926 jiffies s: 2989065 root: 0x4/.
Aug  8 20:23:38 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug  8 20:23:38 MOZART kernel: Task dump for CPU 2:
Aug  8 20:23:38 MOZART kernel: task:swapper/2       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00004808
Aug  8 20:23:38 MOZART kernel: Call Trace:
Aug  8 20:23:38 MOZART kernel: <TASK>
Aug  8 20:23:38 MOZART kernel: ? arch_local_irq_enable+0x7/0x8
Aug  8 20:23:38 MOZART kernel: ? cpuidle_enter_state+0x117/0x1db
Aug  8 20:23:38 MOZART kernel: ? cpuidle_enter+0x2a/0x36
Aug  8 20:23:38 MOZART kernel: ? do_idle+0x1b7/0x225
Aug  8 20:23:38 MOZART kernel: ? cpu_startup_entry+0x1d/0x1f
Aug  8 20:23:38 MOZART kernel: ? secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:23:38 MOZART kernel: </TASK>
Aug  8 20:26:33 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug  8 20:26:33 MOZART kernel: rcu: #0112-....: (420012 ticks this GP) idle=17f/1/0x4000000000000004 softirq=257932014/257932015 fqs=104959 
Aug  8 20:26:33 MOZART kernel: #011(t=420006 jiffies g=483494137 q=2159462)
Aug  8 20:26:33 MOZART kernel: NMI backtrace for cpu 2
Aug  8 20:26:33 MOZART kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W I       5.15.46-Unraid #1
Aug  8 20:26:33 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Aug  8 20:26:33 MOZART kernel: Call Trace:
Aug  8 20:26:33 MOZART kernel: <IRQ>
Aug  8 20:26:33 MOZART kernel: dump_stack_lvl+0x46/0x5a
Aug  8 20:26:33 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Aug  8 20:26:33 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Aug  8 20:26:33 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Aug  8 20:26:33 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Aug  8 20:26:33 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Aug  8 20:26:33 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Aug  8 20:26:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:26:33 MOZART kernel: update_process_times+0x8c/0xab
Aug  8 20:26:33 MOZART kernel: tick_sched_timer+0x38/0x65
Aug  8 20:26:33 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Aug  8 20:26:33 MOZART kernel: hrtimer_interrupt+0x92/0x160
Aug  8 20:26:33 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Aug  8 20:26:33 MOZART kernel: sysvec_apic_timer_interrupt+0x37/0x7d
Aug  8 20:26:33 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug  8 20:26:33 MOZART kernel: RIP: 0010:nf_ct_zone_equal.isra.0+0x24/0x2b [nf_conntrack]
Aug  8 20:26:33 MOZART kernel: Code: 0f fa ff ff 5d c3 44 0f b6 47 0f b8 01 00 00 00 d3 e0 31 c9 41 85 c0 74 04 66 8b 4f 0c 0f b6 d2 85 c2 b8 00 00 00 00 0f 44 f0 <66> 39 ce 0f 94 c0 c3 41 56 49 89 ce 41 55 49 89 d5 41 54 45 31 e4
Aug  8 20:26:33 MOZART kernel: RSP: 0018:ffffc900001588d8 EFLAGS: 00000202
Aug  8 20:26:33 MOZART kernel: RAX: 0000000000000000 RBX: ffff8881e0fa2500 RCX: 0000000000000000
Aug  8 20:26:33 MOZART kernel: RDX: 0000000000000003 RSI: 00000000e0fa0000 RDI: ffff8881e0fa2500
Aug  8 20:26:33 MOZART kernel: RBP: ffff8881e0fa2548 R08: 0000000000000003 R09: 7f5ab9ce7bc1419e
Aug  8 20:26:33 MOZART kernel: R10: a0cf3fad5ce73de0 R11: 45ecf4d24635c54c R12: 0000000000000000
Aug  8 20:26:33 MOZART kernel: R13: ffff8881e0fa28cc R14: ffffffff828e1500 R15: ffff8881e0fa2500
Aug  8 20:26:33 MOZART kernel: nf_ct_key_equal+0x44/0x70 [nf_conntrack]
Aug  8 20:26:33 MOZART kernel: nf_conntrack_tuple_taken+0xc9/0x156 [nf_conntrack]
Aug  8 20:26:33 MOZART kernel: nf_nat_used_tuple+0x33/0x4e [nf_nat]
Aug  8 20:26:33 MOZART kernel: nf_nat_setup_info+0x3f5/0x773 [nf_nat]
Aug  8 20:26:33 MOZART kernel: ? addrtype_mt_v1+0xc2/0x100 [xt_addrtype]
Aug  8 20:26:33 MOZART kernel: __nf_nat_alloc_null_binding+0x68/0x7f [nf_nat]
Aug  8 20:26:33 MOZART kernel: nf_nat_inet_fn+0xa6/0x187 [nf_nat]
Aug  8 20:26:33 MOZART kernel: nf_nat_ipv4_pre_routing+0x23/0x4e [nf_nat]
Aug  8 20:26:33 MOZART kernel: nf_hook_slow+0x3e/0x93
Aug  8 20:26:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:26:33 MOZART kernel: NF_HOOK.isra.0+0xbd/0x119 [br_netfilter]
Aug  8 20:26:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:26:33 MOZART kernel: br_nf_pre_routing+0x226/0x236 [br_netfilter]
Aug  8 20:26:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:26:33 MOZART kernel: br_handle_frame+0x292/0x2da
Aug  8 20:26:33 MOZART kernel: ? tcp_v4_destroy_sock+0x50/0x18c
Aug  8 20:26:33 MOZART kernel: ? br_handle_local_finish+0x13/0x13
Aug  8 20:26:33 MOZART kernel: __netif_receive_skb_core.constprop.0+0x4b6/0x685
Aug  8 20:26:33 MOZART kernel: __netif_receive_skb_list_core+0x81/0x112
Aug  8 20:26:33 MOZART kernel: netif_receive_skb_list_internal+0x1c7/0x1fc
Aug  8 20:26:33 MOZART kernel: gro_normal_list+0x22/0x3e
Aug  8 20:26:33 MOZART kernel: napi_complete_done+0x7f/0x118
Aug  8 20:26:33 MOZART kernel: igb_poll+0xd69/0xf96 [igb]
Aug  8 20:26:33 MOZART kernel: ? update_cfs_rq_load_avg+0x138/0x146
Aug  8 20:26:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:26:33 MOZART kernel: __napi_poll.constprop.0+0x2a/0x114
Aug  8 20:26:33 MOZART kernel: net_rx_action+0xe8/0x1f2
Aug  8 20:26:33 MOZART kernel: __do_softirq+0xef/0x218
Aug  8 20:26:33 MOZART kernel: __irq_exit_rcu+0x4d/0x88
Aug  8 20:26:33 MOZART kernel: common_interrupt+0x73/0x8b
Aug  8 20:26:33 MOZART kernel: </IRQ>
Aug  8 20:26:33 MOZART kernel: <TASK>
Aug  8 20:26:33 MOZART kernel: asm_common_interrupt+0x1e/0x40
Aug  8 20:26:33 MOZART kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Aug  8 20:26:33 MOZART kernel: Code: a2 bc 1c 00 85 db 48 89 e8 79 03 48 63 c3 5b 5d 41 5c c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 0f 1f 44 00 00 55 49 89 d3 48 81 c7 b0 00 00 00 48 83 c6 70 53
Aug  8 20:26:33 MOZART kernel: RSP: 0018:ffffc900000cfea0 EFLAGS: 00000246
Aug  8 20:26:33 MOZART kernel: RAX: ffff8890452abb40 RBX: 0000000000000004 RCX: 000000000000001f
Aug  8 20:26:33 MOZART kernel: RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000000
Aug  8 20:26:33 MOZART kernel: RBP: ffff8890452b4c68 R08: 00000000ffffffff R09: 071c71c71c71c71c
Aug  8 20:26:33 MOZART kernel: R10: 0000000000000020 R11: 000000000000006a R12: ffffffff82311d20
Aug  8 20:26:33 MOZART kernel: R13: 0000000000000004 R14: 000441d7297f9002 R15: 0000000000000000
Aug  8 20:26:33 MOZART kernel: cpuidle_enter_state+0x117/0x1db
Aug  8 20:26:33 MOZART kernel: cpuidle_enter+0x2a/0x36
Aug  8 20:26:33 MOZART kernel: do_idle+0x1b7/0x225
Aug  8 20:26:33 MOZART kernel: cpu_startup_entry+0x1d/0x1f
Aug  8 20:26:33 MOZART kernel: secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:26:33 MOZART kernel: </TASK>
Aug  8 20:26:39 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 2-... } 424150 jiffies s: 2989065 root: 0x4/.
Aug  8 20:26:39 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug  8 20:26:39 MOZART kernel: Task dump for CPU 2:
Aug  8 20:26:39 MOZART kernel: task:swapper/2       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00004808
Aug  8 20:26:39 MOZART kernel: Call Trace:
Aug  8 20:26:39 MOZART kernel: <TASK>
Aug  8 20:26:39 MOZART kernel: ? arch_local_irq_enable+0x7/0x8
Aug  8 20:26:39 MOZART kernel: ? cpuidle_enter_state+0x117/0x1db
Aug  8 20:26:39 MOZART kernel: ? cpuidle_enter+0x2a/0x36
Aug  8 20:26:39 MOZART kernel: ? do_idle+0x1b7/0x225
Aug  8 20:26:39 MOZART kernel: ? cpu_startup_entry+0x1d/0x1f
Aug  8 20:26:39 MOZART kernel: ? secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:26:39 MOZART kernel: </TASK>
Aug  8 20:29:33 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug  8 20:29:33 MOZART kernel: rcu: #0112-....: (600015 ticks this GP) idle=17f/1/0x4000000000000004 softirq=257932014/257932015 fqs=149945 
Aug  8 20:29:33 MOZART kernel: #011(t=600009 jiffies g=483494137 q=3080021)
Aug  8 20:29:33 MOZART kernel: NMI backtrace for cpu 2
Aug  8 20:29:33 MOZART kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W I       5.15.46-Unraid #1
Aug  8 20:29:33 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Aug  8 20:29:33 MOZART kernel: Call Trace:
Aug  8 20:29:33 MOZART kernel: <IRQ>
Aug  8 20:29:33 MOZART kernel: dump_stack_lvl+0x46/0x5a
Aug  8 20:29:33 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Aug  8 20:29:33 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Aug  8 20:29:33 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Aug  8 20:29:33 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Aug  8 20:29:33 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Aug  8 20:29:33 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Aug  8 20:29:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:29:33 MOZART kernel: update_process_times+0x8c/0xab
Aug  8 20:29:33 MOZART kernel: tick_sched_timer+0x38/0x65
Aug  8 20:29:33 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Aug  8 20:29:33 MOZART kernel: hrtimer_interrupt+0x92/0x160
Aug  8 20:29:33 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Aug  8 20:29:33 MOZART kernel: sysvec_apic_timer_interrupt+0x37/0x7d
Aug  8 20:29:33 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug  8 20:29:33 MOZART kernel: RIP: 0010:nf_ct_tuple_equal+0x3/0x47 [nf_conntrack]
Aug  8 20:29:33 MOZART kernel: Code: 80 e8 08 00 00 48 85 c0 74 1c 53 89 fb 48 89 f7 be 04 00 00 00 e8 8a ff ff ff 48 85 c0 74 05 f0 48 0f ab 18 5b c3 c3 48 8b 17 <31> c0 48 33 16 48 8b 4f 08 48 33 4e 08 48 09 ca 75 2e 8b 4e 10 39
Aug  8 20:29:33 MOZART kernel: RSP: 0018:ffffc900001588d8 EFLAGS: 00000246
Aug  8 20:29:33 MOZART kernel: RAX: 00000000475f69c3 RBX: ffff8881e0fa28c0 RCX: ffffffff828e1500
Aug  8 20:29:33 MOZART kernel: RDX: 00000000fc0000e0 RSI: ffff8881e0fa2558 RDI: ffffc90000158970
Aug  8 20:29:33 MOZART kernel: RBP: ffff8881e0fa2548 R08: 0000000000000003 R09: 7f5ab9ce7bc1419e
Aug  8 20:29:33 MOZART kernel: R10: a0cf3fad5ce73de0 R11: 45ecf4d24635c54c R12: 0000000000000000
Aug  8 20:29:33 MOZART kernel: R13: ffff8881e0fa28cc R14: ffffffff828e1500 R15: ffff8881e0fa2500
Aug  8 20:29:33 MOZART kernel: nf_ct_key_equal+0x20/0x70 [nf_conntrack]
Aug  8 20:29:33 MOZART kernel: nf_conntrack_tuple_taken+0xc9/0x156 [nf_conntrack]
Aug  8 20:29:33 MOZART kernel: nf_nat_used_tuple+0x33/0x4e [nf_nat]
Aug  8 20:29:33 MOZART kernel: nf_nat_setup_info+0x3f5/0x773 [nf_nat]
Aug  8 20:29:33 MOZART kernel: ? addrtype_mt_v1+0xc2/0x100 [xt_addrtype]
Aug  8 20:29:33 MOZART kernel: __nf_nat_alloc_null_binding+0x68/0x7f [nf_nat]
Aug  8 20:29:33 MOZART kernel: nf_nat_inet_fn+0xa6/0x187 [nf_nat]
Aug  8 20:29:33 MOZART kernel: nf_nat_ipv4_pre_routing+0x23/0x4e [nf_nat]
Aug  8 20:29:33 MOZART kernel: nf_hook_slow+0x3e/0x93
Aug  8 20:29:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:29:33 MOZART kernel: NF_HOOK.isra.0+0xbd/0x119 [br_netfilter]
Aug  8 20:29:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:29:33 MOZART kernel: br_nf_pre_routing+0x226/0x236 [br_netfilter]
Aug  8 20:29:33 MOZART kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug  8 20:29:33 MOZART kernel: br_handle_frame+0x292/0x2da
Aug  8 20:29:33 MOZART kernel: ? tcp_v4_destroy_sock+0x50/0x18c
Aug  8 20:29:33 MOZART kernel: ? br_handle_local_finish+0x13/0x13
Aug  8 20:29:33 MOZART kernel: __netif_receive_skb_core.constprop.0+0x4b6/0x685
Aug  8 20:29:33 MOZART kernel: __netif_receive_skb_list_core+0x81/0x112
Aug  8 20:29:33 MOZART kernel: netif_receive_skb_list_internal+0x1c7/0x1fc
Aug  8 20:29:33 MOZART kernel: gro_normal_list+0x22/0x3e
Aug  8 20:29:33 MOZART kernel: napi_complete_done+0x7f/0x118
Aug  8 20:29:33 MOZART kernel: igb_poll+0xd69/0xf96 [igb]
Aug  8 20:29:33 MOZART kernel: ? update_cfs_rq_load_avg+0x138/0x146
Aug  8 20:29:33 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug  8 20:29:33 MOZART kernel: __napi_poll.constprop.0+0x2a/0x114
Aug  8 20:29:33 MOZART kernel: net_rx_action+0xe8/0x1f2
Aug  8 20:29:33 MOZART kernel: __do_softirq+0xef/0x218
Aug  8 20:29:33 MOZART kernel: __irq_exit_rcu+0x4d/0x88
Aug  8 20:29:33 MOZART kernel: common_interrupt+0x73/0x8b
Aug  8 20:29:33 MOZART kernel: </IRQ>
Aug  8 20:29:33 MOZART kernel: <TASK>
Aug  8 20:29:33 MOZART kernel: asm_common_interrupt+0x1e/0x40
Aug  8 20:29:33 MOZART kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Aug  8 20:29:33 MOZART kernel: Code: a2 bc 1c 00 85 db 48 89 e8 79 03 48 63 c3 5b 5d 41 5c c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 0f 1f 44 00 00 55 49 89 d3 48 81 c7 b0 00 00 00 48 83 c6 70 53
Aug  8 20:29:33 MOZART kernel: RSP: 0018:ffffc900000cfea0 EFLAGS: 00000246
Aug  8 20:29:33 MOZART kernel: RAX: ffff8890452abb40 RBX: 0000000000000004 RCX: 000000000000001f
Aug  8 20:29:33 MOZART kernel: RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000000
Aug  8 20:29:33 MOZART kernel: RBP: ffff8890452b4c68 R08: 00000000ffffffff R09: 071c71c71c71c71c
Aug  8 20:29:33 MOZART kernel: R10: 0000000000000020 R11: 000000000000006a R12: ffffffff82311d20
Aug  8 20:29:33 MOZART kernel: R13: 0000000000000004 R14: 000441d7297f9002 R15: 0000000000000000
Aug  8 20:29:33 MOZART kernel: cpuidle_enter_state+0x117/0x1db
Aug  8 20:29:33 MOZART kernel: cpuidle_enter+0x2a/0x36
Aug  8 20:29:33 MOZART kernel: do_idle+0x1b7/0x225
Aug  8 20:29:33 MOZART kernel: cpu_startup_entry+0x1d/0x1f
Aug  8 20:29:33 MOZART kernel: secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:29:33 MOZART kernel: </TASK>
Aug  8 20:29:39 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 2-... } 604374 jiffies s: 2989065 root: 0x4/.
Aug  8 20:29:39 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug  8 20:29:39 MOZART kernel: Task dump for CPU 2:
Aug  8 20:29:39 MOZART kernel: task:swapper/2       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00004808
Aug  8 20:29:39 MOZART kernel: Call Trace:
Aug  8 20:29:39 MOZART kernel: <TASK>
Aug  8 20:29:39 MOZART kernel: ? arch_local_irq_enable+0x7/0x8
Aug  8 20:29:39 MOZART kernel: ? cpuidle_enter_state+0x117/0x1db
Aug  8 20:29:39 MOZART kernel: ? cpuidle_enter+0x2a/0x36
Aug  8 20:29:39 MOZART kernel: ? do_idle+0x1b7/0x225
Aug  8 20:29:39 MOZART kernel: ? cpu_startup_entry+0x1d/0x1f
Aug  8 20:29:39 MOZART kernel: ? secondary_startup_64_no_verify+0xb0/0xbb
Aug  8 20:29:39 MOZART kernel: </TASK>

 

 

SYSLOG BEFORE FORMER CRASH:

  

Jul 25 00:46:40 MOZART kernel: BUG: unable to handle page fault for address: fffff8efd6226208
Jul 25 00:46:40 MOZART kernel: #PF: supervisor read access in kernel mode
Jul 25 00:46:40 MOZART kernel: #PF: error_code(0x0000) - not-present page
Jul 25 00:46:40 MOZART kernel: PGD 0 P4D 0 
Jul 25 00:46:40 MOZART kernel: Oops: 0000 [#1] SMP PTI
Jul 25 00:46:40 MOZART kernel: CPU: 7 PID: 11675 Comm: postgres Tainted: G        W I       5.15.46-Unraid #1
Jul 25 00:46:40 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 00:46:40 MOZART kernel: RIP: 0010:_compound_head+0x0/0x11
Jul 25 00:46:40 MOZART kernel: Code: b8 98 0f 00 00 00 00 f0 7f 48 85 c2 0f 95 c0 0f b6 c0 c3 48 c1 ef 3a 83 ff 1f 0f 94 c0 48 83 ff 1e 0f 94 c2 09 d0 0f b6 c0 c3 <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff
Jul 25 00:46:40 MOZART kernel: RSP: 0018:ffffc90004887cc0 EFLAGS: 00010202
Jul 25 00:46:40 MOZART kernel: RAX: 0000000000000001 RBX: 0000147636694000 RCX: 00000000000000ad
Jul 25 00:46:40 MOZART kernel: RDX: 7c00000000000000 RSI: ffffea002f17b1c0 RDI: fffff8efd6226200
Jul 25 00:46:40 MOZART kernel: RBP: ffffc90004887e00 R08: ffffea002f17b1c0 R09: ffff8881789bd000
Jul 25 00:46:40 MOZART kernel: R10: 0000000000000100 R11: 0000000000000297 R12: fffff8efd6226200
Jul 25 00:46:40 MOZART kernel: R13: 7c00003bbf588988 R14: ffff88814eecee08 R15: ffff888599005d80
Jul 25 00:46:40 MOZART kernel: FS:  0000000000000000(0000) GS:ffff8890453c0000(0000) knlGS:0000000000000000
Jul 25 00:46:40 MOZART kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 25 00:46:40 MOZART kernel: CR2: fffff8efd6226208 CR3: 000000000220a001 CR4: 00000000003726e0
Jul 25 00:46:40 MOZART kernel: Call Trace:
Jul 25 00:46:40 MOZART kernel: <TASK>
Jul 25 00:46:40 MOZART kernel: pfn_swap_entry_to_page+0x26/0x35
Jul 25 00:46:40 MOZART kernel: unmap_page_range+0x4c3/0x711
Jul 25 00:46:40 MOZART kernel: unmap_vmas+0x6f/0x9d
Jul 25 00:46:40 MOZART kernel: exit_mmap+0xd6/0x145
Jul 25 00:46:40 MOZART kernel: __mmput+0x43/0xd7
Jul 25 00:46:40 MOZART kernel: do_exit+0x385/0x915
Jul 25 00:46:40 MOZART kernel: do_group_exit+0x93/0x93
Jul 25 00:46:40 MOZART kernel: __x64_sys_exit_group+0x14/0x14
Jul 25 00:46:40 MOZART kernel: do_syscall_64+0x83/0xa5
Jul 25 00:46:40 MOZART kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
Jul 25 00:46:40 MOZART kernel: RIP: 0033:0x147636f7ff0b
Jul 25 00:46:40 MOZART kernel: Code: Unable to access opcode bytes at RIP 0x147636f7fee1.
Jul 25 00:46:40 MOZART kernel: RSP: 002b:00007ffe9b0509c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Jul 25 00:46:40 MOZART kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000147636f7ff0b
Jul 25 00:46:40 MOZART kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jul 25 00:46:40 MOZART kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 25 00:46:40 MOZART kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 00007ffe9b050b60
Jul 25 00:46:40 MOZART kernel: R13: 000014763496e028 R14: 0000000000000003 R15: 0000000000000000
Jul 25 00:46:40 MOZART kernel: </TASK>
Jul 25 00:46:40 MOZART kernel: Modules linked in: ccp veth wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap macvlan xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_multiport xfs dm_crypt dm_mod dax md_mod iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 tun ip6table_filter ip6_tables iptable_filter ip_tables x_tables igb x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm ast drm_vram_helper crct10dif_pclmul iosf_mbi drm_ttm_helper crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd ipmi_ssif cryptd rapl ttm intel_cstate intel_uncore drm_kms_helper nvme i2c_i801 i2c_smbus drm nvme_core mpt3sas acpi_ipmi intel_gtt agpgart ahci i2c_algo_bit raid_class syscopyarea input_leds sysfillrect i2c_core libahci led_class
Jul 25 00:46:40 MOZART kernel: intel_pch_thermal sysimgblt scsi_transport_sas fb_sys_fops ipmi_si video backlight acpi_power_meter thermal acpi_pad button fan [last unloaded: igb]
Jul 25 00:46:40 MOZART kernel: CR2: fffff8efd6226208
Jul 25 00:46:40 MOZART kernel: ---[ end trace 76b9687835eec572 ]---
Jul 25 00:46:40 MOZART kernel: RIP: 0010:_compound_head+0x0/0x11
Jul 25 00:46:40 MOZART kernel: Code: b8 98 0f 00 00 00 00 f0 7f 48 85 c2 0f 95 c0 0f b6 c0 c3 48 c1 ef 3a 83 ff 1f 0f 94 c0 48 83 ff 1e 0f 94 c2 09 d0 0f b6 c0 c3 <48> 8b 57 08 48 89 f8 f6 c2 01 74 04 48 8d 42 ff c3 e8 ea ff ff ff
Jul 25 00:46:40 MOZART kernel: RSP: 0018:ffffc90004887cc0 EFLAGS: 00010202
Jul 25 00:46:40 MOZART kernel: RAX: 0000000000000001 RBX: 0000147636694000 RCX: 00000000000000ad
Jul 25 00:46:40 MOZART kernel: RDX: 7c00000000000000 RSI: ffffea002f17b1c0 RDI: fffff8efd6226200
Jul 25 00:46:40 MOZART kernel: RBP: ffffc90004887e00 R08: ffffea002f17b1c0 R09: ffff8881789bd000
Jul 25 00:46:40 MOZART kernel: R10: 0000000000000100 R11: 0000000000000297 R12: fffff8efd6226200
Jul 25 00:46:40 MOZART kernel: R13: 7c00003bbf588988 R14: ffff88814eecee08 R15: ffff888599005d80
Jul 25 00:46:40 MOZART kernel: FS:  0000000000000000(0000) GS:ffff8890453c0000(0000) knlGS:0000000000000000
Jul 25 00:46:40 MOZART kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 25 00:46:40 MOZART kernel: CR2: fffff8efd6226208 CR3: 000000000220a001 CR4: 00000000003726e0
Jul 25 00:46:40 MOZART kernel: Fixing recursive fault but reboot is needed!
Jul 25 00:48:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 00:48:57 MOZART kernel: rcu: #0113-....: (59999 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=14996 
Jul 25 00:48:57 MOZART kernel: #011(t=60000 jiffies g=440205233 q=184829)
Jul 25 00:48:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 00:48:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 00:48:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 00:48:57 MOZART kernel: Call Trace:
Jul 25 00:48:57 MOZART kernel: <IRQ>
Jul 25 00:48:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 00:48:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 00:48:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 00:48:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 00:48:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 00:48:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 00:48:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 00:48:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 00:48:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 00:48:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 00:48:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 00:48:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 00:48:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 00:48:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 00:48:57 MOZART kernel: </IRQ>
Jul 25 00:48:57 MOZART kernel: <TASK>
Jul 25 00:48:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 00:48:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x76/0x18f
Jul 25 00:48:57 MOZART kernel: Code: 0f ba 2a 08 8b 02 0f 92 c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a <8b> 02 84 c0 74 04 f3 90 eb f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02
Jul 25 00:48:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 00:48:57 MOZART kernel: RAX: 0000000000000101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 00:48:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 00:48:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 00:48:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 00:48:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 00:48:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 00:48:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 00:48:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 00:48:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 00:48:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 00:48:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 00:48:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 00:48:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 00:48:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 00:48:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 00:48:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 00:48:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 00:48:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 00:48:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 00:48:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:48:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:48:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 00:48:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:48:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 00:48:57 MOZART kernel: </TASK>
Jul 25 00:49:03 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 61782 jiffies s: 2757133 root: 0x8/.
Jul 25 00:49:03 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 00:49:03 MOZART kernel: Task dump for CPU 3:
Jul 25 00:49:03 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 00:49:03 MOZART kernel: Call Trace:
Jul 25 00:49:03 MOZART kernel: <TASK>
Jul 25 00:49:03 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 00:49:03 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 00:49:03 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:49:03 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:49:03 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 00:49:03 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:49:03 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 00:49:03 MOZART kernel: </TASK>
Jul 25 00:51:33 MOZART emhttpd: spinning down /dev/sdb
Jul 25 00:51:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 00:51:57 MOZART kernel: rcu: #0113-....: (239966 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=59972 
Jul 25 00:51:57 MOZART kernel: #011(t=240003 jiffies g=440205233 q=1178748)
Jul 25 00:51:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 00:51:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 00:51:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 00:51:57 MOZART kernel: Call Trace:
Jul 25 00:51:57 MOZART kernel: <IRQ>
Jul 25 00:51:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 00:51:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 00:51:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 00:51:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 00:51:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 00:51:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 00:51:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 00:51:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 00:51:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 00:51:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 00:51:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 00:51:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 00:51:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 00:51:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 00:51:57 MOZART kernel: </IRQ>
Jul 25 00:51:57 MOZART kernel: <TASK>
Jul 25 00:51:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 00:51:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 00:51:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 00:51:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 00:51:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 00:51:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 00:51:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 00:51:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 00:51:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 00:51:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 00:51:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 00:51:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 00:51:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 00:51:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 00:51:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 00:51:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 00:51:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 00:51:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 00:51:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 00:51:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 00:51:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 00:51:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 00:51:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 00:51:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:51:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:51:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 00:51:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:51:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 00:51:57 MOZART kernel: </TASK>
Jul 25 00:52:04 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 242006 jiffies s: 2757133 root: 0x8/.
Jul 25 00:52:04 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 00:52:04 MOZART kernel: Task dump for CPU 3:
Jul 25 00:52:04 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 00:52:04 MOZART kernel: Call Trace:
Jul 25 00:52:04 MOZART kernel: <TASK>
Jul 25 00:52:04 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 00:52:04 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 00:52:04 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:52:04 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:52:04 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 00:52:04 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:52:04 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 00:52:04 MOZART kernel: </TASK>
Jul 25 00:54:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 00:54:57 MOZART kernel: rcu: #0113-....: (419941 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=104948 
Jul 25 00:54:57 MOZART kernel: #011(t=420006 jiffies g=440205233 q=1690772)
Jul 25 00:54:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 00:54:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 00:54:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 00:54:57 MOZART kernel: Call Trace:
Jul 25 00:54:57 MOZART kernel: <IRQ>
Jul 25 00:54:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 00:54:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 00:54:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 00:54:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 00:54:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 00:54:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 00:54:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 00:54:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 00:54:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 00:54:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 00:54:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 00:54:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 00:54:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 00:54:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 00:54:57 MOZART kernel: </IRQ>
Jul 25 00:54:57 MOZART kernel: <TASK>
Jul 25 00:54:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 00:54:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 00:54:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 00:54:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 00:54:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 00:54:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 00:54:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 00:54:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 00:54:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 00:54:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 00:54:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 00:54:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 00:54:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 00:54:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 00:54:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 00:54:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 00:54:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 00:54:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 00:54:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 00:54:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 00:54:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 00:54:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 00:54:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 00:54:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:54:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:54:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 00:54:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:54:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 00:54:57 MOZART kernel: </TASK>
Jul 25 00:55:04 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 422230 jiffies s: 2757133 root: 0x8/.
Jul 25 00:55:04 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 00:55:04 MOZART kernel: Task dump for CPU 3:
Jul 25 00:55:04 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 00:55:04 MOZART kernel: Call Trace:
Jul 25 00:55:04 MOZART kernel: <TASK>
Jul 25 00:55:04 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 00:55:04 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 00:55:04 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:55:04 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:55:04 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 00:55:04 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:55:04 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 00:55:04 MOZART kernel: </TASK>
Jul 25 00:57:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 00:57:57 MOZART kernel: rcu: #0113-....: (599916 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=149927 
Jul 25 00:57:57 MOZART kernel: #011(t=600009 jiffies g=440205233 q=2185924)
Jul 25 00:57:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 00:57:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 00:57:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 00:57:57 MOZART kernel: Call Trace:
Jul 25 00:57:57 MOZART kernel: <IRQ>
Jul 25 00:57:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 00:57:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 00:57:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 00:57:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 00:57:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 00:57:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 00:57:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 00:57:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 00:57:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 00:57:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 00:57:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 00:57:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 00:57:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 00:57:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 00:57:57 MOZART kernel: </IRQ>
Jul 25 00:57:57 MOZART kernel: <TASK>
Jul 25 00:57:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 00:57:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 00:57:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 00:57:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 00:57:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 00:57:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 00:57:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 00:57:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 00:57:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 00:57:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 00:57:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 00:57:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 00:57:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 00:57:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 00:57:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 00:57:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 00:57:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 00:57:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 00:57:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 00:57:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 00:57:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 00:57:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 00:57:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 00:57:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:57:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:57:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 00:57:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:57:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 00:57:57 MOZART kernel: </TASK>
Jul 25 00:58:04 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 602454 jiffies s: 2757133 root: 0x8/.
Jul 25 00:58:04 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 00:58:04 MOZART kernel: Task dump for CPU 3:
Jul 25 00:58:04 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 00:58:04 MOZART kernel: Call Trace:
Jul 25 00:58:04 MOZART kernel: <TASK>
Jul 25 00:58:04 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 00:58:04 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 00:58:04 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 00:58:04 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 00:58:04 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 00:58:04 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 00:58:04 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 00:58:04 MOZART kernel: </TASK>
Jul 25 01:00:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 01:00:57 MOZART kernel: rcu: #0113-....: (779891 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=194906 
Jul 25 01:00:57 MOZART kernel: #011(t=780012 jiffies g=440205233 q=2680626)
Jul 25 01:00:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 01:00:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 01:00:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 01:00:57 MOZART kernel: Call Trace:
Jul 25 01:00:57 MOZART kernel: <IRQ>
Jul 25 01:00:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 01:00:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 01:00:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 01:00:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 01:00:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 01:00:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 01:00:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 01:00:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 01:00:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 01:00:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 01:00:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 01:00:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 01:00:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 01:00:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 01:00:57 MOZART kernel: </IRQ>
Jul 25 01:00:57 MOZART kernel: <TASK>
Jul 25 01:00:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 01:00:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 01:00:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 01:00:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 01:00:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 01:00:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 01:00:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 01:00:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 01:00:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 01:00:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 01:00:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 01:00:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 01:00:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 01:00:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 01:00:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 01:00:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 01:00:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 01:00:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 01:00:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 01:00:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 01:00:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 01:00:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 01:00:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 01:00:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:00:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:00:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 01:00:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:00:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 01:00:57 MOZART kernel: </TASK>
Jul 25 01:01:04 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 782678 jiffies s: 2757133 root: 0x8/.
Jul 25 01:01:04 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 01:01:04 MOZART kernel: Task dump for CPU 3:
Jul 25 01:01:04 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 01:01:04 MOZART kernel: Call Trace:
Jul 25 01:01:04 MOZART kernel: <TASK>
Jul 25 01:01:04 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 01:01:04 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 01:01:04 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:01:04 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:01:04 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 01:01:04 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:01:04 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 01:01:04 MOZART kernel: </TASK>
Jul 25 01:03:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 01:03:57 MOZART kernel: rcu: #0113-....: (959866 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=239882 
Jul 25 01:03:57 MOZART kernel: #011(t=960015 jiffies g=440205233 q=3163400)
Jul 25 01:03:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 01:03:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 01:03:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 01:03:57 MOZART kernel: Call Trace:
Jul 25 01:03:57 MOZART kernel: <IRQ>
Jul 25 01:03:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 01:03:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 01:03:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 01:03:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 01:03:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 01:03:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 01:03:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 01:03:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 01:03:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 01:03:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 01:03:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 01:03:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 01:03:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 01:03:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 01:03:57 MOZART kernel: </IRQ>
Jul 25 01:03:57 MOZART kernel: <TASK>
Jul 25 01:03:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 01:03:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 01:03:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 01:03:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 01:03:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 01:03:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 01:03:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 01:03:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 01:03:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 01:03:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 01:03:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 01:03:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 01:03:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 01:03:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 01:03:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 01:03:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 01:03:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 01:03:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 01:03:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 01:03:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 01:03:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 01:03:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 01:03:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 01:03:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:03:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:03:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 01:03:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:03:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 01:03:57 MOZART kernel: </TASK>
Jul 25 01:04:05 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 962902 jiffies s: 2757133 root: 0x8/.
Jul 25 01:04:05 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 01:04:05 MOZART kernel: Task dump for CPU 3:
Jul 25 01:04:05 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 01:04:05 MOZART kernel: Call Trace:
Jul 25 01:04:05 MOZART kernel: <TASK>
Jul 25 01:04:05 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 01:04:05 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 01:04:05 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:04:05 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:04:05 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 01:04:05 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:04:05 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 01:04:05 MOZART kernel: </TASK>
Jul 25 01:06:34 MOZART emhttpd: spinning down /dev/sdb
Jul 25 01:06:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 01:06:57 MOZART kernel: rcu: #0113-....: (1139841 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=284856 
Jul 25 01:06:57 MOZART kernel: #011(t=1140018 jiffies g=440205233 q=3674408)
Jul 25 01:06:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 01:06:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 01:06:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 01:06:57 MOZART kernel: Call Trace:
Jul 25 01:06:57 MOZART kernel: <IRQ>
Jul 25 01:06:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 01:06:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 01:06:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 01:06:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 01:06:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 01:06:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 01:06:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 01:06:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 01:06:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 01:06:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 01:06:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 01:06:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 01:06:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 01:06:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 01:06:57 MOZART kernel: </IRQ>
Jul 25 01:06:57 MOZART kernel: <TASK>
Jul 25 01:06:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 01:06:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 01:06:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 01:06:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 01:06:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 01:06:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 01:06:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 01:06:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 01:06:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 01:06:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 01:06:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 01:06:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 01:06:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 01:06:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 01:06:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 01:06:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 01:06:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 01:06:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 01:06:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 01:06:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 01:06:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 01:06:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 01:06:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 01:06:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:06:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:06:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 01:06:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:06:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 01:06:57 MOZART kernel: </TASK>
Jul 25 01:07:05 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 1143126 jiffies s: 2757133 root: 0x8/.
Jul 25 01:07:05 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 01:07:05 MOZART kernel: Task dump for CPU 3:
Jul 25 01:07:05 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 01:07:05 MOZART kernel: Call Trace:
Jul 25 01:07:05 MOZART kernel: <TASK>
Jul 25 01:07:05 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 01:07:05 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 01:07:05 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:07:05 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:07:05 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 01:07:05 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:07:05 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 01:07:05 MOZART kernel: </TASK>
Jul 25 01:09:57 MOZART kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 25 01:09:57 MOZART kernel: rcu: #0113-....: (1319816 ticks this GP) idle=149/1/0x4000000000000000 softirq=205821660/205821660 fqs=329827 
Jul 25 01:09:57 MOZART kernel: #011(t=1320021 jiffies g=440205233 q=4202851)
Jul 25 01:09:57 MOZART kernel: NMI backtrace for cpu 3
Jul 25 01:09:57 MOZART kernel: CPU: 3 PID: 58 Comm: kcompactd0 Tainted: G      D W I       5.15.46-Unraid #1
Jul 25 01:09:57 MOZART kernel: Hardware name: Supermicro Super Server/X11SSH-LN4F, BIOS 2.2 05/23/2018
Jul 25 01:09:57 MOZART kernel: Call Trace:
Jul 25 01:09:57 MOZART kernel: <IRQ>
Jul 25 01:09:57 MOZART kernel: dump_stack_lvl+0x46/0x5a
Jul 25 01:09:57 MOZART kernel: nmi_cpu_backtrace+0xae/0xd2
Jul 25 01:09:57 MOZART kernel: ? lapic_can_unplug_cpu+0x93/0x93
Jul 25 01:09:57 MOZART kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Jul 25 01:09:57 MOZART kernel: rcu_dump_cpu_stacks+0xc3/0xea
Jul 25 01:09:57 MOZART kernel: rcu_sched_clock_irq+0x22d/0x631
Jul 25 01:09:57 MOZART kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Jul 25 01:09:57 MOZART kernel: ? tick_sched_do_timer+0x3e/0x3e
Jul 25 01:09:57 MOZART kernel: update_process_times+0x8c/0xab
Jul 25 01:09:57 MOZART kernel: tick_sched_timer+0x38/0x65
Jul 25 01:09:57 MOZART kernel: __hrtimer_run_queues+0xfa/0x18a
Jul 25 01:09:57 MOZART kernel: hrtimer_interrupt+0x92/0x160
Jul 25 01:09:57 MOZART kernel: __sysvec_apic_timer_interrupt+0x99/0xdb
Jul 25 01:09:57 MOZART kernel: sysvec_apic_timer_interrupt+0x61/0x7d
Jul 25 01:09:57 MOZART kernel: </IRQ>
Jul 25 01:09:57 MOZART kernel: <TASK>
Jul 25 01:09:57 MOZART kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 25 01:09:57 MOZART kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x18f
Jul 25 01:09:57 MOZART kernel: Code: c1 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 42 01 00 eb 14 85 c0 74 0a 8b 02 84 c0 74 04 f3 90 <eb> f6 66 c7 02 01 00 c3 48 c7 c1 40 c8 02 00 65 48 03 0d f7 93 f6
Jul 25 01:09:57 MOZART kernel: RSP: 0018:ffffc9000026bb20 EFLAGS: 00000202
Jul 25 01:09:57 MOZART kernel: RAX: 0000000000180101 RBX: ffffea0004457540 RCX: 0000000000000000
Jul 25 01:09:57 MOZART kernel: RDX: ffffea000f20b0e8 RSI: 0000000000000000 RDI: ffffea000f20b0e8
Jul 25 01:09:57 MOZART kernel: RBP: ffffc9000026bb70 R08: ffffea0004457540 R09: ffff8881789bd000
Jul 25 01:09:57 MOZART kernel: R10: 0000000000000100 R11: 00000000000305e0 R12: ffff88815840aec0
Jul 25 01:09:57 MOZART kernel: R13: 0000147636673000 R14: ffff8883c82c3390 R15: 0000000000000003
Jul 25 01:09:57 MOZART kernel: queued_spin_lock_slowpath+0x7/0xa
Jul 25 01:09:57 MOZART kernel: page_vma_mapped_walk+0x417/0x51b
Jul 25 01:09:57 MOZART kernel: remove_migration_pte+0x5e/0x22e
Jul 25 01:09:57 MOZART kernel: rmap_walk_anon+0xd2/0x13e
Jul 25 01:09:57 MOZART kernel: remove_migration_ptes+0x4e/0x68
Jul 25 01:09:57 MOZART kernel: ? page_is_file_lru+0x16/0x16
Jul 25 01:09:57 MOZART kernel: migrate_pages+0x632/0xa08
Jul 25 01:09:57 MOZART kernel: ? compact_lock_irqsave+0x5e/0x5e
Jul 25 01:09:57 MOZART kernel: ? release_freepages+0x8f/0x8f
Jul 25 01:09:57 MOZART kernel: compact_zone+0x84f/0xa29
Jul 25 01:09:57 MOZART kernel: ? __switch_to_xtra+0x50/0x3d0
Jul 25 01:09:57 MOZART kernel: ? __raw_spin_unlock+0x5/0x6
Jul 25 01:09:57 MOZART kernel: proactive_compact_node+0x7f/0xac
Jul 25 01:09:57 MOZART kernel: kcompactd+0x24e/0x29c
Jul 25 01:09:57 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:09:57 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:09:57 MOZART kernel: kthread+0xde/0xe3
Jul 25 01:09:57 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:09:57 MOZART kernel: ret_from_fork+0x22/0x30
Jul 25 01:09:57 MOZART kernel: </TASK>
Jul 25 01:10:00 MOZART webGUI: Successful login user root from 192.168.92.185
Jul 25 01:10:05 MOZART kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 1323350 jiffies s: 2757133 root: 0x8/.
Jul 25 01:10:05 MOZART kernel: rcu: blocking rcu_node structures (internal RCU debug):
Jul 25 01:10:05 MOZART kernel: Task dump for CPU 3:
Jul 25 01:10:05 MOZART kernel: task:kcompactd0      state:R  running task     stack:    0 pid:   58 ppid:     2 flags:0x00004808
Jul 25 01:10:05 MOZART kernel: Call Trace:
Jul 25 01:10:05 MOZART kernel: <TASK>
Jul 25 01:10:05 MOZART kernel: ? proactive_compact_node+0x7f/0xac
Jul 25 01:10:05 MOZART kernel: ? kcompactd+0x24e/0x29c
Jul 25 01:10:05 MOZART kernel: ? init_wait_entry+0x29/0x29
Jul 25 01:10:05 MOZART kernel: ? kcompactd_do_work+0x1bd/0x1bd
Jul 25 01:10:05 MOZART kernel: ? kthread+0xde/0xe3
Jul 25 01:10:05 MOZART kernel: ? set_kthread_struct+0x32/0x32
Jul 25 01:10:05 MOZART kernel: ? ret_from_fork+0x22/0x30
Jul 25 01:10:05 MOZART kernel: </TASK>

 

 

 

 

Edited by Opawesome
fixed typos
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing