- Urgent
Happened 3 times over 2 weeks following upgrade to 6.12.3. Previously ran with 6months uptime before upgrading
Symptoms :
I have a Docker instance running PiHole. The first indication is the DNS entries stop resolving as this Docker container crashes.
If I'm quick I can get onto the WebUI and see 1 x CPU core at 100% (Intel® Xeon® CPU E3-1226 v3 @ 3.30GHz - 4 cores)
As soon as I try to stop the array etc the rest of the cores follow until the server becomes unresponsive requiring hard reset
I managed to retrieve a log for the latest crash as the log console responded before the systam started running hot.
I do have Syslog server enabled and a share created to store the logs, but no files have been logged to that share. This is what the log console contaned as the system began to fail.
-------------------------------
Aug 23 16:14:33 Tower kernel: Workqueue: events bpf_prog_free_deferred
Aug 23 16:14:33 Tower kernel: RIP: 0010:free_vmap_area_noflush+0x222/0x285
Aug 23 16:14:33 Tower kernel: Code: 46 2e 82 31 c0 48 89 73 10 4c 89 ff 48 c7 c6 b8 88 97 82 48 89 43 18 48 89 43 20 4c 89 39 e8 e0 71 65 00 49 8b 16 48 8d 43 28 <48> 89 42 08 48 89 53 28 4c 89 73 30 49 89 06 48 c7 c7 b0 88 97 82
Aug 23 16:14:33 Tower kernel: RSP: 0018:ffffc90013be7e20 EFLAGS: 00010202
Aug 23 16:14:33 Tower kernel: RAX: ffff8881a7d98f68 RBX: ffff8881a7d98f40 RCX: ffff888158be1a68
Aug 23 16:14:33 Tower kernel: RDX: 0000000000000000 RSI: ffff888158be1a60 RDI: ffff8882c2a25b91
Aug 23 16:14:33 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff829788b8
Aug 23 16:14:33 Tower kernel: R10: ffff8881a7c88800 R11: fefefefefefefeff R12: 0000000000000000
Aug 23 16:14:33 Tower kernel: R13: 0000000000004833 R14: ffff888158be1a78 R15: ffff8881a7d98f50
Aug 23 16:14:33 Tower kernel: FS: 0000000000000000(0000) GS:ffff88880dd00000(0000) knlGS:0000000000000000
Aug 23 16:14:33 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 23 16:14:33 Tower kernel: CR2: 0000000000000008 CR3: 000000000220a003 CR4: 00000000001706e0
Aug 23 16:14:33 Tower kernel: Call Trace:
Aug 23 16:14:33 Tower kernel: <TASK>
Aug 23 16:14:33 Tower kernel: ? __die_body+0x1a/0x5c
Aug 23 16:14:33 Tower kernel: ? page_fault_oops+0x329/0x376
Aug 23 16:14:33 Tower kernel: ? fixup_exception+0x22/0x24b
Aug 23 16:14:33 Tower kernel: ? exc_page_fault+0xfb/0x11d
Aug 23 16:14:33 Tower kernel: ? asm_exc_page_fault+0x22/0x30
Aug 23 16:14:33 Tower kernel: ? free_vmap_area_noflush+0x222/0x285
Aug 23 16:14:33 Tower kernel: remove_vm_area+0x5a/0x74
Aug 23 16:14:33 Tower kernel: __vunmap+0x7c/0x1a3
Aug 23 16:14:33 Tower kernel: process_one_work+0x1ab/0x295
Aug 23 16:14:33 Tower kernel: worker_thread+0x18b/0x244
Aug 23 16:14:33 Tower kernel: ? rescuer_thread+0x281/0x281
Aug 23 16:14:33 Tower kernel: kthread+0xe7/0xef
Aug 23 16:14:33 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 23 16:14:33 Tower kernel: ret_from_fork+0x22/0x30
Aug 23 16:14:33 Tower kernel: </TASK>
Aug 23 16:14:33 Tower kernel: Modules linked in: udp_diag xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls amdgpu gpu_sched drm_buddy intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm radeon crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 video sr_mod i2c_algo_bit aesni_intel cdrom drm_ttm_helper crypto_simd ttm cryptd drm_display_helper rapl mei_hdcp mei_pxp wmi_bmof intel_cstate drm_kms_helper tpm_infineon intel_uncore ahci drm i2c_i801 i2c_smbus backlight agpgart
Aug 23 16:14:33 Tower kernel: libahci e1000e i2c_core mei_me syscopyarea sysfillrect mei tpm_tis sysimgblt tpm_tis_core fb_sys_fops fan thermal wmi tpm button unix
Aug 23 16:14:33 Tower kernel: CR2: 0000000000000008
Aug 23 16:14:33 Tower kernel: ---[ end trace 0000000000000000 ]---
Aug 23 16:14:33 Tower kernel: RIP: 0010:free_vmap_area_noflush+0x222/0x285
Aug 23 16:14:33 Tower kernel: Code: 46 2e 82 31 c0 48 89 73 10 4c 89 ff 48 c7 c6 b8 88 97 82 48 89 43 18 48 89 43 20 4c 89 39 e8 e0 71 65 00 49 8b 16 48 8d 43 28 <48> 89 42 08 48 89 53 28 4c 89 73 30 49 89 06 48 c7 c7 b0 88 97 82
Aug 23 16:14:33 Tower kernel: RSP: 0018:ffffc90013be7e20 EFLAGS: 00010202
Aug 23 16:14:33 Tower kernel: RAX: ffff8881a7d98f68 RBX: ffff8881a7d98f40 RCX: ffff888158be1a68
Aug 23 16:14:33 Tower kernel: RDX: 0000000000000000 RSI: ffff888158be1a60 RDI: ffff8882c2a25b91
Aug 23 16:14:33 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff829788b8
Aug 23 16:14:33 Tower kernel: R10: ffff8881a7c88800 R11: fefefefefefefeff R12: 0000000000000000
Aug 23 16:14:33 Tower kernel: R13: 0000000000004833 R14: ffff888158be1a78 R15: ffff8881a7d98f50
Aug 23 16:14:33 Tower kernel: FS: 0000000000000000(0000) GS:ffff88880dd00000(0000) knlGS:0000000000000000
Aug 23 16:14:33 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 23 16:14:33 Tower kernel: CR2: 0000000000000008 CR3: 000000000220a003 CR4: 00000000001706e0
Aug 23 16:14:33 Tower kernel: note: kworker/2:2[12330] exited with irqs disabled
Aug 23 16:14:33 Tower kernel: note: kworker/2:2[12330] exited with preempt_count 1
Aug 23 16:15:33 Tower kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 23 16:15:33 Tower kernel: rcu: 1-....: (60000 ticks this GP) idle=435c/1/0x4000000000000000 softirq=4661696/4661734 fqs=18620
Aug 23 16:15:33 Tower kernel: (t=60001 jiffies g=7201641 q=86503 ncpus=4)
Aug 23 16:15:33 Tower kernel: CPU: 1 PID: 5507 Comm: kworker/1:1 Tainted: P D W O 6.1.38-Unraid #2
Aug 23 16:15:33 Tower kernel: Hardware name: Hewlett-Packard HP Z230 Tower Workstation/1905, BIOS L51 v01.63 04/22/2020
Aug 23 16:15:33 Tower kernel: Workqueue: events free_work
Aug 23 16:15:33 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf
Aug 23 16:15:33 Tower kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 4a 3f ff ff 49 c7 c4 80 e1
Aug 23 16:15:33 Tower kernel: RSP: 0018:ffffc90007c83de0 EFLAGS: 00000202
Aug 23 16:15:33 Tower kernel: RAX: 0000000000000101 RBX: ffffffff829788b0 RCX: ffff88823a32c451
Aug 23 16:15:33 Tower kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff829788b0
Aug 23 16:15:33 Tower kernel: RBP: ffff8881031d0500 R08: ffffffff829788c0 R09: 0000000000000000
Aug 23 16:15:33 Tower kernel: R10: 8080808080808080 R11: fefefefefefefeff R12: 0000000000000000
Aug 23 16:15:33 Tower kernel: R13: 0000000000004838 R14: 0000000000000000 R15: ffff88815783a7d0
Aug 23 16:15:33 Tower kernel: FS: 0000000000000000(0000) GS:ffff88880dc80000(0000) knlGS:0000000000000000
Aug 23 16:15:33 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 23 16:15:33 Tower kernel: CR2: 000000c0001a3000 CR3: 000000013ef56001 CR4: 00000000001706e0
Aug 23 16:15:33 Tower kernel: Call Trace:
Aug 23 16:15:33 Tower kernel: <IRQ>
Aug 23 16:15:33 Tower kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Aug 23 16:15:33 Tower kernel: ? rcu_sched_clock_irq+0x337/0xa37
Aug 23 16:15:33 Tower kernel: ? tick_init_jiffy_update+0x7c/0x7c
Aug 23 16:15:33 Tower kernel: ? update_process_times+0x62/0x81
Aug 23 16:15:33 Tower kernel: ? tick_sched_timer+0x43/0x71
Aug 23 16:15:33 Tower kernel: ? __hrtimer_run_queues+0xeb/0x190
Aug 23 16:15:33 Tower kernel: ? hrtimer_interrupt+0x9c/0x16e
Aug 23 16:15:33 Tower kernel: ? __sysvec_apic_timer_interrupt+0xc5/0x12f
Aug 23 16:15:33 Tower kernel: ? sysvec_apic_timer_interrupt+0x80/0xa6
Aug 23 16:15:33 Tower kernel: </IRQ>
Aug 23 16:15:33 Tower kernel: <TASK>
Aug 23 16:15:33 Tower kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Aug 23 16:15:33 Tower kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf
Aug 23 16:15:33 Tower kernel: do_raw_spin_lock+0x14/0x1a
Aug 23 16:15:33 Tower kernel: free_vmap_area_noflush+0x7a/0x285
Aug 23 16:15:33 Tower kernel: remove_vm_area+0x5a/0x74
Aug 23 16:15:33 Tower kernel: __vunmap+0x7c/0x1a3
Aug 23 16:15:33 Tower kernel: free_work+0x26/0x34
Aug 23 16:15:33 Tower kernel: process_one_work+0x1ab/0x295
Aug 23 16:15:33 Tower kernel: worker_thread+0x18b/0x244
Aug 23 16:15:33 Tower kernel: ? rescuer_thread+0x281/0x281
Aug 23 16:15:33 Tower kernel: kthread+0xe7/0xef
Aug 23 16:15:33 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 23 16:15:33 Tower kernel: ret_from_fork+0x22/0x30
Aug 23 16:15:33 Tower kernel: </TASK>
---------------------------