[6.12.10] Can anyone tell me what might be causing this?

[rossiscool123]

Hello all,

My current home server has had a fair history of being unstable but today I had it lock up with something interesting in the logs that maybe someone can explain what caused it. I'll post  the diagnosis but also a snippet from my log before the server became inaccessible and locked up with no display output.

 

Apr 26 17:24:28 Cerberus kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth2522d83: link becomes ready
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth2522d83) entered disabled state
Apr 26 17:24:55 Cerberus kernel: veth7502abf: renamed from eth0
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth2522d83) entered disabled state
Apr 26 17:24:55 Cerberus kernel: device veth2522d83 left promiscuous mode
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth2522d83) entered disabled state
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth99c133d) entered blocking state
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth99c133d) entered disabled state
Apr 26 17:24:55 Cerberus kernel: device veth99c133d entered promiscuous mode
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth99c133d) entered blocking state
Apr 26 17:24:55 Cerberus kernel: docker0: port 1(veth99c133d) entered forwarding state
Apr 26 17:24:55 Cerberus kernel: eth0: renamed from veth632fc46
Apr 26 17:24:55 Cerberus kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth99c133d: link becomes ready
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth99c133d) entered disabled state
Apr 26 17:25:22 Cerberus kernel: veth632fc46: renamed from eth0
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth99c133d) entered disabled state
Apr 26 17:25:22 Cerberus kernel: device veth99c133d left promiscuous mode
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth99c133d) entered disabled state
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth3bed3c1) entered blocking state
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth3bed3c1) entered disabled state
Apr 26 17:25:22 Cerberus kernel: device veth3bed3c1 entered promiscuous mode
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth3bed3c1) entered blocking state
Apr 26 17:25:22 Cerberus kernel: docker0: port 1(veth3bed3c1) entered forwarding state
Apr 26 17:25:22 Cerberus kernel: eth0: renamed from veth99fcc42
Apr 26 17:25:22 Cerberus kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth3bed3c1: link becomes ready
Apr 26 17:25:45 Cerberus kernel: docker0: port 1(veth3bed3c1) entered disabled state
Apr 26 17:25:45 Cerberus kernel: veth99fcc42: renamed from eth0
Apr 26 17:25:45 Cerberus kernel: docker0: port 1(veth3bed3c1) entered disabled state
Apr 26 17:25:45 Cerberus kernel: device veth3bed3c1 left promiscuous mode
Apr 26 17:25:45 Cerberus kernel: docker0: port 1(veth3bed3c1) entered disabled state
Apr 26 17:25:46 Cerberus kernel: docker0: port 1(vethbb5725b) entered blocking state
Apr 26 17:25:46 Cerberus kernel: docker0: port 1(vethbb5725b) entered disabled state
Apr 26 17:25:46 Cerberus kernel: device vethbb5725b entered promiscuous mode
Apr 26 17:25:46 Cerberus kernel: docker0: port 1(vethbb5725b) entered blocking state
Apr 26 17:25:46 Cerberus kernel: docker0: port 1(vethbb5725b) entered forwarding state
Apr 26 17:25:46 Cerberus kernel: eth0: renamed from vethde12cb4
Apr 26 17:25:46 Cerberus kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethbb5725b: link becomes ready
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethbb5725b) entered disabled state
Apr 26 17:26:15 Cerberus kernel: vethde12cb4: renamed from eth0
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethbb5725b) entered disabled state
Apr 26 17:26:15 Cerberus kernel: device vethbb5725b left promiscuous mode
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethbb5725b) entered disabled state
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethc879951) entered blocking state
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethc879951) entered disabled state
Apr 26 17:26:15 Cerberus kernel: device vethc879951 entered promiscuous mode
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethc879951) entered blocking state
Apr 26 17:26:15 Cerberus kernel: docker0: port 1(vethc879951) entered forwarding state
Apr 26 17:26:15 Cerberus kernel: eth0: renamed from vethb6d9465
Apr 26 17:26:15 Cerberus kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethc879951: link becomes ready
Apr 26 17:26:36 Cerberus kernel: general protection fault, probably for non-canonical address 0x636c81c1ee0b8c88: 0000 [#1] PREEMPT SMP NOPTI
Apr 26 17:26:36 Cerberus kernel: CPU: 0 PID: 119 Comm: khugepaged Tainted: P           O       6.1.79-Unraid #1
Apr 26 17:26:36 Cerberus kernel: Hardware name: ASUS System Product Name/ROG STRIX B560-F GAMING WIFI, BIOS 2001 02/01/2023
Apr 26 17:26:36 Cerberus kernel: RIP: 0010:dentry_unlink_inode+0xe0/0xef
Apr 26 17:26:36 Cerberus kernel: Code: 31 c9 e8 c7 6b 02 00 58 48 89 ef e8 fa 6a 02 00 48 8b 43 60 48 85 c0 74 16 48 8b 40 40 48 85 c0 74 0d 48 89 ee 48 89 df 5b 5d <ff> e0 cc 66 90 5b 48 89 ef 5d e9 73 38 00 00 48 0f ba 37 00 bf 01
Apr 26 17:26:36 Cerberus kernel: RSP: 0018:ffffc90000513868 EFLAGS: 00010206
Apr 26 17:26:36 Cerberus kernel: RAX: 636c81c1ee0b8c88 RBX: ffff88843e3798c0 RCX: 0000000000000000
Apr 26 17:26:36 Cerberus kernel: RDX: 0000000000000001 RSI: ffff88843d932118 RDI: ffff88843e3798c0
Apr 26 17:26:36 Cerberus kernel: RBP: ffff888450dcc6c0 R08: 0000000000000000 R09: 000000000000024a
Apr 26 17:26:36 Cerberus kernel: R10: ffff888450b27700 R11: 0000000000000000 R12: ffff88843e379918
Apr 26 17:26:36 Cerberus kernel: R13: ffff88843e379918 R14: 0000000000202627 R15: 0000000000000000
Apr 26 17:26:36 Cerberus kernel: FS:  0000000000000000(0000) GS:ffff888c2c200000(0000) knlGS:0000000000000000
Apr 26 17:26:36 Cerberus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 26 17:26:36 Cerberus kernel: CR2: 000000c000133010 CR3: 00000001deb70006 CR4: 00000000007726f0
Apr 26 17:26:36 Cerberus kernel: PKRU: 55555554
Apr 26 17:26:36 Cerberus kernel: Call Trace:
Apr 26 17:26:36 Cerberus kernel: <TASK>
Apr 26 17:26:36 Cerberus kernel: ? __die_body+0x1a/0x5c
Apr 26 17:26:36 Cerberus kernel: ? die_addr+0x38/0x51
Apr 26 17:26:36 Cerberus kernel: ? exc_general_protection+0x30f/0x345
Apr 26 17:26:36 Cerberus kernel: ? asm_exc_general_protection+0x22/0x30
Apr 26 17:26:36 Cerberus kernel: ? dentry_unlink_inode+0xe0/0xef
Apr 26 17:26:36 Cerberus kernel: __dentry_kill+0xcb/0x131
Apr 26 17:26:36 Cerberus kernel: shrink_dentry_list+0xaa/0xba
Apr 26 17:26:36 Cerberus kernel: prune_dcache_sb+0x51/0x73
Apr 26 17:26:36 Cerberus kernel: super_cache_scan+0xf4/0x17c
Apr 26 17:26:36 Cerberus kernel: do_shrink_slab+0x188/0x2a1
Apr 26 17:26:36 Cerberus kernel: shrink_slab+0x1f9/0x267
Apr 26 17:26:36 Cerberus kernel: shrink_node+0x318/0x549
Apr 26 17:26:36 Cerberus kernel: do_try_to_free_pages+0x1b1/0x3a7
Apr 26 17:26:36 Cerberus kernel: try_to_free_pages+0x102/0x181
Apr 26 17:26:36 Cerberus kernel: __alloc_pages_slowpath.constprop.0+0x3a4/0x8de
Apr 26 17:26:36 Cerberus kernel: ? free_unref_page_list+0x25a/0x27d
Apr 26 17:26:36 Cerberus kernel: ? release_pages+0x2e6/0x30e
Apr 26 17:26:36 Cerberus kernel: __alloc_pages+0x132/0x1e8
Apr 26 17:26:36 Cerberus kernel: alloc_charge_hpage+0xd0/0x188
Apr 26 17:26:36 Cerberus kernel: collapse_huge_page+0x62/0xa49
Apr 26 17:26:36 Cerberus kernel: hpage_collapse_scan_pmd+0x344/0x40d
Apr 26 17:26:36 Cerberus kernel: khugepaged+0x31d/0x640
Apr 26 17:26:36 Cerberus kernel: ? __schedule+0x5ba/0x612
Apr 26 17:26:36 Cerberus kernel: ? preempt_latency_start+0x2b/0x46
Apr 26 17:26:36 Cerberus kernel: ? collapse_pte_mapped_thp+0x383/0x383
Apr 26 17:26:36 Cerberus kernel: kthread+0xe4/0xef
Apr 26 17:26:36 Cerberus kernel: ? kthread_complete_and_exit+0x1b/0x1b
Apr 26 17:26:36 Cerberus kernel: ret_from_fork+0x1f/0x30
Apr 26 17:26:36 Cerberus kernel: </TASK>
Apr 26 17:26:36 Cerberus kernel: Modules linked in: nf_tables xt_connmark xt_comment iptable_raw xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) nct6775 nct6775_core hwmon_vid corefreqk(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge 8021q garp mrp stp llc i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm kvm drm_display_helper crct10dif_pclmul crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel
Apr 26 17:26:36 Cerberus kernel: sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel drm crypto_simd cryptd btusb btrtl rapl btbcm btintel intel_cstate mei_hdcp mei_pxp nvme wmi_bmof bluetooth mpt3sas intel_gtt intel_uncore i2c_i801 mei_me agpgart i2c_smbus igc nvme_core mei ahci i2c_core raid_class ecdh_generic libahci syscopyarea ecc sysfillrect scsi_transport_sas sysimgblt fb_sys_fops thermal fan tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix
Apr 26 17:26:36 Cerberus kernel: ---[ end trace 0000000000000000 ]---
Apr 26 17:26:36 Cerberus kernel: RIP: 0010:dentry_unlink_inode+0xe0/0xef
Apr 26 17:26:36 Cerberus kernel: Code: 31 c9 e8 c7 6b 02 00 58 48 89 ef e8 fa 6a 02 00 48 8b 43 60 48 85 c0 74 16 48 8b 40 40 48 85 c0 74 0d 48 89 ee 48 89 df 5b 5d <ff> e0 cc 66 90 5b 48 89 ef 5d e9 73 38 00 00 48 0f ba 37 00 bf 01
Apr 26 17:26:36 Cerberus kernel: RSP: 0018:ffffc90000513868 EFLAGS: 00010206
Apr 26 17:26:36 Cerberus kernel: RAX: 636c81c1ee0b8c88 RBX: ffff88843e3798c0 RCX: 0000000000000000
Apr 26 17:26:36 Cerberus kernel: RDX: 0000000000000001 RSI: ffff88843d932118 RDI: ffff88843e3798c0
Apr 26 17:26:36 Cerberus kernel: RBP: ffff888450dcc6c0 R08: 0000000000000000 R09: 000000000000024a
Apr 26 17:26:36 Cerberus kernel: R10: ffff888450b27700 R11: 0000000000000000 R12: ffff88843e379918
Apr 26 17:26:36 Cerberus kernel: R13: ffff88843e379918 R14: 0000000000202627 R15: 0000000000000000
Apr 26 17:26:36 Cerberus kernel: FS:  0000000000000000(0000) GS:ffff888c2c200000(0000) knlGS:0000000000000000
Apr 26 17:26:36 Cerberus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 26 17:26:36 Cerberus kernel: CR2: 000000c000133010 CR3: 00000001deb70006 CR4: 00000000007726f0
Apr 26 17:26:36 Cerberus kernel: PKRU: 55555554

 

cerberus-diagnostics-20240427-2135.zip

[rossiscool123]

Before anyone posts about memory and doing a memtest. I've done this plenty of times before running it for over 24h+ with over 4 passes and not a single error.

[bmartino1]

you appear to have a macvlan bug trace. its weird to see it on a veth interface, though.

I need to know the flags for your network interface to verify promisc mode

what is the output of netstat -i

I would recommend editing your unraid network settings and turn off bridging.