Is the CPU/Motherboard dying on me? Null pointers, CMCI Storms and tainted kernals.

[Meehan]

TL;DR: Server will intermittently lock up requiring a hardreset. It may then run for a few days without issue, or it might immediatly lockup again upon reboot, requiring multiple powercycles to regain functionality. Are Null pointers, CMCI Storms and tainted kernals indicitave of an old CPU dying?

 

The server was built using my old gaming PC parts - specifically an i5 3570k and a MSI - Z77A-GD65 motherboard which I have added drives to using the mothboards SATA headers (no HBA cards for me). The CPU and Mobo are from 2012 so they're certainly some pretty old hardward that have seen extensive use over the last decade+.

 

I have run a 10 hour Memtest84 load on the 16GB of RAM and they have reported no errors.

 

I am not super "syslog literate" but at lines 1333 - 1385 & 1548 - 1626: it looks like something is going wrong. Lines 1548 - 1626 are the exact moment that the server locked up and would not respond to pings, ssh logins, web-ui etc. and can only be recovered by repeatedly powercycling the system.

 

I'm hoping some of the more experienced members of the forum might have an idea of what's going on or where I should start looking.

 

 

______

 

Syslog exerpts

 

Lines 1333 - 1385:

Aug 18 23:06:56 Zeus kernel: BUG: unable to handle page fault for address: 0000000000032190
Aug 18 23:06:56 Zeus kernel: #PF: supervisor write access in kernel mode
Aug 18 23:06:56 Zeus kernel: #PF: error_code(0x0002) - not-present page
Aug 18 23:06:56 Zeus kernel: PGD 8000000145484067 P4D 8000000145484067 PUD 13f5b6067 PMD 0 
Aug 18 23:06:56 Zeus kernel: Oops: 0002 [#1] PREEMPT SMP PTI
Aug 18 23:06:56 Zeus kernel: CPU: 0 PID: 207 Comm: kworker/u8:4 Tainted: P           O       6.1.38-Unraid #2
Aug 18 23:06:56 Zeus kernel: Hardware name: MSI MS-7751/Z77A-GD65 GAMING (MS-7751), BIOS V25.0 03/20/2013
Aug 18 23:06:56 Zeus kernel: Workqueue: btrfs-endio btrfs_end_bio_work
Aug 18 23:06:56 Zeus kernel: RIP: 0010:endio_readpage_release_extent+0xb0/0xb7
Aug 18 23:06:56 Zeus kernel: Code: 89 6b 08 48 89 6b 10 44 88 63 18 48 8b 44 24 08 65 48 2b 04 25 28 00 00 00 74 05 e8 0c cb 51 00 48 83 c4 10 5b 5d 41 5c 41 5d <41> 5e c3 cc cc cc cc e8 de e2 ff ff f0 48 0f ba 28 00 0f 93 c0 0f
Aug 18 23:06:56 Zeus kernel: RSP: 0018:ffffc9000046bdb0 EFLAGS: 00010286
Aug 18 23:06:56 Zeus kernel: RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000000
Aug 18 23:06:56 Zeus kernel: RDX: 0000000001203400 RSI: ffffc9000046bcd8 RDI: 0000000000032190
Aug 18 23:06:56 Zeus kernel: RBP: ffffc9000046be90 R08: ffff8881615011f8 R09: ffffffff813b89df
Aug 18 23:06:56 Zeus kernel: R10: ffff88815a36f750 R11: fefefefefefefeff R12: 00000001ea9e3000
Aug 18 23:06:56 Zeus kernel: R13: 0000000500000000 R14: 0000000000000000 R15: ffff8881357e8ae8
Aug 18 23:06:56 Zeus kernel: FS:  0000000000000000(0000) GS:ffff88840f600000(0000) knlGS:0000000000000000
Aug 18 23:06:56 Zeus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 18 23:06:56 Zeus kernel: CR2: 0000000000032190 CR3: 0000000104cb0001 CR4: 00000000001706f0
Aug 18 23:06:56 Zeus kernel: Call Trace:
Aug 18 23:06:56 Zeus kernel: <TASK>
Aug 18 23:06:56 Zeus kernel: ? __die_body+0x1a/0x5c
Aug 18 23:06:56 Zeus kernel: ? page_fault_oops+0x329/0x376
Aug 18 23:06:56 Zeus kernel: ? fixup_exception+0x22/0x24b
Aug 18 23:06:56 Zeus kernel: ? exc_page_fault+0xfb/0x11d
Aug 18 23:06:56 Zeus kernel: ? asm_exc_page_fault+0x22/0x30
Aug 18 23:06:56 Zeus kernel: ? __clear_extent_bit+0x314/0x329
Aug 18 23:06:56 Zeus kernel: ? endio_readpage_release_extent+0xb0/0xb7
Aug 18 23:06:56 Zeus kernel: end_bio_extent_readpage+0x4a5/0x4fe
Aug 18 23:06:56 Zeus kernel: ? process_one_work+0x1ab/0x295
Aug 18 23:06:56 Zeus kernel: process_one_work+0x1ab/0x295
Aug 18 23:06:56 Zeus kernel: worker_thread+0x18b/0x244
Aug 18 23:06:56 Zeus kernel: ? rescuer_thread+0x281/0x281
Aug 18 23:06:56 Zeus kernel: kthread+0xe7/0xef
Aug 18 23:06:56 Zeus kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 18 23:06:56 Zeus kernel: ret_from_fork+0x22/0x30
Aug 18 23:06:56 Zeus kernel: </TASK>
Aug 18 23:06:56 Zeus kernel: Modules linked in: macvlan veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) f71882fg iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 drm aesni_intel crypto_simd cryptd rapl intel_cstate mei_hdcp mei_pxp mxm_wmi intel_gtt i2c_i801 intel_uncore agpgart mei_me i2c_smbus i2c_core syscopyarea
Aug 18 23:06:56 Zeus kernel: sysfillrect sysimgblt firewire_ohci ahci alx firewire_core libahci mdio mei fb_sys_fops fan thermal video wmi backlight button unix
Aug 18 23:06:56 Zeus kernel: CR2: 0000000000032190
Aug 18 23:06:56 Zeus kernel: ---[ end trace 0000000000000000 ]---
Aug 18 23:06:56 Zeus kernel: RIP: 0010:endio_readpage_release_extent+0xb0/0xb7
Aug 18 23:06:56 Zeus kernel: Code: 89 6b 08 48 89 6b 10 44 88 63 18 48 8b 44 24 08 65 48 2b 04 25 28 00 00 00 74 05 e8 0c cb 51 00 48 83 c4 10 5b 5d 41 5c 41 5d <41> 5e c3 cc cc cc cc e8 de e2 ff ff f0 48 0f ba 28 00 0f 93 c0 0f
Aug 18 23:06:56 Zeus kernel: RSP: 0018:ffffc9000046bdb0 EFLAGS: 00010286
Aug 18 23:06:56 Zeus kernel: RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000000
Aug 18 23:06:56 Zeus kernel: RDX: 0000000001203400 RSI: ffffc9000046bcd8 RDI: 0000000000032190
Aug 18 23:06:56 Zeus kernel: RBP: ffffc9000046be90 R08: ffff8881615011f8 R09: ffffffff813b89df
Aug 18 23:06:56 Zeus kernel: R10: ffff88815a36f750 R11: fefefefefefefeff R12: 00000001ea9e3000
Aug 18 23:06:56 Zeus kernel: R13: 0000000500000000 R14: 0000000000000000 R15: ffff8881357e8ae8
Aug 18 23:06:56 Zeus kernel: FS:  0000000000000000(0000) GS:ffff88840f600000(0000) knlGS:0000000000000000
Aug 18 23:06:56 Zeus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 18 23:06:56 Zeus kernel: CR2: 0000000000032190 CR3: 0000000104cb0001 CR4: 00000000001706f0
Aug 18 23:06:56 Zeus kernel: note: kworker/u8:4[207] exited with irqs disabled

 

1548 - 1626:

 

Aug 18 23:07:16 Zeus kernel: BUG: kernel NULL pointer dereference, address: 0000000000000065
Aug 18 23:07:16 Zeus kernel: #PF: supervisor read access in kernel mode
Aug 18 23:07:16 Zeus kernel: #PF: error_code(0x0000) - not-present page
Aug 18 23:07:16 Zeus kernel: PGD 80000002f3602067 P4D 80000002f3602067 PUD 2f357b067 PMD 0 
Aug 18 23:07:16 Zeus kernel: Oops: 0000 [#2] PREEMPT SMP PTI
Aug 18 23:07:16 Zeus kernel: CPU: 0 PID: 10285 Comm: influxd Tainted: P      D    O       6.1.38-Unraid #2
Aug 18 23:07:16 Zeus kernel: Hardware name: MSI MS-7751/Z77A-GD65 GAMING (MS-7751), BIOS V25.0 03/20/2013
Aug 18 23:07:16 Zeus kernel: RIP: 0010:_btrfs_printk+0x6f/0x154
Aug 18 23:07:16 Zeus kernel: Code: 48 89 54 24 40 31 d2 48 89 74 24 18 48 8d 74 24 48 66 c7 44 24 2c 00 00 c6 44 24 2e 00 c7 44 24 10 10 00 00 00 48 89 74 24 20 <80> 38 01 75 4b 8a 48 01 84 c9 74 44 80 f9 37 0f be d1 7f 31 80 f9
Aug 18 23:07:16 Zeus kernel: RSP: 0018:ffffc9000a1ef920 EFLAGS: 00010246
Aug 18 23:07:16 Zeus kernel: RAX: 0000000000000065 RBX: 0000000045879000 RCX: 0000000000004000
Aug 18 23:07:16 Zeus kernel: RDX: 0000000000000000 RSI: ffffc9000a1ef968 RDI: ffffffff822fc5c0
Aug 18 23:07:16 Zeus kernel: RBP: ffffc9000a1ef9a8 R08: 0000000000000000 R09: 0000000000000011
Aug 18 23:07:16 Zeus kernel: R10: 00000000000071b4 R11: ffff8881fff82f68 R12: ffffffff8207bfb6
Aug 18 23:07:16 Zeus kernel: R13: ffff8881a91c5000 R14: 0000000000000000 R15: ffffc9000a1efaf7
Aug 18 23:07:16 Zeus kernel: FS:  0000145ffdff3b20(0000) GS:ffff88840f600000(0000) knlGS:0000000000000000
Aug 18 23:07:16 Zeus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 18 23:07:16 Zeus kernel: CR2: 0000000000000065 CR3: 00000002ecfda003 CR4: 00000000001706f0
Aug 18 23:07:16 Zeus kernel: Call Trace:
Aug 18 23:07:16 Zeus kernel: <TASK>
Aug 18 23:07:16 Zeus kernel: ? __die_body+0x1a/0x5c
Aug 18 23:07:16 Zeus kernel: ? page_fault_oops+0x329/0x376
Aug 18 23:07:16 Zeus kernel: ? fixup_exception+0x22/0x24b
Aug 18 23:07:16 Zeus kernel: ? exc_page_fault+0xfb/0x11d
Aug 18 23:07:16 Zeus kernel: ? asm_exc_page_fault+0x22/0x30
Aug 18 23:07:16 Zeus kernel: ? _btrfs_printk+0x6f/0x154
Aug 18 23:07:16 Zeus kernel: check_eb_range+0x2f/0x39
Aug 18 23:07:16 Zeus kernel: read_extent_buffer+0x22/0x9b
Aug 18 23:07:16 Zeus kernel: btrfs_verify_level_key+0x100/0x175
Aug 18 23:07:16 Zeus kernel: read_block_for_search+0x10a/0x2a1
Aug 18 23:07:16 Zeus kernel: btrfs_next_old_leaf+0x1bf/0x343
Aug 18 23:07:16 Zeus kernel: ? kmem_cache_alloc+0x122/0x14d
Aug 18 23:07:16 Zeus kernel: btrfs_load_inode_props+0x9c/0x300
Aug 18 23:07:16 Zeus kernel: ? __set_extent_bit+0x19f/0x499
Aug 18 23:07:16 Zeus kernel: ? acls_after_inode_item+0x34/0x113
Aug 18 23:07:16 Zeus kernel: ? btrfs_get_64+0x72/0xdf
Aug 18 23:07:16 Zeus kernel: btrfs_iget_path+0x45a/0x57b
Aug 18 23:07:16 Zeus kernel: ? btrfs_lookup_dentry+0xfd/0x442
Aug 18 23:07:16 Zeus kernel: btrfs_lookup_dentry+0x16e/0x442
Aug 18 23:07:16 Zeus kernel: ? __d_lookup+0x8b/0x9d
Aug 18 23:07:16 Zeus kernel: btrfs_lookup+0xf/0x24
Aug 18 23:07:16 Zeus kernel: path_openat+0x515/0xa4d
Aug 18 23:07:16 Zeus kernel: do_filp_open+0x55/0xb8
Aug 18 23:07:16 Zeus kernel: ? btrfs_real_readdir+0x36d/0x39e
Aug 18 23:07:16 Zeus kernel: ? getname_flags+0x29/0x152
Aug 18 23:07:16 Zeus kernel: ? kmem_cache_alloc+0x122/0x14d
Aug 18 23:07:16 Zeus kernel: ? _raw_spin_unlock+0x14/0x29
Aug 18 23:07:16 Zeus kernel: do_sys_openat2+0x6c/0xd9
Aug 18 23:07:16 Zeus kernel: do_sys_open+0x3a/0x5a
Aug 18 23:07:16 Zeus kernel: do_syscall_64+0x6b/0x81
Aug 18 23:07:16 Zeus kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Aug 18 23:07:16 Zeus kernel: RIP: 0033:0x489d6a
Aug 18 23:07:16 Zeus kernel: Code: e8 1b ae fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 4c 8b 54 24 28 4c 8b 44 24 30 4c 8b 4c 24 38 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 40 ff ff ff ff 48 c7 44 24 48
Aug 18 23:07:16 Zeus kernel: RSP: 002b:000000c000d78ec8 EFLAGS: 00000216 ORIG_RAX: 0000000000000101
Aug 18 23:07:16 Zeus kernel: RAX: ffffffffffffffda RBX: 000000c000063800 RCX: 0000000000489d6a
Aug 18 23:07:16 Zeus kernel: RDX: 0000000000080000 RSI: 000000c001b672c0 RDI: ffffffffffffff9c
Aug 18 23:07:16 Zeus kernel: RBP: 000000c000d78f58 R08: 0000000000000000 R09: 0000000000000000
Aug 18 23:07:16 Zeus kernel: R10: 00000000000001b6 R11: 0000000000000216 R12: 000000c001b672c0
Aug 18 23:07:16 Zeus kernel: R13: 0000000000000001 R14: 000000c0016d9380 R15: ffffffffffffffff
Aug 18 23:07:16 Zeus kernel: </TASK>
Aug 18 23:07:16 Zeus kernel: Modules linked in: xt_mark tun macvlan veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) f71882fg iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 drm aesni_intel crypto_simd cryptd rapl intel_cstate mei_hdcp mei_pxp mxm_wmi intel_gtt i2c_i801 intel_uncore agpgart mei_me i2c_smbus i2c_core
Aug 18 23:07:16 Zeus kernel: syscopyarea sysfillrect sysimgblt firewire_ohci ahci alx firewire_core libahci mdio mei fb_sys_fops fan thermal video wmi backlight button unix
Aug 18 23:07:16 Zeus kernel: CR2: 0000000000000065
Aug 18 23:07:16 Zeus kernel: ---[ end trace 0000000000000000 ]---
Aug 18 23:07:16 Zeus kernel: RIP: 0010:endio_readpage_release_extent+0xb0/0xb7
Aug 18 23:07:16 Zeus kernel: Code: 89 6b 08 48 89 6b 10 44 88 63 18 48 8b 44 24 08 65 48 2b 04 25 28 00 00 00 74 05 e8 0c cb 51 00 48 83 c4 10 5b 5d 41 5c 41 5d <41> 5e c3 cc cc cc cc e8 de e2 ff ff f0 48 0f ba 28 00 0f 93 c0 0f
Aug 18 23:07:16 Zeus kernel: RSP: 0018:ffffc9000046bdb0 EFLAGS: 00010286
Aug 18 23:07:16 Zeus kernel: RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000000
Aug 18 23:07:16 Zeus kernel: RDX: 0000000001203400 RSI: ffffc9000046bcd8 RDI: 0000000000032190
Aug 18 23:07:16 Zeus kernel: RBP: ffffc9000046be90 R08: ffff8881615011f8 R09: ffffffff813b89df
Aug 18 23:07:16 Zeus kernel: R10: ffff88815a36f750 R11: fefefefefefefeff R12: 00000001ea9e3000
Aug 18 23:07:16 Zeus kernel: R13: 0000000500000000 R14: 0000000000000000 R15: ffff8881357e8ae8
Aug 18 23:07:16 Zeus kernel: FS:  0000145ffdff3b20(0000) GS:ffff88840f600000(0000) knlGS:0000000000000000
Aug 18 23:07:16 Zeus kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 18 23:07:16 Zeus kernel: CR2: 0000000000000065 CR3: 00000002ecfda003 CR4: 00000000001706f0
Aug 18 23:07:23 Zeus kernel: mce: CMCI storm detected: switching to poll mode
Aug 18 23:07:23 Zeus kernel: celery[10404]: segfault at 0 ip 0000146edc78617d sp 00007fff06ad8370 error 6 in libpython3.9.so.1.0[146edc661000+198000] likely on CPU 0 (core 0, socket 0)
Aug 18 23:07:23 Zeus kernel: Code: a8 00 00 00 25 00 02 00 00 c3 0f 1f 00 41 55 41 54 48 83 ec 28 4c 8b 25 41 7a 16 00 64 48 8b 04 25 28 00 00 00 48 89 44 24 18 <31> c0 4c 39 66 08 75 6b 80 7e 20 00 79 65 48 83 7e 10 64 7f 5e f6
Aug 18 23:08:55 Zeus sshd[4518]: fatal: Timeout before authentication for 10.10.10.167 port 53301

 

 

syslog.txt