Segfault and I don't know why

Naonak

By Naonak
in General Support

Recommended Posts

Naonak Noob

Naonak

Posted
Posted

I had a segfault the other day, and I can't figure out why. My system was stable for years, but I recently replaced the motherboard, ram, and CPU. It's been running fine for a few weeks, but then segfaulted out of the blue a couple days ago.  Syslog captured on remote syslog server seems to indicate it was a tainted user space application (related to php?), but I'm not that great with interpreting the segfault output. 

Can anyone who is better at deciphering it tell me what might be causing it? Everything on the new hardware is brand new, but obviously something could be wrong. 

System config is:

Motherboard: ASUS Z690+
CPU: Intel 12900KS

RAM: 64GB

  

Dec 24 15:00:01 NewMediaServer root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' '' '' '' '' '' '' ''
Dec 24 15:00:02 NewMediaServer root: Restoring original turbo write mode
Dec 24 15:00:02 NewMediaServer kernel: mdcmd (944): set md_write_method 1
Dec 24 15:00:02 NewMediaServer kernel:
Dec 24 15:05:03 NewMediaServer kernel: general protection fault, probably for non-canonical address 0xad30c494322a44ba: 0000 [#1] PREEMPT SMP NOPTI
Dec 24 15:05:03 NewMediaServer kernel: CPU: 10 PID: 23301 Comm: php Tainted: G        W  O      5.19.17-Unraid #2
Dec 24 15:05:03 NewMediaServer kernel: Hardware name: ASUS System Product Name/TUF GAMING Z690-PLUS WIFI D4, BIOS 2103 09/30/2022
Dec 24 15:05:03 NewMediaServer kernel: RIP: 0010:nf_nat_setup_info+0x142/0x7b1 [nf_nat]
Dec 24 15:05:03 NewMediaServer kernel: Code: 4c 89 f7 e8 2f f8 ff ff 48 8b 15 66 6a 00 00 89 c0 48 8d 04 c2 4c 8b 28 4d 85 ed 74 2a 49 81 ed 90 00 00 00 eb 21 8a 44 24 46 <41> 38 45 46 74
21 49 8b 95 90 00 00 00 48 85 d2 0f 84 53 ff ff ff
Dec 24 15:05:03 NewMediaServer kernel: RSP: 0018:ffffc900003d8730 EFLAGS: 00010286
Dec 24 15:05:03 NewMediaServer kernel: RAX: ffff888161fe1f11 RBX: ffff88892d20a200 RCX: 649c9a780a3d31bc
Dec 24 15:05:03 NewMediaServer kernel: RDX: ad30c494322a4504 RSI: 9c5013fa07f68b41 RDI: 25db6bbc99aa595c
Dec 24 15:05:03 NewMediaServer kernel: RBP: ffffc900003d87f8 R08: 4d2bc01a6505df09 R09: 631e26a4a71d4f78
Dec 24 15:05:03 NewMediaServer kernel: R10: a7bc318f1352538e R11: dcc38a19e8944726 R12: ffffc900003d880c
Dec 24 15:05:03 NewMediaServer kernel: R13: ad30c494322a4474 R14: ffffffff82909480 R15: 0000000000000000
Dec 24 15:05:03 NewMediaServer kernel: FS:  00001462d9d99b48(0000) GS:ffff88903f480000(0000) knlGS:0000000000000000
Dec 24 15:05:03 NewMediaServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 24 15:05:03 NewMediaServer kernel: CR2: 0000000042d70000 CR3: 000000087cdb6002 CR4: 0000000000770ee0
Dec 24 15:05:03 NewMediaServer kernel: PKRU: 55555554
Dec 24 15:05:03 NewMediaServer kernel: Call Trace:
Dec 24 15:05:03 NewMediaServer kernel: <IRQ>
Dec 24 15:05:03 NewMediaServer kernel: ? krealloc+0x7f/0x90
Dec 24 15:05:03 NewMediaServer kernel: nf_nat_masquerade_ipv4+0x114/0x13c [nf_nat]
Dec 24 15:05:03 NewMediaServer kernel: masquerade_tg+0x48/0x66 [xt_MASQUERADE]
Dec 24 15:05:03 NewMediaServer kernel: ipt_do_table+0x51b/0x5bf [ip_tables]
Dec 24 15:05:03 NewMediaServer kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Dec 24 15:05:03 NewMediaServer kernel: ? __local_bh_enable_ip+0x56/0x6b
Dec 24 15:05:03 NewMediaServer kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables]
Dec 24 15:05:03 NewMediaServer kernel: nf_nat_inet_fn+0x123/0x1a8 [nf_nat]
Dec 24 15:05:03 NewMediaServer kernel: nf_nat_ipv4_out+0x15/0x91 [nf_nat]
Dec 24 15:05:03 NewMediaServer kernel: nf_hook_slow+0x3a/0x96
Dec 24 15:05:03 NewMediaServer kernel: ? __ip_finish_output+0x144/0x144
Dec 24 15:05:03 NewMediaServer kernel: nf_hook+0xdf/0x110
Dec 24 15:05:03 NewMediaServer kernel: ? ethnl_parse_bit+0xce/0x202
Dec 24 15:05:03 NewMediaServer kernel: ? __ip_finish_output+0x144/0x144
Dec 24 15:05:03 NewMediaServer kernel: ip_output+0x78/0x88
Dec 24 15:05:03 NewMediaServer kernel: ? __ip_finish_output+0x144/0x144
Dec 24 15:05:03 NewMediaServer kernel: ip_sabotage_in+0x47/0x58 [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: nf_hook_slow+0x3a/0x96
Dec 24 15:05:03 NewMediaServer kernel: ? ip_rcv_finish_core.constprop.0+0x3b7/0x3b7
Dec 24 15:05:03 NewMediaServer kernel: NF_HOOK.constprop.0+0x79/0xd9
Dec 24 15:05:03 NewMediaServer kernel: ? ip_rcv_finish_core.constprop.0+0x3b7/0x3b7
Dec 24 15:05:03 NewMediaServer kernel: __netif_receive_skb_one_core+0x77/0x9c
Dec 24 15:05:03 NewMediaServer kernel: netif_receive_skb+0xbf/0x127
Dec 24 15:05:03 NewMediaServer kernel: br_handle_frame_finish+0x476/0x4b0 [bridge]
Dec 24 15:05:03 NewMediaServer kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec 24 15:05:03 NewMediaServer kernel: br_nf_hook_thresh+0xe2/0x109 [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec 24 15:05:03 NewMediaServer kernel: br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec 24 15:05:03 NewMediaServer kernel: ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: br_nf_pre_routing+0x226/0x23a [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Dec 24 15:05:03 NewMediaServer kernel: br_handle_frame+0x27c/0x2e7 [bridge]
Dec 24 15:05:03 NewMediaServer kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec 24 15:05:03 NewMediaServer kernel: __netif_receive_skb_core.constprop.0+0x4f6/0x6e3
Dec 24 15:05:03 NewMediaServer kernel: ? enqueue_entity+0x150/0x1ae
Dec 24 15:05:03 NewMediaServer kernel: ? update_overutilized_status+0x33/0x6e
Dec 24 15:05:03 NewMediaServer kernel: ? virt_to_slab+0x5/0x19
Dec 24 15:05:03 NewMediaServer kernel: __netif_receive_skb_one_core+0x40/0x9c
Dec 24 15:05:03 NewMediaServer kernel: process_backlog+0x8c/0x116
Dec 24 15:05:03 NewMediaServer kernel: __napi_poll.constprop.0+0x28/0x124
Dec 24 15:05:03 NewMediaServer kernel: RIP: 0010:nf_nat_setup_info+0x142/0x7b1 [nf_nat]
Dec 24 15:05:03 NewMediaServer kernel: Code: 4c 89 f7 e8 2f f8 ff ff 48 8b 15 66 6a 00 00 89 c0 48 8d 04 c2 4c 8b 28 4d 85 ed 74 2a 49 81 ed 90 00 00 00 eb 21 8a 44 24 46 <41> 38 45 46 74
21 49 8b 95 90 00 00 00 48 85 d2 0f 84 53 ff ff ff
Dec 24 15:05:03 NewMediaServer kernel: RSP: 0018:ffffc900003d8730 EFLAGS: 00010286
Dec 24 15:05:03 NewMediaServer kernel: RAX: ffff888161fe1f11 RBX: ffff88892d20a200 RCX: 649c9a780a3d31bc
Dec 24 15:05:03 NewMediaServer kernel: RDX: ad30c494322a4504 RSI: 9c5013fa07f68b41 RDI: 25db6bbc99aa595c
Dec 24 15:05:03 NewMediaServer kernel: RBP: ffffc900003d87f8 R08: 4d2bc01a6505df09 R09: 631e26a4a71d4f78
Dec 24 15:05:03 NewMediaServer kernel: R10: a7bc318f1352538e R11: dcc38a19e8944726 R12: ffffc900003d880c
Dec 24 15:05:03 NewMediaServer kernel: R13: ad30c494322a4474 R14: ffffffff82909480 R15: 0000000000000000
Dec 24 15:05:03 NewMediaServer kernel: FS:  00001462d9d99b48(0000) GS:ffff88903f480000(0000) knlGS:0000000000000000
Dec 24 15:05:03 NewMediaServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 24 15:05:03 NewMediaServer kernel: CR2: 0000000042d70000 CR3: 000000087cdb6002 CR4: 0000000000770ee0

 

Link to comment
Naonak Noob

Naonak

Posted
  • Author
Posted

The segfault is different this time... memtest passed the memory without any issues. I'm not sure where to look next.

  

Jan  2 19:50:31 NewMediaServer kernel: ------------[ cut here ]------------
Jan  2 19:50:31 NewMediaServer kernel: refcount_t: underflow; use-after-free.
Jan  2 19:50:31 NewMediaServer kernel: WARNING: CPU: 4 PID: 8713 at lib/refcount.c:28 refcount_warn_saturate+0xb3/0x100
Jan  2 19:50:31 NewMediaServer kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 macvlan ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tc
pudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs md_mod nct6775 nct6775_core hwmon_vid efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_
ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_
tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls wmi_bmof x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
aesni_intel crypto_simd cryptd rapl intel_cstate intel_uncore i915 i2c_i801 i2c_smbus igc iosf_mbi drm_buddy i2c_algo_bit ahci ttm libahci drm_display_helper drm_kms_helper cp210x usbserial hid_lg_g15
input_leds led_class
Jan  2 19:50:31 NewMediaServer kernel: drm mpt3sas nvme intel_gtt nvme_core agpgart i2c_core vmd raid_class scsi_transport_sas syscopyarea sysfillrect sysimgblt fb_sys_fops fan thermal wmi video backli
ght tpm_crb tpm_tis tpm_tis_core tpm acpi_tad acpi_pad button unix
Jan  2 19:50:31 NewMediaServer kernel: CPU: 4 PID: 8713 Comm: cloudflared Tainted: G        W  O      5.19.17-Unraid #2
Jan  2 19:50:31 NewMediaServer kernel: Hardware name: ASUS System Product Name/TUF GAMING Z690-PLUS WIFI D4, BIOS 2103 09/30/2022
Jan  2 19:50:31 NewMediaServer kernel: RIP: 0010:refcount_warn_saturate+0xb3/0x100
Jan  2 19:50:31 NewMediaServer kernel: Code: 00 01 e8 e6 c4 40 00 0f 0b c3 cc cc cc cc 80 3d 09 0c f7 00 00 75 5b 48 c7 c7 85 fd 0f 82 c6 05 f9 0b f7 00 01 e8 c3 c4 40 00 <0f> 0b c3 cc cc cc cc 80 3d e
5 0b f7 00 00 75 38 48 c7 c7 ad fd 0f
Jan  2 19:50:31 NewMediaServer kernel: RSP: 0018:ffffc900002d0f10 EFLAGS: 00010282
Jan  2 19:50:31 NewMediaServer kernel: RAX: 0000000000000000 RBX: ffff8883fde102e0 RCX: 0000000000000027
Jan  2 19:50:31 NewMediaServer kernel: RDX: 0000000000000102 RSI: ffffffff820d7be1 RDI: 00000000ffffffff
Jan  2 19:50:31 NewMediaServer kernel: RBP: ffff8883fde102e0 R08: 0000000000000000 R09: ffffffff828653f0
Jan  2 19:50:31 NewMediaServer kernel: R10: 0000000000000000 R11: ffffc900002d0ff8 R12: ffffc900002d0f48
Jan  2 19:50:31 NewMediaServer kernel: R13: 000000012f2fb835 R14: 000000000000012c R15: ffffc900002d0f58
Jan  2 19:50:31 NewMediaServer kernel: FS:  000000c0006ac090(0000) GS:ffff88903f300000(0000) knlGS:0000000000000000
Jan  2 19:50:31 NewMediaServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  2 19:50:31 NewMediaServer kernel: CR2: 000014ef9f376f98 CR3: 0000000117b40005 CR4: 0000000000770ee0
Jan  2 19:50:31 NewMediaServer kernel: PKRU: 55555554
Jan  2 19:50:31 NewMediaServer kernel: Call Trace:
Jan  2 19:50:31 NewMediaServer kernel: <IRQ>
Jan  2 19:50:31 NewMediaServer kernel: refcount_dec_and_test+0x1f/0x26
Jan  2 19:50:31 NewMediaServer kernel: napi_consume_skb+0x18/0x47
Jan  2 19:50:31 NewMediaServer kernel: net_rx_action+0x115/0x24f
Jan  2 19:50:31 NewMediaServer kernel: __do_softirq+0x126/0x288
Jan  2 19:50:31 NewMediaServer kernel: do_softirq+0x7f/0xab
Jan  2 19:50:31 NewMediaServer kernel: </IRQ>
Jan  2 19:50:31 NewMediaServer kernel: <TASK>
Jan  2 19:50:31 NewMediaServer kernel: __local_bh_enable_ip+0x4c/0x6b
Jan  2 19:50:31 NewMediaServer kernel: ip_finish_output2+0x37d/0x3b0
Jan  2 19:50:31 NewMediaServer kernel: ip_send_skb+0x15/0x3b
Jan  2 19:50:31 NewMediaServer kernel: udp_send_skb+0x278/0x2e6
Jan  2 19:50:31 NewMediaServer kernel: udp_sendmsg+0x72c/0x991
Jan  2 19:50:31 NewMediaServer kernel: ? ip_neigh_gw4+0x8b/0x8b
Jan  2 19:50:31 NewMediaServer kernel: ? udpv6_sendmsg+0x251/0xacc
Jan  2 19:50:31 NewMediaServer kernel: udpv6_sendmsg+0x251/0xacc
Jan  2 19:50:31 NewMediaServer kernel: ? __local_bh_enable_ip+0x56/0x6b
Jan  2 19:50:31 NewMediaServer kernel: ? __fpu_restore_sig+0x2e8/0x4d0
Jan  2 19:50:31 NewMediaServer kernel: ? sock_sendmsg_nosec+0x1c/0x40
Jan  2 19:50:31 NewMediaServer kernel: sock_sendmsg_nosec+0x1c/0x40
Jan  2 19:50:31 NewMediaServer kernel: __sys_sendto+0xc2/0x101
Jan  2 19:50:31 NewMediaServer kernel: ? __seccomp_filter+0x89/0x313
Jan  2 19:50:31 NewMediaServer kernel: __x64_sys_sendto+0x20/0x27
Jan  2 19:50:31 NewMediaServer kernel: do_syscall_64+0x68/0x81
Jan  2 19:50:31 NewMediaServer kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Jan  2 19:50:31 NewMediaServer kernel: RIP: 0033:0x40394e
Jan  2 19:50:31 NewMediaServer kernel: Code: 48 89 6c 24 38 48 8d 6c 24 38 e8 0d 00 00 00 48 8b 6c 24 38 48 83 c4 40 c3 cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f
7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
Jan  2 19:50:31 NewMediaServer kernel: RSP: 002b:000000c000a8cac8 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
Jan  2 19:50:31 NewMediaServer kernel: RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 000000000040394e
Jan  2 19:50:31 NewMediaServer kernel: RDX: 000000000000005b RSI: 000000c0009ba000 RDI: 0000000000000009
Jan  2 19:50:31 NewMediaServer kernel: RBP: 000000c000a8cb08 R08: 000000c000a8cde4 R09: 000000000000001c
Jan  2 19:50:31 NewMediaServer kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000001af9928
Jan  2 19:50:31 NewMediaServer kernel: R13: 0000000000000000 R14: 000000c0007024e0 R15: 000000c000a8cca8
Jan  2 19:50:31 NewMediaServer kernel: </TASK>
Jan  2 19:50:31 NewMediaServer kernel: ---[ end trace 0000000000000000 ]---
Jan  2 19:50:31 NewMediaServer kernel: stack segment: 0000 [#1] PREEMPT SMP NOPTI
Jan  2 19:50:31 NewMediaServer kernel: CPU: 4 PID: 8713 Comm: cloudflared Tainted: G        W  O      5.19.17-Unraid #2
Jan  2 19:50:31 NewMediaServer kernel: Hardware name: ASUS System Product Name/TUF GAMING Z690-PLUS WIFI D4, BIOS 2103 09/30/2022
Jan  2 19:50:31 NewMediaServer kernel: RIP: 0010:net_rx_action+0xff/0x24f
Jan  2 19:50:31 NewMediaServer kernel: RIP: 0010:net_rx_action+0xff/0x24f
Jan  2 19:50:31 NewMediaServer kernel: Code: 00 48 8b 3c 24 31 d2 48 89 c6 31 c0 48 8b ab d0 02 00 00 89 93 c4 02 00 00 48 89 83 d0 02 00 00 e8 ee 9b 16 00 48 85 ed 74 c2 <48> 8b 45 00 48 89 ef be 01 0
0 00 00 48 89 44 24 08 e8 dc c7 fe ff
Jan  2 19:50:31 NewMediaServer kernel: RSP: 0018:ffffc900002d0f30 EFLAGS: 00010282
Jan  2 19:50:31 NewMediaServer kernel: RAX: 0000000000000000 RBX: ffff88903f32d300 RCX: 0000000000000027
Jan  2 19:50:31 NewMediaServer kernel: RDX: 0000000000000102 RSI: ffffffff820d7be1 RDI: 00000000ffffffff
Jan  2 19:50:31 NewMediaServer kernel: RBP: a0507fd678ab54f2 R08: 0000000000000000 R09: ffffffff828653f0
Jan  2 19:50:31 NewMediaServer kernel: R10: 0000000000000000 R11: ffffc900002d0ff8 R12: ffffc900002d0f48
Jan  2 19:50:31 NewMediaServer kernel: R13: 000000012f2fb835 R14: 000000000000012c R15: ffffc900002d0f58
Jan  2 19:50:31 NewMediaServer kernel: FS:  000000c0006ac090(0000) GS:ffff88903f300000(0000) knlGS:0000000000000000
Jan  2 19:50:31 NewMediaServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  2 19:50:31 NewMediaServer kernel: CR2: 000014ef9f376f98 CR3: 0000000117b40005 CR4: 0000000000770ee0
Jan  2 19:50:31 NewMediaServer kernel: PKRU: 55555554
Jan  2 19:50:31 NewMediaServer kernel: Kernel panic - not syncing: Fatal exception in interrupt

 

Link to comment
MAM59 Collaborator

MAM59

Posted
Posted (edited)

sorry for you, but if it is not the ram, it maybe either the cpu or the mobo (chipset) itself. These Errors have nothing to do with software, their origin is surely in the hardware.

If I would be you, I would disable the onboard 2,5Gbe Lan card and plug in something real. Those 2,5G cards are usually lousy and the drivers are unstable.

Just try it, I'm sure you will find an old 1Gbe card in your shelfs or buy one for less than 10€...

(disable the onboard wifi too, it makes not much sense in a server)

 

Edited by MAM59
  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing