I've been troubleshooting some instability that has been leading to crashes about once per month.

I turned on syslog and have been capturing and investigation any errors that i can see.

The first I found that was ZFS corruption on a NVMe SSD that i dedicated to plex metadata. Since it was a single-disk ZFS pool that did not have parity, I wiped it and changed back to XFS. I only used ZFS for snapshots and replication, but with all of the memory issues i kept seeing, i decided to just go XFS for this disk and will just let plex rebuild in the future if needed.

I also had a bunch of Syslinux Configuration paramters set when i was passing hardware through to a VM a long time ago. I removed all of those as i read it could cause the system to not enforce memory mgmt or something along those lines.

the one fault i continue to see (just about daily) is a "smartctl[3052408]: segfault at 5583dc000000 ip 0000150a32cb7f45 sp 00007ffd2a3533e0 error 4 in libc.so.6[98f45,150a32c45000+155000] likely on CPU 0 (core 0, socket 0)"Z

Each time ive seen it, ive tried to run the errors though either Grok or Google AI to see if it could point me in the right direction.

Im at 14 days of uptime right now, so i am hoping that the previous issues that i found will let me get past the 30days - at 30 days, my UI typically gets hung.

i also did run mem test for 4 passes and received zero errors there.

Diagnostics zip and syslog attached.

mclovinii-diagnostics-20250819-1137.zip syslog.latest.txt

Here is the Grok analysis of the segfault error i saw this morning.

Analysis

1. Segfault Details:

   • First Incident (03:25:28):

     • Process: smartctl[3047366]

     • Segfault at address 1f1, instruction pointer (IP) 00001544b9f0720a, stack pointer (SP) 00007ffe0fe57f10.

     • Error code 6 (invalid permissions for mapped object, likely trying to write to a read-only memory region).

     • Occurred in libc-2.41.so, suggesting a potential issue in the C library or how smartctl interacts with it.

     • Likely on CPU 0 (core 0, socket 0).

   • Second Incident (03:27:30):

     • Process: smartctl[3052408]

     • Segfault at address 5583dc000000, IP 0000150a32cb7f45, SP 00007ffd2a3533e0.

     • Error code 4 (invalid memory reference, likely accessing an invalid or unmapped address).

     • Occurred in libc.so.6, again pointing to a C library interaction.

     • Also on CPU 0.

2. Commonalities:

   • Both errors involve smartctl and libc, suggesting a possible issue with smartctl’s memory handling or a corrupted library.

   • The different error codes (6 and 4) indicate varying memory access issues, but both are likely related to the same root cause.

   • The timing (close together) suggests a recurring issue, possibly triggered by a scheduled task (e.g., a cron job running smartctl to check disk health).

3. Possible Causes:

   • Bug in smartctl: A known issue in the version of smartmontools installed, especially if it’s outdated or incompatible with the system’s libc.

   • Corrupted libc: The C library itself might be corrupted or mismatched with the system.

   • Hardware Issues: Faulty RAM or disk could cause memory corruption, leading to segfaults.

   • Driver or Firmware Issues: Problems with the storage controller or disk firmware could cause smartctl to crash when querying drives.

   • System Overload: If the system is under heavy load or low on memory, it might cause unexpected behavior.

Steps to Resolve

1. Check smartctl Version:

   • Run smartctl --version to check the installed version.

   • Compare it with the latest version from the smartmontools website or your package manager. Update if outdated:

     bash

     sudo apt update && sudo apt install smartmontools  # For Debian/Ubuntu
     sudo yum install smartmontools  # For CentOS/RHEL

   • Check for known bugs in your version on the smartmontools bug tracker or forums.

2. Verify libc Integrity:

   • Check the integrity of libc:

     bash

     sudo ldconfig -v 2>/dev/null | grep libc.so

   • Reinstall libc if suspected to be corrupted:

     bash

     sudo apt install --reinstall libc6  # For Debian/Ubuntu
     sudo yum reinstall glibc  # For CentOS/RHEL

3. Inspect System Logs:

   • Look for additional errors around the same time:

     bash

     sudo journalctl -k -t kernel -S "2025-08-19 03:25:00" -U "2025-08-19 03:28:00"

   • Check for disk or controller errors that might be triggering smartctl crashes.

4. Test Hardware:

   • Run a memory test to rule out RAM issues:

     bash

     sudo memtest86+

   • Check disk health manually with smartctl (run cautiously, as it’s crashing):

     bash

     sudo smartctl -a /dev/sdX  # Replace sdX with your disk (e.g., sda)

   • If it crashes, try running with debug output:

     bash

     sudo smartctl -d sat -a /dev/sdX

5. Check Scheduled Tasks:

   • Verify if smartctl is run by a cron job:

     bash

     sudo crontab -l
     cat /etc/cron.*/*

   • Temporarily disable any smartctl-related jobs to prevent crashes while troubleshooting.

6. Update System:

   • Ensure the kernel and all packages are up to date:

     bash

     sudo apt full-upgrade  # For Debian/Ubuntu
     sudo yum update  # For CentOS/RHEL

7. Debug with gdb:

   • If the issue persists, run smartctl under gdb to get a backtrace:

     bash

     sudo gdb --args smartctl -a /dev/sdX
     (gdb) run
     (gdb) bt  # After crash, to get backtrace

   • Share the backtrace with the smartmontools community or on a relevant forum.

8. Check for Known Issues:

   • Search for similar issues on X or the web:

     • Keywords: smartctl segfault libc 2025, smartmontools crash libc.so.6.

     • Check smartmontools mailing lists or GitHub issues.

   • If needed, I can perform a real-time search for recent reports (let me know).

Next Steps

• Start by checking the smartctl version and updating it.

• Verify libc integrity and system updates.

• If the issue persists, test hardware and inspect logs for clues.

• Report the issue to the smartmontools community with details (version, system info, logs).

I've seen this happening with some NVMe devices; try a different one if possible.

Thank you! If you have seen this before with NVMe disks, I know exactly which one it is. I'm going to order one now and will respond here if it fixes. Appreciate the response @JorgeB

3 hours ago, JorgeB said:

I've seen this happening with some NVMe devices; try a different one if possible.

Changed the Kioxia NVMe SSD to a Samsung 990 pro last night. First night I have not seen this default error pop up. And wow do these NVE disks copy over fast. 280GB moved over in a couple minutes!

Once I see a few days of stability, i will close this out as solved. Thanks again @JorgeB for saving my sanity here. I have been trouble shooting this issue for months.

On 8/19/2025 at 1:01 PM, JorgeB said:

I've seen this happening with some NVMe devices; try a different one if possible.

This was definitely my issues. However, it was not the Kioxia drive that I suspected.

I found the culprit was a Lycom DT-120 M.2 Interface Card and Adapter that i used to connect a Samsung 980 m.2 to the motherboard. I would see random high temp warnings for this disk, but it would never show high on the main page.

As soon as i removed this pcie to m.2 card, i have not see the error since.

I have that same m.2 and Lycom card in my test server now to see if I can reproduce the issue.

Marking this as solved as it was definitely NVMe disk related.

Hopefully this can help someone else in their troubleshooting. If i do see more issues on the test server, i will add those notes here.