Update after the last round of hardware tests. Short version: replacing both CPUs didn't change anything.

WHAT HAPPENED SINCE MY LAST UPDATE

1) Sept 1 (still original CPUs, after BIOS defaults + PSU2)

After the ~25 h clean window the box got progressively worse. On top of the shfs crash I already reported, at 16:25 there was a kernel Oops inside the SPL/ZFS module:

BUG: kernel NULL pointer dereference, address: 0000000000000008

RIP: 0010:spl_cache_flush+0x5e/0x160 [spl]

note: z_wr_int_2[9680] exited with irqs disabled

After that the webGUI went down: php-fpm workers crash-looped (tens of thousands of SIGABRT exits), and from 17:00 several php-fpm workers segfaulted at the exact same address on different cores and both sockets:

php-fpm[...]: segfault at 6e696159 ip 0000000000a35c80 ... error 4 in php-fpm[635c80,600000+49b000]

For the record, this did not survive a reboot: the reboot I issued at 17:09 never completed, the same session kept running until I powered it off at 17:23, and there's no new boot in the logs.

2) Sept 8-10: swapped the two CPUs between sockets (RAM untouched)

Over about 2 days: 18 userspace segfaults, 8 on socket 0 and 10 on socket 1. Before the swap it was 9 of 11 on socket 0 (Sep 1 and Sep 5-6 windows). Also:

Sep 9 18:07:06 BUG: Bad rss-counter state mm:... type:MM_FILEPAGES val:45 Comm:selkies

BUG: non-zero pgtables_bytes on freeing mm: 8192

(dbus-daemon segfault in the same second)

Sep 10 03:28:15 BUG: kernel NULL pointer dereference, address: 0000000000000360

RIP: 0010:fuse_dentry_revalidate+0xb7/0x2f0 (CPU 6, Comm: libuv-worker, openat path)

3) Sept 22: installed two replacement E5-2450 v2 (same model)

First boot only showed 80 GB (DIMM_B1 and B4 not detected). I reseated CPU2 and cleaned the DIMM contacts, and after that all 12 DIMMs / 96 GB were detected. It then ran clean for about 36.5 h, the longest clean stretch so far, and then the same things came back:

Sep 24 01:46:34 smartctl_type: segfault at 0 ip 0000000000a66303 error 4 in php[666303,600000+48d000] (socket 0)

Sep 24 03:00:05 shfs[10594]: segfault at 10 ip 000014ecbb55634c error 4 in libfuse3.so.3.16.2[b34c,...] (socket 0)

Sep 24 03:00:21 traps: postgres[55755] general protection fault ... error:0 in libc.so.6[2650f,...]

Sep 24 03:00:26 traps: postgres[56467] general protection fault ... error:0 in libc.so.6[2650f,...]

The shfs crash has the same offset as the Sept 1 one (libfuse3 +0xb34c; Sept 1 was on socket 1 with the original CPUs, this one on socket 0 with the new ones). It took /mnt/user down again (Transport endpoint is not connected), both Immich Postgres containers died 16 s later, and most containers were erroring until I remounted /mnt/user and restarted them. Over the full 46 h on the new CPUs, those four events (all between 01:46 and 03:00 on Sep 24) were the only ones, fewer than on the old CPUs but including the most damaging one.

Also during this boot:

- A file inside one container (Carol-immich) had its first 416 bytes zeroed on disk: a 687-byte JS file, the rest intact, cut mid-word. The same file in a second container built from the same image was fine, and recreating the container fixed it. Details attached.

- winbindd started aborting every 15 minutes from Sep 23 23:36 (about 34 h after boot), always in a forked child: messaging_reinit -> reinit_after_fork -> talloc destructor -> abort(). The parent winbindd has been running since boot.

Throughout all of this: EDAC 0 CE / 0 UE on both memory controllers, no MCE, SMART OK on all array disks, CPU temps 33-48 C.

OFFSETS THAT KEEP REPEATING ACROSS THE ORIGINAL, SWAPPED AND NEW CPUS

- php +0x666303 (smartctl_type): since July, again on the new CPUs

- libfuse3.so.3.16.2 +0xb34c (shfs): Sep 1 and Sep 24

- ld-linux-x86-64-2.43.so +0xb34c (smartctl): Sep 1, Sep 9, Sep 10

- libc-2.43.so +0xa7ddd (smartctl): August and Sep 9

- libpython3.14.so +0x1bed15 (celeryd): Sep 9 and Sep 10

So: different CPU silicon, same crash sites. I think that rules out the two original CPUs as individual bad units, but I'm not sure what it points to instead.

NEW: WHAT THE OLD DIAGNOSTICS ON THE FLASH SHOW

I went through the 30 diagnostics zips saved in /boot/logs (2024 to now) and compared lspci and the syslog in each:

2024-12-01 6.12.13 (6.1.106) no NVMe 0 segfaults

2026-04-02 7.0.0 (6.6.68) NVMe = MAXIO MAP1202 [1e4b:1202] 0 segfaults, syslog covers Feb 18 - Apr 2

2026-07-09 7.3.0 (6.18.x) NVMe = FORESEE E2M2 [1217:8760] 12,227 segfaults, 125 Oopses (syslog from Jul 7)

(The 91 segfaults in a Jan 2024 zip are all Plex "PMS LoudnessCmd" in libswresample, a known Plex bug, unrelated.)

Between the last clean snapshot and the first bad one, the only differences in lspci are the NVMe on the PCIe adapter at 0c:00.0 (MAXIO swapped for a FORESEE E2M2, O2 Micro controller, together with a different M.2-to-PCIe adapter) plus the Unraid upgrade 7.0.0 -> 7.3.0. The flash backup git history dates both:

- 2026-05-15: config rewrite matching the 7.3.0 upgrade (7.3.0 was released May 12)

- 2026-05-29: pool "nvme" renamed to "emmc" and the E2M2 appears for the first time, i.e. the FORESEE went in

- There were no scheduled parity checks before Aug 1, yet one ran on the morning of May 29, starting ~07:04 right at a boot/array start, finishing 12:54. The FORESEE swap was committed at 14:41 that afternoon. If that check was the automatic one after an unclean shutdown, the box was already misbehaving on 7.3.0 with the old MAXIO, before the FORESEE. I can't confirm that from the system (notifications live in RAM), and I honestly don't remember whether I started that check by hand.

- I swapped the MAXIO out because I wanted it for another machine, not because it was failing. It's been working fine there since.

So the two changes are two weeks apart, and the diagnostics can't tell me which one lines up with the start. The 7.2.8 test doesn't separate them either, because the FORESEE was installed during that test too.

Some things about that FORESEE:

- All its SMART counters read 0 (power-on hours, power cycles, data written).

- The kernel logs "missing or invalid SUBNQN field" and "nvme nvme0: using unchecked data buffer". As I understand it, the second one means passthrough commands (which smartctl uses) go to a device without SGL, so the kernel can't bound how much the device DMAs into the buffer.

- IOMMU is enabled but the default domain is Passthrough, so a device writing where it shouldn't wouldn't be caught.

- It sits on the C600 PCH root port 00:1c.0, x1 Gen2.

- Since Sep 6 it hasn't had a pool on it and it's mostly idle, but it's still on the bus and still gets SMART polls.

I'm not claiming that's the cause. The "unchecked data buffer" warning shows up on lots of consumer NVMe drives without problems, and the crashes didn't slow down once the drive went idle. But it's the one hardware change I can line up with when this started, and it's free to test.

(Side note: I also tried to boot Ubuntu from that same card to test a different kernel, but the T420 can't boot from the adapter, so that test never happened.)

WHAT I'M DOING NEXT

Removing the FORESEE (and the PCIe adapter) now and staying on 7.3.2, so only one thing changes. The longest clean stretch so far was about 36.5 h, so I'll let it run at least a week before drawing conclusions.

If it keeps crashing without it, the other half of that window is the 7.0.0 -> 7.3.0 upgrade. 7.0.0 (kernel 6.6.68) is the last version I have evidence of running clean on this hardware (6 weeks), and 7.2.8 (6.12.98) already reproduced it, so a regression would have to be somewhere between 6.6 and 6.12. I could boot 7.0.0 from a spare stick with the array stopped (the ZFS pools were feature-upgraded on Jul 28, so 7.0.0 probably can't import them). I know you said to stop comparing versions, but this one would be the last known-good. Other candidates: removing the Quadro P2200 and the Nvidia driver plugin (so the kernel isn't tainted), running on half the RAM and then the other half, and moving the HBA to another slot. Beyond that it's a replacement motherboard or new hardware.

Does that order make sense to you, or is there something you'd check first?

Attached:

- alpha-diagnostics-20260924-1136.zip: current boot (new CPUs, the Sep 24 shfs crash, the winbindd loop)

- alpha-diagnostics-20260402-1312.zip: the last clean snapshot (7.0.0, MAXIO NVMe)

- alpha-diagnostics-20260709-1910.zip: the first bad snapshot (7.3.0, FORESEE)

- alpha-syslog-excerpts-2026-09-24.zip: kernel lines from my remote syslog for Sep 1, 5, 6, 8, 9, 10, 22 and 24, one winbindd panic with full backtrace, and the zeroed-file hexdump

alpha-diagnostics-20260924-1136.zipalpha-syslog-excerpts-2026-09-24.zipalpha-diagnostics-20260402-1312.zipalpha-diagnostics-20260709-1910.zip

Your order is sound. I have one change and some supporting evidence.

What the diagnostics confirm

I compared the PCI inventory in your three archives directly. Between alpha-diagnostics-20260402-1312 (clean, 7.0.0) and alpha-diagnostics-20260709-1910 (first bad, 7.3.0), the only hardware difference is the NVMe device:

-0c:00.0 MAXIO Technology MAP1202 (DRAM-less) [1e4b:1202]

+0c:00.0 O2 Micro, Inc. FORESEE E2M2 NVMe SSD [1217:8760]

Everything else is identical, including BIOS 2.9.0, the HBA, both NICs and the Quadro. Your reading of the flash history is correct.

The FORESEE is also the one component that no test has removed. It was installed during the 7.2.8 comparison, both fresh-media tests, the Safe Mode runs, the BIOS reset, the PSU2 swap and both CPU changes. Every failing configuration contained it. The last configuration with no faults did not.

The drive's own data is not valid

Its SMART health log reports all zeros. Data Units Read, Data Units Written, Power Cycles and Power On Hours are each 0, although the drive held a pool. Its EUI is 01000000000000000000000000000000. The kernel reports missing or invalid SUBNQN field and allocates 1/0/0 default/read/poll queues. This firmware does not implement the NVMe identify and log pages correctly.

The kernel also reports using unchecked data buffer for this device. That message appears when a passthrough command carries a data buffer whose length the kernel cannot check against the transfer the device performs. smartctl issues those commands. smartctl_type is the process that faults most often in your whole investigation.

Two corrections

First, the July 9 archive is not evidence of broad corruption starting there. Its syslog holds 3,698 segfaults, but only two processes produce them: 3,205 in xclip/libuuid.so.1.3.0 and 493 in celery/libcrypto.so.3. Its 62 oopses are all containerd-shim at filp_flush+0x37. The oops counter already reads [#3847] at the start of that log, so most of that boot's damage happened before the window you kept. It marks a boot that had already failed, not the first failure.

Second, your time to first fault on the new CPUs is about 34.5 hours, not 36.5. winbindd began aborting at 23:36:49 on September 23, about two hours before the smartctl_type fault. Use winbindd as the marker in later tests.

The signature is zeros, not random bits

Across every layer, the corruption reads back as zero:

- filp_flush+0x37: RAX is 0, so file->f_op was zero, then CR2 is 0x70.

- smartctl_type at php +0x666303: mov rax,[rdi] returns 0, then the fault address is 0.

- The xclip faults: the instruction bytes at the fault dump as all 00. The mapped library page was zero.

- Your call-bound/index.js: the first 416 bytes are NUL and the rest is intact.

Random DRAM cell errors flip bits. They do not write runs of zeros into pointers, code pages and file contents. Memtest86+ searches for bit errors under its own patterns, so a fault of this shape can pass 25 hours and five passes without a single error. Your clean Memtest and Dell results are consistent with this. They did not exclude it.

Why your ECC counters cannot settle this

When a device writes to memory over DMA, the memory controller computes correct ECC for the bytes the device supplied. The data is wrong, but the ECC is valid. EDAC and MCE cannot report it.

Unraid also runs the IOMMU in pass-through mode by default, which your diagnostics confirm:

iommu: Default domain type: Passthrough

In this mode device DMA is neither translated nor bounds-checked, so a misbehaving device can write anywhere in RAM and nothing is logged. Your April 2 clean boot has the same setting, so this is not the cause. It does mean that a DMA-driven fault produces no record at all on this platform.

But do not switch the IOMMU to DMA-translation mode to try to make such a fault visible. Unraid changed that default to pass-through in 6.10.3 specifically to fix data corruption on some platforms with Intel VT-d enabled, and your T420 uses the tg3 driver that appeared in those early reports. On a host that is already corrupting data, that change carries real risk and I do not recommend it.

Since we cannot instrument the DMA path safely, removing the suspect device is the correct test.

Recommended order

1. Remove the FORESEE and its adapter, as you planned, and stay on 7.3.2. Run at least a week. This is the strongest single-variable test you have left, and your reasoning for doing it first is right.

2. If faults continue without the drive, remove the Quadro P2200 and the NVIDIA plugin next. This also clears the kernel taint.

3. Move the HBA to another slot after that.

4. Test reduced DIMM populations last, following the T420 population rules.

I suggest you drop the 7.0.0 test rather than keep it in the list. 7.2.8 already reproduced the faults with the FORESEE installed, so a 7.0.0 run with the drive still present is confounded the same way. After the drive is out, a clean 7.3.2 result answers the question and a 7.0.0 comparison adds nothing.

The winbindd abort loop is a talloc destructor failure in messaging_reinit after fork. I read it as a downstream effect of the damaged state, not a separate defect. It is still useful as your earliest marker.

Quick update, 2+ days after pulling the FORESEE eMMC

The FORESEE E2M2 and its NVMe adapter came out on Sep 24 around 12:20. Everything else is unchanged (7.3.2, same CPUs, RAM, HBA, Quadro, IOMMU left alone).

Status as of Sep 26 15:20: up 2 days 3 hours with nothing at all.

• 0 winbindd aborts, which I'm using as the earliest marker as you suggested.

• 0 segfaults, GPF, oops, BUG or hung tasks.

• EDAC and MCE clean, /mnt/user fine.

• All array disks and pools healthy, SMART clean.

For comparison, the best run on the new CPUs was about 34.5 h to the first fault (winbindd). The server hasn't been idle either:

• normal container use (about 30 containers)

• Plex intro detection pegging all cores

• Paperless re-processing 51 documents

• full reads of every database

• a parallel read of every file on all user shares (still running)

While checking everything, I found that the zeroing had also persisted inside application databases during the bad period. Details are in the attached txt; summary:

1. Immich, two separate PostgreSQL 16 containers. pg_dump failed on corrupt tuples:

   • One asset row had its tail zeroed: enum columns "invalid internal value for enum: 0", a bad varlena length, and timestamps at the PG zero date.

   • One smart_search row had a missing TOAST chunk.

   • In the other instance, one asset_face row was fully zeroed (all-zero UUID).

   • amcheck heapallindexed showed the btree indexes still holding the ORIGINAL values for the zeroed row. The heap page was changed underneath PostgreSQL, not through SQL.

   • The photo file itself was intact (SHA1 matches).

2. Plex (SQLite). integrity_check reported "row missing from index" x5 for one metadata_items row:

   • The row has "index", air date, added_at, created_at and updated_at all = 0, while the neighbouring rows are normal.

   • A query through the indexes can't find it; a NOT INDEXED full scan does. Same pattern: the indexes kept the originals.

   • Plex's own 3-day DB backups stopped after Jul 27.

3. Nextcloud (MariaDB/InnoDB): 600x "Index for table 'oc_filecache' is corrupt" in nextcloud.log. Only on Jul 11-14, Aug 9-13, Aug 17-22 and Sep 1, i.e. the episode windows. The table checks clean today, so that one looks like it stayed in memory.

4. Code files with NUL bytes inside containers:

   • a Carol-immich JS file (fixed earlier by recreating the container)

   • Paperless numpy .py files on Aug 2-3 ("source code string cannot contain null bytes")

   • plus 122 Paperless OCR worker SIGSEGVs between Jul 26 and Aug 18.

Everything above has been repaired or re-imported from intact originals. Everything else checks clean: Sonarr/Radarr/Vaultwarden/Calibre/Syncthing/NPM SQLite, the Linkwarden PG17 amcheck, CouchDB full reads, and Paperless checksums. None of it is new: every damaged item dates from the episode period, and nothing has appeared since the eMMC came out. It fits the theory of zero runs landing in page cache / buffer memory and later being written back to disk.

Plan: keep it running untouched until the one-week mark (Oct 1). Then I'll post a final update with fresh diagnostics and the result of the zero-file scan across all shares. If anything shows up before then, I'll follow your order: Quadro + NVIDIA plugin first, then the HBA slot, then reduced DIMM sets.

Is there anything else you'd like me to capture at the one-week mark besides the diagnostics?

This is encouraging. You have now exceeded the previous 34.5-hour window while continuing normal workloads. Keep the configuration unchanged through October 1.

One correction to my earlier reply: the zero-filled data does not prove DMA corruption. Removing the FORESEE and its adapter is our strongest lead so far, but this test does not distinguish the drive, adapter, or their interaction with the system.

At the one-week mark, please capture:

- Fresh diagnostics before rebooting, plus the uptime and any configuration changes or restarts during the test.

- The external syslog covering the whole interval since removal, including rotated logs.

- A dated summary of the database checks after repair, and confirmation that backups are completing again, especially Plex.

- The completed share-scan results and the method used.

For that scan, zero bytes alone do not establish corruption. Some files legitimately contain them, and a scan for completely zeroed files can miss partial damage. Compare suspect files with known-good originals or checksums where available.

Keep older known-good backups alongside the new ones. Finding another damaged file now would not necessarily mean corruption continued after removal; it could be older damage discovered later.

If the week remains clean, leave the drive and adapter out and continue monitoring. There is no need to reinstall them against live data just to prove the connection.

One-week update: clean week without the FORESEE

The server has now been up for 7 days without a reboot: booted 2026-09-24 12:20 after removing the FORESEE E2M2 and its adapter; diagnostics taken at the 7-day mark (2026-10-01 12:21, before any reboot). Nothing was changed at system level during the week: same hardware, 7.3.2, no plugin, OS, share or array changes, IOMMU untouched. Everything else was application-level and is listed with dates in changes-and-restarts-during-test.txt.

Results for the whole window:

• Kernel: 0 winbindd aborts, 0 segfaults, GPF, Oops, BUG or hung tasks; no MCE; EDAC CE/UE 0 on both memory controllers; no I/O errors; /mnt/user fine.

• External syslog (Ubuntu rsyslog, rotated files included): zero for every one of those patterns on every day from the boot to now.

  • The Sep 24 file is split at the boot. Before 12:20 it has the known faults (336 winbindd panics, segfaults, GPF, Oops, BUG, shfs); after 12:20 it has none.

  • The only post-boot "EDAC" lines are sb_edac driver initialisation.

• Array, pools and disks:

  • Scheduled parity check on Sep 27: 0 errors.

  • All disks DISK_OK with 0 errors; SMART clean on all 15 drives (full smartctl -x attached).

  • ZFS pools: no errors.

Database checks, re-run today (dated results in database_checks.txt):

• PostgreSQL (Immich x2 and Linkwarden): amcheck verify_heapam with TOAST, bt_index_check heapallindexed on every btree, a full row scan and a hidden-zero scan. All clean. The only all-zero UUID left is the audit row created by our own deletion of the zeroed face row.

• Plex (Plex SQLite integrity_check, library and blobs): ok.

• 29 other SQLite DBs (Vaultwarden, Paperless, Sonarr/Radarr, Maintainerr, Calibre, NPM, Syncthing…): ok.

• MariaDB CHECK TABLE: 179/179 OK. Nextcloud core integrity: OK.

• CouchDB: full read of every document, 0 errors.

• Paperless sanity checker: no errors.

• Immich nightly integrity checks: 0 open reports.

Backups:

• Immich pg_dump completes every night on both instances (Sep 26 through Oct 1).

• Plex's own DB backup is running again: Sep 28 and Oct 1. There were none between Jul 27 and Sep 28, i.e. while the DB was corrupt.

Share scan, method and results (share-scan-method-and-results.txt):

• Zero-headed-file scan of all shares (~470k files): only expected formats (ISO system areas, NDS saves, 1-byte Wii U placeholders) and a few 2021 torrent leftovers.

• Following your point that zero bytes alone prove nothing and can miss partial damage, I compared against known-good checksums where they exist. Immich: SHA1 of EVERY original file versus the checksum Immich recorded at upload, on both instances: 81,549 files, 320.5 GB, 0 mismatches, 0 missing. Paperless's checksum verification is also clean.

• For the media share, the only checksums available would be the torrent piece hashes. I can force-recheck the seeding torrents if you think it's worth it.

I take your correction: the zero-filled data is a lead, not proof of DMA, and this test doesn't separate the drive from the adapter or from how they interact with the system.

So the plan is: the FORESEE and adapter stay out, I keep monitoring, and I keep the older known-good backups alongside the new ones. If anything comes back, I'll follow your order (Quadro + NVIDIA plugin, then HBA slot, then DIMM sets).

One anecdotal observation: the drive-bay activity LEDs behave normally now, and they didn't seem to during the bad period. I haven't verified whether it's related.

Thanks a lot for the guidance through all of this.

Is there anything else you'd like me to check? Otherwise I'll keep monitoring and only post again if anything shows up.

Attachment: alpha-one-week-test-2026-10-01.zip (diagnostics, external syslog + per-day summary, health snapshot, list of changes/restarts, database checks, Immich checksum verification, share-scan method/results/scripts, smartctl -x per disk)

alpha-one-week-test-2026-10-01.zip

Thanks for the thorough update. I checked the bundle, and it matches what you describe. The diagnostics syslog and the external syslog both cover the full 7 days from the September 24 boot. After that boot, neither log has a single segfault, GPF, oops, BUG, hung task, MCE, I/O error or winbindd abort. Before the reboot, the same external log still shows the old faults. EDAC is 0 on both memory controllers, no NVMe device is present, and both parity checks (September 24 and September 27) finished with 0 errors.

Your previous best on the new CPUs was about 34.5 hours to the first fault. This is about five times longer, under real load. I consider the FORESEE and its adapter to be the cause. We still do not know if it was the drive, the adapter or how they worked together, but you do not need to know that to fix it. Keep both out permanently, and do not reuse them in any system that holds data.

To answer your question, here are a few optional clean-up steps for damage that may still be hidden from the bad period:

1. Docker image. You already found NUL bytes inside container code files, such as the Immich JS file and the Paperless numpy files. Those files live in image layers inside docker.img, which was written while the server was faulting. Recreate the Docker image and reinstall your containers from Apps > Previous Apps. Everything is downloaded again, and your appdata is not affected. This removes any damaged image files that you have not found yet.

2. Torrent recheck. Yes, it is worth doing for the torrents you still seed. A force-recheck is a free checksum pass over that media, and any failed pieces show you exactly which files are damaged.

3. Backups. Treat backups taken between early July and September 24 as suspect, because they may have copied damaged data. Keep your pre-July backups, and restore from them first if you ever need older data.

About the drive-bay LEDs: I cannot confirm a connection from the logs, so I would treat that as anecdotal.

Otherwise, keep monitoring as you planned, and keep the external syslog running for a while longer. If anything returns, save the logs before you reboot, then follow the same order: Quadro and NVIDIA plugin first, then the HBA slot, then the DIMMs. If nothing returns, please mark the thread as solved. Thanks for the careful testing. It made this much easier to narrow down.