Thiago Alexander

One-week update: clean week without the FORESEE The server has now been up for 7 days without a reboot: booted 2026-09-24 12:20 after removing the FORESEE E2M2 and its adapter; diagnostics taken at the 7-day mark (2026-10-01 12:21, before any reboot). Nothing was changed at system level during the week: same hardware, 7.3.2, no plugin, OS, share or array changes, IOMMU untouched. Everything else was application-level and is listed with dates in changes-and-restarts-during-test.txt. Results for the whole window: Kernel: 0 winbindd aborts, 0 segfaults, GPF, Oops, BUG or hung tasks; no MCE; EDAC CE/UE 0 on both memory controllers; no I/O errors; /mnt/user fine. External syslog (Ubuntu rsyslog, rotated files included): zero for every one of those patterns on every day from the boot to now. The Sep 24 file is split at the boot. Before 12:20 it has the known faults (336 winbindd panics, segfaults, GPF, Oops, BUG, shfs); after 12:20 it has none. The only post-boot "EDAC" lines are sb_edac driver initialisation. Array, pools and disks: Scheduled parity check on Sep 27: 0 errors. All disks DISK_OK with 0 errors; SMART clean on all 15 drives (full smartctl -x attached). ZFS pools: no errors. Database checks, re-run today (dated results in database_checks.txt): PostgreSQL (Immich x2 and Linkwarden): amcheck verify_heapam with TOAST, bt_index_check heapallindexed on every btree, a full row scan and a hidden-zero scan. All clean. The only all-zero UUID left is the audit row created by our own deletion of the zeroed face row. Plex (Plex SQLite integrity_check, library and blobs): ok. 29 other SQLite DBs (Vaultwarden, Paperless, Sonarr/Radarr, Maintainerr, Calibre, NPM, Syncthing…): ok. MariaDB CHECK TABLE: 179/179 OK. Nextcloud core integrity: OK. CouchDB: full read of every document, 0 errors. Paperless sanity checker: no errors. Immich nightly integrity checks: 0 open reports. Backups: Immich pg_dump completes every night on both instances (Sep 26 through Oct 1). Plex's own DB backup is running again: Sep 28 and Oct 1. There were none between Jul 27 and Sep 28, i.e. while the DB was corrupt. Share scan, method and results (share-scan-method-and-results.txt): Zero-headed-file scan of all shares (~470k files): only expected formats (ISO system areas, NDS saves, 1-byte Wii U placeholders) and a few 2021 torrent leftovers. Following your point that zero bytes alone prove nothing and can miss partial damage, I compared against known-good checksums where they exist. Immich: SHA1 of EVERY original file versus the checksum Immich recorded at upload, on both instances: 81,549 files, 320.5 GB, 0 mismatches, 0 missing. Paperless's checksum verification is also clean. For the media share, the only checksums available would be the torrent piece hashes. I can force-recheck the seeding torrents if you think it's worth it. I take your correction: the zero-filled data is a lead, not proof of DMA, and this test doesn't separate the drive from the adapter or from how they interact with the system. So the plan is: the FORESEE and adapter stay out, I keep monitoring, and I keep the older known-good backups alongside the new ones. If anything comes back, I'll follow your order (Quadro + NVIDIA plugin, then HBA slot, then DIMM sets). One anecdotal observation: the drive-bay activity LEDs behave normally now, and they didn't seem to during the bad period. I haven't verified whether it's related. Thanks a lot for the guidance through all of this. Is there anything else you'd like me to check? Otherwise I'll keep monitoring and only post again if anything shows up. Attachment: alpha-one-week-test-2026-10-01.zip (diagnostics, external syslog + per-day summary, health snapshot, list of changes/restarts, database checks, Immich checksum verification, share-scan method/results/scripts, smartctl -x per disk) alpha-one-week-test-2026-10-01.zip

Quick update, 2+ days after pulling the FORESEE eMMC The FORESEE E2M2 and its NVMe adapter came out on Sep 24 around 12:20. Everything else is unchanged (7.3.2, same CPUs, RAM, HBA, Quadro, IOMMU left alone). Status as of Sep 26 15:20: up 2 days 3 hours with nothing at all. 0 winbindd aborts, which I'm using as the earliest marker as you suggested. 0 segfaults, GPF, oops, BUG or hung tasks. EDAC and MCE clean, /mnt/user fine. All array disks and pools healthy, SMART clean. For comparison, the best run on the new CPUs was about 34.5 h to the first fault (winbindd). The server hasn't been idle either: normal container use (about 30 containers) Plex intro detection pegging all cores Paperless re-processing 51 documents full reads of every database a parallel read of every file on all user shares (still running) While checking everything, I found that the zeroing had also persisted inside application databases during the bad period. Details are in the attached txt; summary: Immich, two separate PostgreSQL 16 containers. pg_dump failed on corrupt tuples: One asset row had its tail zeroed: enum columns "invalid internal value for enum: 0", a bad varlena length, and timestamps at the PG zero date. One smart_search row had a missing TOAST chunk. In the other instance, one asset_face row was fully zeroed (all-zero UUID). amcheck heapallindexed showed the btree indexes still holding the ORIGINAL values for the zeroed row. The heap page was changed underneath PostgreSQL, not through SQL. The photo file itself was intact (SHA1 matches). Plex (SQLite). integrity_check reported "row missing from index" x5 for one metadata_items row: The row has "index", air date, added_at, created_at and updated_at all = 0, while the neighbouring rows are normal. A query through the indexes can't find it; a NOT INDEXED full scan does. Same pattern: the indexes kept the originals. Plex's own 3-day DB backups stopped after Jul 27. Nextcloud (MariaDB/InnoDB): 600x "Index for table 'oc_filecache' is corrupt" in nextcloud.log. Only on Jul 11-14, Aug 9-13, Aug 17-22 and Sep 1, i.e. the episode windows. The table checks clean today, so that one looks like it stayed in memory. Code files with NUL bytes inside containers: a Carol-immich JS file (fixed earlier by recreating the container) Paperless numpy .py files on Aug 2-3 ("source code string cannot contain null bytes") plus 122 Paperless OCR worker SIGSEGVs between Jul 26 and Aug 18. Everything above has been repaired or re-imported from intact originals. Everything else checks clean: Sonarr/Radarr/Vaultwarden/Calibre/Syncthing/NPM SQLite, the Linkwarden PG17 amcheck, CouchDB full reads, and Paperless checksums. None of it is new: every damaged item dates from the episode period, and nothing has appeared since the eMMC came out. It fits the theory of zero runs landing in page cache / buffer memory and later being written back to disk. Plan: keep it running untouched until the one-week mark (Oct 1). Then I'll post a final update with fresh diagnostics and the result of the zero-file scan across all shares. If anything shows up before then, I'll follow your order: Quadro + NVIDIA plugin first, then the HBA slot, then reduced DIMM sets. Is there anything else you'd like me to capture at the one-week mark besides the diagnostics?

Update after the last round of hardware tests. Short version: replacing both CPUs didn't change anything. WHAT HAPPENED SINCE MY LAST UPDATE 1) Sept 1 (still original CPUs, after BIOS defaults + PSU2) After the ~25 h clean window the box got progressively worse. On top of the shfs crash I already reported, at 16:25 there was a kernel Oops inside the SPL/ZFS module: BUG: kernel NULL pointer dereference, address: 0000000000000008 RIP: 0010:spl_cache_flush+0x5e/0x160 [spl] note: z_wr_int_2[9680] exited with irqs disabled After that the webGUI went down: php-fpm workers crash-looped (tens of thousands of SIGABRT exits), and from 17:00 several php-fpm workers segfaulted at the exact same address on different cores and both sockets: php-fpm[...]: segfault at 6e696159 ip 0000000000a35c80 ... error 4 in php-fpm[635c80,600000+49b000] For the record, this did not survive a reboot: the reboot I issued at 17:09 never completed, the same session kept running until I powered it off at 17:23, and there's no new boot in the logs. 2) Sept 8-10: swapped the two CPUs between sockets (RAM untouched) Over about 2 days: 18 userspace segfaults, 8 on socket 0 and 10 on socket 1. Before the swap it was 9 of 11 on socket 0 (Sep 1 and Sep 5-6 windows). Also: Sep 9 18:07:06 BUG: Bad rss-counter state mm:... type:MM_FILEPAGES val:45 Comm:selkies BUG: non-zero pgtables_bytes on freeing mm: 8192 (dbus-daemon segfault in the same second) Sep 10 03:28:15 BUG: kernel NULL pointer dereference, address: 0000000000000360 RIP: 0010:fuse_dentry_revalidate+0xb7/0x2f0 (CPU 6, Comm: libuv-worker, openat path) 3) Sept 22: installed two replacement E5-2450 v2 (same model) First boot only showed 80 GB (DIMM_B1 and B4 not detected). I reseated CPU2 and cleaned the DIMM contacts, and after that all 12 DIMMs / 96 GB were detected. It then ran clean for about 36.5 h, the longest clean stretch so far, and then the same things came back: Sep 24 01:46:34 smartctl_type: segfault at 0 ip 0000000000a66303 error 4 in php[666303,600000+48d000] (socket 0) Sep 24 03:00:05 shfs[10594]: segfault at 10 ip 000014ecbb55634c error 4 in libfuse3.so.3.16.2[b34c,...] (socket 0) Sep 24 03:00:21 traps: postgres[55755] general protection fault ... error:0 in libc.so.6[2650f,...] Sep 24 03:00:26 traps: postgres[56467] general protection fault ... error:0 in libc.so.6[2650f,...] The shfs crash has the same offset as the Sept 1 one (libfuse3 +0xb34c; Sept 1 was on socket 1 with the original CPUs, this one on socket 0 with the new ones). It took /mnt/user down again (Transport endpoint is not connected), both Immich Postgres containers died 16 s later, and most containers were erroring until I remounted /mnt/user and restarted them. Over the full 46 h on the new CPUs, those four events (all between 01:46 and 03:00 on Sep 24) were the only ones, fewer than on the old CPUs but including the most damaging one. Also during this boot: - A file inside one container (Carol-immich) had its first 416 bytes zeroed on disk: a 687-byte JS file, the rest intact, cut mid-word. The same file in a second container built from the same image was fine, and recreating the container fixed it. Details attached. - winbindd started aborting every 15 minutes from Sep 23 23:36 (about 34 h after boot), always in a forked child: messaging_reinit -> reinit_after_fork -> talloc destructor -> abort(). The parent winbindd has been running since boot. Throughout all of this: EDAC 0 CE / 0 UE on both memory controllers, no MCE, SMART OK on all array disks, CPU temps 33-48 C. OFFSETS THAT KEEP REPEATING ACROSS THE ORIGINAL, SWAPPED AND NEW CPUS - php +0x666303 (smartctl_type): since July, again on the new CPUs - libfuse3.so.3.16.2 +0xb34c (shfs): Sep 1 and Sep 24 - ld-linux-x86-64-2.43.so +0xb34c (smartctl): Sep 1, Sep 9, Sep 10 - libc-2.43.so +0xa7ddd (smartctl): August and Sep 9 - libpython3.14.so +0x1bed15 (celeryd): Sep 9 and Sep 10 So: different CPU silicon, same crash sites. I think that rules out the two original CPUs as individual bad units, but I'm not sure what it points to instead. NEW: WHAT THE OLD DIAGNOSTICS ON THE FLASH SHOW I went through the 30 diagnostics zips saved in /boot/logs (2024 to now) and compared lspci and the syslog in each: 2024-12-01 6.12.13 (6.1.106) no NVMe 0 segfaults 2026-04-02 7.0.0 (6.6.68) NVMe = MAXIO MAP1202 [1e4b:1202] 0 segfaults, syslog covers Feb 18 - Apr 2 2026-07-09 7.3.0 (6.18.x) NVMe = FORESEE E2M2 [1217:8760] 12,227 segfaults, 125 Oopses (syslog from Jul 7) (The 91 segfaults in a Jan 2024 zip are all Plex "PMS LoudnessCmd" in libswresample, a known Plex bug, unrelated.) Between the last clean snapshot and the first bad one, the only differences in lspci are the NVMe on the PCIe adapter at 0c:00.0 (MAXIO swapped for a FORESEE E2M2, O2 Micro controller, together with a different M.2-to-PCIe adapter) plus the Unraid upgrade 7.0.0 -> 7.3.0. The flash backup git history dates both: - 2026-05-15: config rewrite matching the 7.3.0 upgrade (7.3.0 was released May 12) - 2026-05-29: pool "nvme" renamed to "emmc" and the E2M2 appears for the first time, i.e. the FORESEE went in - There were no scheduled parity checks before Aug 1, yet one ran on the morning of May 29, starting ~07:04 right at a boot/array start, finishing 12:54. The FORESEE swap was committed at 14:41 that afternoon. If that check was the automatic one after an unclean shutdown, the box was already misbehaving on 7.3.0 with the old MAXIO, before the FORESEE. I can't confirm that from the system (notifications live in RAM), and I honestly don't remember whether I started that check by hand. - I swapped the MAXIO out because I wanted it for another machine, not because it was failing. It's been working fine there since. So the two changes are two weeks apart, and the diagnostics can't tell me which one lines up with the start. The 7.2.8 test doesn't separate them either, because the FORESEE was installed during that test too. Some things about that FORESEE: - All its SMART counters read 0 (power-on hours, power cycles, data written). - The kernel logs "missing or invalid SUBNQN field" and "nvme nvme0: using unchecked data buffer". As I understand it, the second one means passthrough commands (which smartctl uses) go to a device without SGL, so the kernel can't bound how much the device DMAs into the buffer. - IOMMU is enabled but the default domain is Passthrough, so a device writing where it shouldn't wouldn't be caught. - It sits on the C600 PCH root port 00:1c.0, x1 Gen2. - Since Sep 6 it hasn't had a pool on it and it's mostly idle, but it's still on the bus and still gets SMART polls. I'm not claiming that's the cause. The "unchecked data buffer" warning shows up on lots of consumer NVMe drives without problems, and the crashes didn't slow down once the drive went idle. But it's the one hardware change I can line up with when this started, and it's free to test. (Side note: I also tried to boot Ubuntu from that same card to test a different kernel, but the T420 can't boot from the adapter, so that test never happened.) WHAT I'M DOING NEXT Removing the FORESEE (and the PCIe adapter) now and staying on 7.3.2, so only one thing changes. The longest clean stretch so far was about 36.5 h, so I'll let it run at least a week before drawing conclusions. If it keeps crashing without it, the other half of that window is the 7.0.0 -> 7.3.0 upgrade. 7.0.0 (kernel 6.6.68) is the last version I have evidence of running clean on this hardware (6 weeks), and 7.2.8 (6.12.98) already reproduced it, so a regression would have to be somewhere between 6.6 and 6.12. I could boot 7.0.0 from a spare stick with the array stopped (the ZFS pools were feature-upgraded on Jul 28, so 7.0.0 probably can't import them). I know you said to stop comparing versions, but this one would be the last known-good. Other candidates: removing the Quadro P2200 and the Nvidia driver plugin (so the kernel isn't tainted), running on half the RAM and then the other half, and moving the HBA to another slot. Beyond that it's a replacement motherboard or new hardware. Does that order make sense to you, or is there something you'd check first? Attached: - alpha-diagnostics-20260924-1136.zip: current boot (new CPUs, the Sep 24 shfs crash, the winbindd loop) - alpha-diagnostics-20260402-1312.zip: the last clean snapshot (7.0.0, MAXIO NVMe) - alpha-diagnostics-20260709-1910.zip: the first bad snapshot (7.3.0, FORESEE) - alpha-syslog-excerpts-2026-09-24.zip: kernel lines from my remote syslog for Sep 1, 5, 6, 8, 9, 10, 22 and 24, one winbindd panic with full backtrace, and the zeroed-file hexdump alpha-diagnostics-20260924-1136.zipalpha-syslog-excerpts-2026-09-24.zipalpha-diagnostics-20260402-1312.zipalpha-diagnostics-20260709-1910.zip

Quick update. After the BIOS factory-reset + PSU2 swap, the system ran completely clean for about 25.5 hours (previous failures were typically showing up within just a few hours of boot) — first sign of trouble was a segfault at 23:12 on 31/08. Since then we've had 6 userspace segfaults total over about 12 hours, plus one incident that had a more visible impact: at 09:24 today, Unraid's own "shfs" User Share daemon (the FUSE process backing /mnt/user) segfaulted and took the whole /mnt/user mount down ("Transport endpoint is not connected") until I manually unmounted/remounted it — several containers with an open handle into the old mount (two Postgres instances, Vaultwarden, Paperless) needed a restart afterward to recover cleanly. All 6 segfaults share the same signature: fault address near-zero (0x0–0xa9), error code 4 (user-mode read on a not-present page) — a pointer that should point to a valid struct coming back null in every case, across 6 completely unrelated processes (a Plex updater, smartctl, shfs, node, and Celery/Python twice). Two of them — smartctl_type and shfs — faulted at the exact same offset (0xb34c) inside two totally unrelated shared libraries, which reads more like something corrupting memory at a consistent relative position than a bug in either piece of software. EDAC and the MCE log stayed silent through all of it (0/0 CE/UE, no machine-check entries), so whatever this is still isn't tripping the standard ECC/MCE detection path.* For what it's worth, 5 of the 6 crashes landed on CPU socket 0, only 1 on socket 1 — not enough to draw a conclusion, but noting it since we're about to try swapping CPUs. So: longer time-to-first-failure than before, but once it started again the pattern and frequency look similar to what we'd seen previously, just with a new symptom this time (a core Unraid service crashing instead of an application). Continuing to monitor, and will report back once the CPU swap is done.

Understood, and that matches what we were starting to suspect ourselves given the offset repeating across kernel versions. Appreciate you laying out the reasoning clearly. 7.2.8 FINAL NUMBERS (stopped the count here, past your 20.5h reference point) 32h29m elapsed: zero kernel oopses, five smartctl/smartctl_type segfaults total (one more since the last update, at 12:42:18, offset 0x666314 in php — same family as the 0x666303 one, one byte apart). EDAC stayed active and clean the entire time (0/0/0/0 across both memory controllers). Array, pools, Docker and VMs stayed stopped/disabled throughout, zfs and spl modules loaded but pools never imported. Not running a fresh-7.3.2 retest per your note, since you don't think it's necessary at this point. MOVING TO HARDWARE ISOLATION Putting the original, untouched boot USB back in for regular use while we work through this, since the fault clearly isn't tied to which media or which Unraid version is booted. Going to start with your suggestion of substituting known-good compatible parts. Ordering a couple of replacement CPUs first (they're inexpensive for this platform) to rule those in or out before touching memory — with the current state of things, going down the path of buying replacement RDIMMs as well if the CPUs don't resolve it is going to get expensive fast, so we're prioritizing the cheaper substitution first. Also checking the PSU2 situation now that you mention power delivery as a candidate — it's been sitting in a "input lost" state for a while, which we'd previously assumed was unrelated, but given the pattern reset given by this reply we're going to make sure it's actually getting power before ruling it out. Will keep this thread updated as we work through the hardware substitutions.

Hi again, still running Unraid on 7.2.8 on other flash drive I used for the 7.3.2 test, this time unlicensed since it isn't offering me the free trial option anymore. Still have the original USB completely untouched with the original OS on it, for what it's worth. After all these tests, honestly not sure what else to try beyond what you've already suggested. On the ARC/shrinker point: fair correction, and I was overreaching. Re-checked the complete original trace, and it does not touch anything ZFS-specific it's the generic workingset shadow-LRU path throughout, no ARC callback frame anywhere in it. Full trace for reference: kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 kernel: #PF: supervisor read access in kernel mode kernel: #PF: error_code(0x0000) - not-present page kernel: Oops: Oops: 0000 [#1] SMP PTI kernel: CPU: 24 UID: 0 PID: 352 Comm: kswapd0 Tainted: P O 6.18.38-Unraid #1 PREEMPT(voluntary) kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE kernel: RIP: 0010:_raw_spin_trylock+0x13/0x50 kernel: Call Trace: kernel: shadow_lru_isolate+0x1f/0x150 kernel: __list_lru_walk_one+0xb8/0x140 kernel: list_lru_walk_one_irq+0xc/0x20 kernel: do_shrink_slab+0x199/0x2d0 kernel: shrink_slab+0x16e/0x2a0 kernel: shrink_node+0x515/0x7a0 kernel: balance_pgdat+0x456/0x670 kernel: kswapd+0x292/0x2e0 kernel: kthread+0x1ce/0x1e0 kernel: ret_from_fork+0x24/0x170 kernel: note: kswapd0[352] exited with irqs disabled kernel: note: kswapd0[352] exited with preempt_count 2 Agreed the zfs/spl modules being loaded without an active pool proves nothing either way — recording it as requested, not claiming it as evidence. 7.2.8 ARRAY-STOPPED SAFE MODE TEST — IN PROGRESS Matching the previous test conditions: array and pools stopped (not importing the upgraded pools, per your note), Docker and VM services disabled, sb_edac loaded 3 minutes after boot (earliest yet), remote logging confirmed working throughout. At 17h11m so far: zero kernel oopses, but four smartctl/smartctl_type segfaults: kernel: smartctl_type[129171]: segfault at 0 ip 0000000000000000 sp 00007ffff7b96708 error 14 kernel: smartctl[631166]: segfault at bfa60 ip 00000000000bfa60 sp 00007ffc58a65228 error 14 kernel: smartctl_type[1003123]: segfault at c ip 00000000009be224 sp 00007ffc4afa9dc0 error 4 in php[5be224,600000+48d000] kernel: smartctl_type[1476102]: segfault at 0 ip 0000000000a66303 sp 00007ffd5e931128 error 4 in php[666303,600000+48d000] The second entry (fault address bfa60, ip bfa60, error 14) is byte-for-byte identical to a fault we saw on the original 7.3.2 stick. The fourth (offset 0x666303 in php) is the same offset you already flagged as matching between two earlier 7.3.2 occurrences — now a third occurrence, this time under 7.2.8's kernel (6.12.98) instead of 7.3.2's (6.18.38). Continuing to run this past the 20.5h mark, targeting at least 48h as you suggested, before moving to the fresh-7.3.2 retest under matching conditions. prueba728_2026-08-29.logprueba-7-2-8-safemode-diagnostics-20260829-2106.zip Thank you :)

Results from the extended Dell diagnostics and the memtest86+ run, plus something we should have surfaced earlier: an OpenZFS pool feature upgrade done before this thread started that may be relevant. DELL EXTENDED DIAGNOSTICS (Thorough Test Mode, all devices) First pass (standard, not thorough) came back "Fail" only because of the event-log-content check — actual hardware results were Pass=267, Warning=65, Fail=0, Critical=67, all 67 "Critical" being the already-known PSU2/chassis-intrusion history. Re-ran with Thorough Test Mode enabled across all devices, including the Processor and Memory categories and full DST Long Tests on all six array disks: result was a clean "All tests passed", including the disk that took the longest (3h25m). System Health throughout: fans steady ~2640-2760 RPM, CPU1/CPU2 in the mid-30s C under load, PSU1 normal voltage (232-240V), system power draw 98-112W. MEMTEST86+ (all cores, all 95.9GB) 25h31m elapsed, 5 complete passes, 0 errors. Screenshot attached. CPU stayed in the mid-30s to mid-50s C throughout, no thermal concerns. ZFS HISTORY WE SHOULD HAVE MENTIONED SOONER Before this thread started, we'd already made two changes trying to address the instability, both ZFS-related on the two pools (cache, seedeo, both raidz/mirror on top of the array): 1. 2026-07-28: ran zpool upgrade (feature flag upgrade) on both pools, followed by a full scrub (0 errors on both). This was prompted by an unrelated product-board report describing a similar Oops pattern tied to a pending ZFS feature upgrade. First night after was clean (~9.5h, no oops, previously-known segfaults temporarily absent), but the same failure pattern resumed about a week later (2026-08-04), so this wasn't the root cause, or at least not the only one. 2. 2026-08-06 17:08:59: raised zfs_arc_max to ~19.2GB (currently still set: 20615843020 bytes, confirmed just now). The same evening, two new segfault types appeared that hadn't been seen before — one in a node process, one in emhttpd against libc. Memory pressure was checked and ruled out at the time (94GiB total, 73GiB available), but whether the ARC size change itself was a factor was never confirmed either way. Worth flagging: ZFS's ARC registers its own shrinker with the kernel's memory-reclaim subsystem — the same list_lru/shrinker path as the shadow_lru_isolate oops we reported early in this thread. We hadn't connected those two things until now. All of our more recent array-stopped Safe Mode observations (which is where essentially all of our strongest evidence has come from) happened with these pools unmounted, so they can't be a confound for those specific results — but the ARC/shrinker connection is a loose thread from earlier in the investigation that we don't think has been ruled out. Checked just now, pools are healthy: both ONLINE, 0 errors, scrub timestamps match the 2026-07-28 event exactly. Current active feature set on cache includes several from the OpenZFS 2.2+/2.3+ era: log_spacemap, zilsaxattr, head_errlog, block_cloning, vdev_zaps_v2, block_cloning_endian (seedeo has a similar but slightly smaller set — no zilsaxattr or block_cloning, presumably because those datasets never triggered them). Running OpenZFS 2.4.3-1. The practical concern: the feature upgrade is irreversible, and 7.2.8 ships an older OpenZFS than 7.3.x. It's likely 7.2.8 will refuse to import these pools given the newer active features. For the planned array-stopped Safe Mode comparison test this shouldn't matter, since the pools won't be touched. If we later want to test with the array/pools active on 7.2.8, we're expecting a possible import failure there and will treat that as a version-compatibility issue rather than new evidence, not a surprise. Given Dell diagnostics and memtest are both clean so far, planning to proceed with the 7.2.8 array-stopped Safe Mode comparison next, same methodology as the fresh-7.3.2-media test.

Ran the exact same array-stopped Safe Mode test on the freshly created 7.3.2 media (identical boot-file hashes to both our original stick and the official release, verified beforehand). Result: reproduced the same failure signature. In just under 20 hours: zero kernel oopses, but two smartctl_type segfaults in php, same pattern as before. This weighs against the boot media itself being the source of corruption, since this stick was written fresh and verified byte-identical before the test started. prueba_2026-08-26.logprueba_2026-08-27.logidrac_power.jsonidrac_sel.json prueba-safemode-diagnostics-20260827-1555.zip Also checked something we hadn't looked at directly yet: iDRAC power supply telemetry and the full SEL. PSU1 (the one actually powering the system) reports normal voltage and OK health throughout; PSU2 shows the same long-known "input lost" condition we'd already identified as unrelated. Total system power draw is a steady ~120W. The SEL itself has zero new entries of any kind since December 2025 nothing logged during this entire investigation, including through the physical USB swap. One more note, expected rather than new: the WebGUI on this fresh install doesn't show live CPU/memory stats, and the Unraid API service is offline. Confirmed this is the same Safe-Mode-disables-the-Connect-plugin behavior we already identified on the original stick, not a repeat of the php-fpm corruption incident php-fpm itself is healthy here (workers spawned, classic GUI pages load normally), it's specifically the API-backed dashboard widgets that are unavailable by design in Safe Mode. Starting the extended Dell diagnostics and memtest passes now rather than scheduling for later have someone watching the console for this one. Will report back once those finish.

Thank you a lot :)

Sounds good, starting with these. Reordering slightly for efficiency (nothing dropped, just resequencing) hashes captured now, then preparing fresh 7.3.2 boot media in parallel (doesn't require touching Alpha), then repeating the array-stopped Safe Mode test on that fresh media before committing to the Dell diagnostics + memtest window, since that one costs the most downtime and is worth running after the cheaper tests narrow things down. 7.2.8 comparison last, as you suggested. Will report back as each stage finishes.

Update from the Safe Mode + array-stopped test. Ran for 20 hours 31 minutes before I checked in well past both the 13-hour target and the previous 4h38m first-fault interval. ZERO KERNEL OOPSES None of the three tracked fault types (shadow_lru_isolate, xfs_iget, mas_ascend) occurred in over 20 hours with the array stopped and Docker off. FOUR MORE SMARTCTL_TYPE SEGFAULTS 17:08:01, 18:39:31, and 20:08:29 yesterday, then 01:40:59 this morning same recurring crash as before, three in php, one in the dynamic linker (ld-linux-x86-64-2.43.so) this time instead of php itself. THE SIGNIFICANT ONE: A LIBC SEGFAULT IN ELOGIND-DAEMON At 10:48:38 this morning, elogind-daemon (PID 2564, the persistent core session-management daemon, not a periodic script) segfaulted inside libc-2.43.so itself. In the same second, an incoming SSH login recorded: "pam_elogind(sshd:session): Failed to create session: Message recipient disconnected from message bus without replying" — the daemon died mid-handshake of an entirely routine session open. It self-healed in 8 seconds: a fresh elogind-daemon (new PID) came up, re-registered the seat, and normal session handling resumed immediately after. This is a segfault in the single most fundamental, most heavily linked shared library on the system, inside an always-running core daemon, with Docker, VMs, plugins, and the array all fully stopped. About as little as we can strip away from this box while it's still bootable. EDAC COVERAGE THIS TIME sb_edac was loaded at 14:21:05 yesterday, about 20 hours before this crash. So unlike the smartctl_type/php-fpm incidents from the first Safe Mode run, this one has full ECC visibility. Checked immediately after: 0/0/0/0 across both memory controllers, same as always. ONE MORE THING WE FOUND WHILE PULLING LOGS Going back through yesterday's full daily log to grab it for you, we found that the pool www child-respawn storm from the php-fpm corruption (Aug 25, starting 09:49:59, before the reboot) ran long enough and hard enough to fill Alpha's local /var/log (tmpfs) completely — over 4.1 million lines, ~594MB, in a single day. rsyslogd itself started throwing "No space left on device" write errors on the local syslog at 14:14:09, right as we were shutting down for the clean reboot. Remote logging to the archive kept working throughout since that's a separate destination, so nothing was lost, but worth flagging as its own side effect of that incident — the same disk-full-tmpfs failure mode we hit once earlier in this whole investigation, this time self-inflicted by the crash-loop's own log volume. ATTACHMENTS Two files this time, kept small on purpose: the pool www spam block from yesterday morning is redundant with what we already reported and was millions of lines, so I trimmed it out. Attaching only the post-reboot portion of yesterday's log (from the clean reboot at 14:14 onward, ~2,760 lines) plus today's full log so far, covering the elogind/libc crash. Diagnostics zip to follow separately. alpha_2026-08-26.logalpha_2026-08-25_postreboot.logalpha-safemode-diagnostics-20260826-1110.zip Array and Docker remain stopped. Will keep this configuration running further unless you'd like us to move on to the boot-archive hash verification next.

You're right on every point here, and I appreciate the correction. Addressing each one, plus the data you asked for, plus an update on where we are right now. EDAC TIMING Confirmed exactly. sb_edac registered at 09:57:35. The smartctl_type fault was at 01:39:58 and the php-fpm collapse started at 09:49:59 (syslog shows php-fpm's pool www children exiting with code 127 starting at that exact second, hundreds of them in the same burst). Both predate EDAC coming up by hours/minutes respectively. You're correct: neither failure has any ECC visibility, and that overnight run can't be used to exclude a hardware/memory-path cause. That was a real mistake on my end, not a rounding error — I should have checked the module load time against the fault times before writing that up. "NOT COMPLETELY CLEAN" Agreed. smartctl_type executing address zero with Docker off, plugins skipped, and NVIDIA modules absent is a real, notable data point, not a null result. I shouldn't have flattened it into "0 Oops = clean." Also agreed it doesn't establish a base-OS defect on its own — could just be the utility hitting already-corrupted state rather than corrupting anything itself. PHP-FPM "PROOF" Agreed, dropping caches and having it come back clean does not prove RAM-page corruption specifically. I overstated that. It's equally consistent with the read path, decompression path, or boot media, as you said. THE TWO EARLIER SQUASHFS INCIDENTS, IN FULL Incident 1 (2026-08-13, ~03:05:40): 2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression error: 20 2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression failed, data probably corrupt 2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: Failed to read block 0x3374df0: -5 2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression error: 20 2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression failed, data probably corrupt 2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: Failed to read block 0x3374df0: -5 Surrounding window: preceded by disks spinning down at 02:53-03:05 and a .NET segfault at 02:48:52. Followed by normal SMART reads and the scheduled mover run at 03:40:01 with no further errors. Same block (0x3374df0) failed identically twice in the same second. Incident 2 (2026-08-14, two bursts at 08:36:37 and 08:36:50): 2026-08-14T08:36:37+02:00 Alpha kernel: SQUASHFS error: zstd decompression error: 20 2026-08-14T08:36:37+02:00 Alpha kernel: SQUASHFS error: zstd decompression failed, data probably corrupt 2026-08-14T08:36:37+02:00 Alpha kernel: SQUASHFS error: Failed to read block 0x206e1fb2: -5 (repeated 4x at 08:36:37, then 3x more at 08:36:50, same block 0x206e1fb2 every time) Surrounding window: preceded by a smartctl_type segfault at 05:27:21 and a celeryd/libpython segfault at 07:20:00 earlier that morning. Nothing unusual immediately after — normal SSH session, no further errors in that window. Both incidents self-cleared on the next read without manual intervention at the time; this was noted days ago as "came back clean on retry," before we started saving raw excerpts, which is why we couldn't cite the original lines until now — this is what we had in the archive. FORENSICS ON THE UNRAID-API FILE, AS REQUESTED findmnt -T /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js TARGET SOURCE FSTYPE OPTIONS /usr /dev/loop0 squashfs ro,relatime,errors=continue /usr overlay overlay rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr,uuid=on sha256sum: d66f16d9ba336f833ed6e19fc46388ea01ff8f27d33292874ea8f6b6d45b0c15 file-validator.interface.js file: file-validator.interface.js: JavaScript source, ASCII text xxd wasn't available on this box (busybox/minimal userland), so I used od -A x -t x1z -v instead. First 128 bytes: 000000 22 75 73 65 20 73 74 72 69 63 74 22 3b 0a 4f 62 >"use strict";.Ob< 000010 6a 65 63 74 2e 64 65 66 69 6e 65 50 72 6f 70 65 >ject.definePrope< 000020 72 74 79 28 65 78 70 6f 72 74 73 2c 20 22 5f 5f >rty(exports, "__< 000030 65 73 4d 6f 64 75 6c 65 22 2c 20 7b 20 76 61 6c >esModule", { val< 000040 75 65 3a 20 74 72 75 65 20 7d 29 3b 0a 65 78 70 >ue: true });.exp< 000050 6f 72 74 73 2e 46 69 6c 65 56 61 6c 69 64 61 74 >orts.FileValidat< 000060 6f 72 20 3d 20 76 6f 69 64 20 30 3b 0a 2f 2a 2a >or = void 0;./**< 000070 0a 20 2a 20 49 6e 74 65 72 66 61 63 65 20 64 65 >. * Interface de< 000080 73 63 72 69 62 69 6e 67 20 46 69 6c 65 56 61 6c >scribing FileVal< 000090 69 64 61 74 6f 72 73 2c 20 77 68 69 63 68 20 63 >idators, which c< That's clean, well-formed CommonJS nothing visibly garbled. So I went a step further and tried loading it directly with a plain, uninstrumented require() outside of the unraid-api process. It loaded without error. So right now, on disk, this specific file is not corrupted by any check available to me. That undercuts my "corrupted file" read of the API failure specifically. The original failing require() went through require-in-the-middle (an instrumentation/hook library the API's tracing setup uses to patch Node's module loader) rather than a plain require. Given the file itself checks out clean now, my best guess is either that hook doing something to the module at load time, or a transient read at the moment it was first loaded that's since cleared the same way the two SquashFS incidents did not persistent on-disk corruption of this particular file. I don't have a "before" checksum to compare against, so I can't fully close this out either way. I also don't have before/after evidence for the php-fpm binary — the cache-drop fix was already applied before I saw your evidence-preservation request, so there's no clean "before" state left to check now. If it recurs, I'll capture filesystem/checksum/ldd first, before touching anything. WHERE WE ARE RIGHT NOW Rebooted cleanly and back in Safe Mode. This time sb_edac was loaded manually 3 minutes after boot (14:18:02 boot, 14:21:05 EDAC up) rather than 8+ hours in like last time still not from second zero, since Safe Mode skips our persistent boot script entirely by design, so it has to be loaded by hand every time we boot this way. Also, since the array happened to still be stopped after this boot, we're deliberately leaving it that way for a while rather than starting it — which covers your "Safe Mode with the array stopped" suggestion directly, as its own distinct phase, before we move on to Dell diagnostics and memtest. ATTACHMENTS Attaching four full daily log files from the remote syslog archive so you can check the surrounding context yourself rather than just my excerpts: the two covering the original SquashFS incidents (2026-08-13 and 2026-08-14), plus the two from the more recent overnight/reboot incident (2026-08-23 and 2026-08-24). Also attaching a fresh diagnostics zip taken just now, in the current state (Safe Mode, array stopped, sb_edac loaded). alpha-safemode-diagnostics-20260825-1523.zip2026-08-25.zip alpha_2026-08-13.logalpha_2026-08-14.log

alpha-safemode-diagnostics-20260825-1138.zip 2026-08-24.log 2026-08-25.zip Sorry I didn't attach it in my previous message, the ZIP file was still being created. It wasn't letting me upload today's log because it's massive, so I had to compress it.

Quick update from this morning. The Safe Mode test survived overnight cleanly 13 hours, zero kernel Oopses of any of the three types we'd been tracking. One segfault did occur, though, and it's telling: it was Unraid's own built-in "smartctl_type" SMART-monitoring helper, part of the base system, not a Docker container or a plugin. With everything else stripped away, something in the base OS still crashed once. Separately, I made a mistake worth flagging: I didn't realize Safe Mode skips our custom boot script, so the sb_edac module (our ECC monitor) wasn't loaded for the entire overnight run. We had zero memory-error visibility during the exact window we most wanted it. Reloaded it now; keeping Safe Mode running for the rest of today with it active this time. Also ran into something more significant this morning: the WebGUI became completely unusable every PHP page just hung indefinitely rather than erroring. Root cause: php-fpm itself was broken at the binary level, throwing "undefined symbol" errors on startup (its master process was spinning with zero worker processes spawned). php-fpm lives in the read-only squashfs image that gets cached into RAM at boot the same image involved in the two squashfs decompression-error incidents we mentioned earlier in this thread. Dropping the page cache (echo 3 > /proc/sys/vm/drop_caches) to force a clean re-read from the physical USB fixed it immediately no reinstall, no file changes, just forcing RAM to re-fetch the same bytes from disk. That's about as direct a confirmation as we're likely to get that pages are getting corrupted in RAM specifically, since the exact same on-disk bytes came back clean on a forced re-read. Separately (and not fixed the same way): the Unraid API service (dynamix.my.servers backend, powers the modern dashboard's live stats) is also throwing the identical "SyntaxError: Invalid or unexpected token" in one of its own node_modules files, but it lives in the writable overlay rather than the read-only squashfs, so the cache-drop trick didn't touch it. Left it stopped for now rather than let it crash-loop; will reinstall the plugin once we're done with Safe Mode testing. Plan for today: staying in Safe Mode with Docker and VMs off for the rest of the day, now with EDAC properly active. If it stays clean, we'll start bringing Docker containers back up in small batches tomorrow rather than all at once, watching the log between each batch. php-fpm-corruption-addendum.txt

Good afternoon! Understood, I'll get started on those tests and keep you updated. In a couple of hours once I finish work, I'll boot it up in Safe Mode with Docker disabled (VMs are already off since I don't really use them). I'll leave it running like that overnight to see how it holds up, and tomorrow morning I'll start enabling the Dockers little by little. Thanks a lot for the help! :)