Hi, good evening everyone!

Apologies in advance for my English, it’s not my native language, im using a translator :)

I'm no tech expert, I really enjoy it, but I have to admit some of this stuff is a bit tough for me hahahah.

I have a modest server setup: a Dell T420 with dual Xeon E5-2450 v2 CPUs. For over a month now, I've been running into a lot of issues. A friend recommended I use Claude in Visual Studio Code to help diagnose what's going on with the server.

I even connected an old laptop to capture the Syslog in real time, just in case it froze completely again like before, leaving me with no SSH access or anything.

After analyzing the error logs for a couple of weeks, it seems the conclusion is a kernel panic/error related to the dual-CPU setup. This started right when I updated to version 7.3.2, so I'm considering rolling back to the previous version, but I'd love to get your opinion first.

I wanted to have at least some context before coming here so I wouldn't be completely clueless ;-;

Here is the summary Claude gave me. Thank you so much in advance for your help! :)

SYSTEM

- Unraid 7.3.2, kernel 6.18.38-Unraid

- Dell PowerEdge T420, 2x Intel Xeon E5-2450 v2 (dual-socket, dual NUMA node), 96GB DDR3 ECC RAM

- Broadcom/LSI SAS2308 HBA in IT mode (mpt3sas), 15 disks total across array + 2 ZFS pools (cache, seedeo)

- NVIDIA Quadro P2200 for Plex transcoding

- ~30 Docker containers, mix of custom bridge and host networking

SUMMARY

Over the past several weeks this host has shown a recurring pattern: individual containers dying or hanging one at a time, occasional filesystem-level weirdness, at least one full hard lockup with a clean kernel Oops captured, and — most recently — a second, different kernel Oops plus a batch of application segfaults, all while every hardware indicator (ECC, BMC SEL, SMART, memtest) stays completely clean. Writing up the whole pattern rather than a single incident, because no individual piece looks conclusive on its own but together they point pretty consistently at the kernel's memory-management path. We're now seriously considering downgrading from the 7.3.x/6.18.x line to 7.2.8/6.12.x and would like feedback before doing it.

------------------------------------------------------------

PART 1 — THE TWO CAPTURED KERNEL OOPSES (full stack traces)

------------------------------------------------------------

OOPS #1 — full host hard lockup, kswapd0, shadow_lru_isolate

Captured via a remote syslog server (this host's own local syslog is tmpfs and gets wiped on crash/reboot, so without the remote copy this would have been lost entirely). The host became completely unresponsive — no SSH, no WebGUI, no SMB, and the BMC's remote KVM showed no video signal at all, as if powered off (fans were still spinning). Only a hard power cycle via the BMC recovered it; the console showed a normal "clean shutdown" style prompt frozen mid-boot-flow, suggesting the crash happened very early/deep rather than mid-userspace.

kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000

kernel: #PF: supervisor read access in kernel mode

kernel: #PF: error_code(0x0000) - not-present page

kernel: Oops: Oops: 0000 [#1] SMP PTI

kernel: CPU: 24 UID: 0 PID: 352 Comm: kswapd0 Tainted: P O 6.18.38-Unraid #1 PREEMPT(voluntary)

kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE

kernel: RIP: 0010:_raw_spin_trylock+0x13/0x50

kernel: Call Trace:

kernel: shadow_lru_isolate+0x1f/0x150

kernel: __list_lru_walk_one+0xb8/0x140

kernel: list_lru_walk_one_irq+0xc/0x20

kernel: do_shrink_slab+0x199/0x2d0

kernel: shrink_slab+0x16e/0x2a0

kernel: shrink_node+0x515/0x7a0

kernel: balance_pgdat+0x456/0x670

kernel: kswapd+0x292/0x2e0

kernel: kthread+0x1ce/0x1e0

kernel: ret_from_fork+0x24/0x170

kernel: note: kswapd0[352] exited with irqs disabled

kernel: note: kswapd0[352] exited with preempt_count 2

kernel: pstore: backend (erst) writing error (-38)

Reading: kswapd0 died mid-reclaim inside shadow_lru_isolate() while holding a non-preemptible spinlock, with IRQs disabled (CPU 24, i.e. on the second NUMA node). That's consistent with an orphaned list_lru spinlock — every other CPU that subsequently enters memory reclaim piles up trying to take the same lock, and the whole host goes completely unresponsive. This matches a pattern others have reported on different platforms (see "related reports" below).

OOPS #2 — shfs / XFS inode lookup, non-fatal (host survived)

Happened roughly two days later, same uptime session (i.e. after the reboot that followed Oops #1). This one did NOT take the whole host down — only one shfs worker task died; the shfs master process and the /mnt/user share stayed accessible throughout, confirmed immediately after by listing shares successfully.

kernel: BUG: kernel NULL pointer dereference, address: 0000000000000a18

Comm: shfs, PID 19197

RIP: xfs_iget+0x544/0x8f0

Call trace (top to bottom):

xfs_lookup

xfs_vn_lookup

__lookup_slow

lookup_slow

walk_component

path_lookupat

filename_lookup

vfs_statx

vfs_fstatat

__do_sys_newfstatat

Reading: a plain stat() syscall (from shfs, Unraid's user-share FUSE layer) walked into the XFS inode-lookup path and hit a NULL pointer inside xfs_iget(). Different subsystem than Oops #1 (XFS inode cache vs. page/shadow LRU reclaim), but the same general area — kernel-side cache/lookup structures misbehaving under this workload — and the same "no hardware fault reported anywhere" signature.

------------------------------------------------------------

PART 2 — APPLICATION-LEVEL SYMPTOMS, IN ROUGH CHRONOLOGICAL ORDER

------------------------------------------------------------

Several Docker containers, running completely different software stacks (Mono/.NET, Python, PHP, Node/V8), have shown segfaults or silent deaths over the same multi-week window. Individually each looks like "that app has a bug"; together, the pattern (identical faulting offsets recurring over days, containers reporting Docker "healthy" while the real process is dead, corruption that clears on a fresh read but not on a normal restart) reads more like something underneath all of them is occasionally corrupting state than N unrelated app bugs.

- A media-management container (.NET/Mono runtime) was seen fully hung consuming 100% of a single CPU core with an unresponsive web UI; separately, at another point, we captured a tight burst of 4 segfaults in ~7 seconds, all in libclrjit.so (the .NET JIT compiler) at the identical crash offset:

/home/nobody/start.sh: line 10: Segmentation fault /usr/lib/radarr/bin/Radarr -nobrowser -data=/config

(repeated 4x within 7 seconds, error 6, fault address 0xffffffff each time)

Its process supervisor gave up retrying after the 4th crash ("entered FATAL state, too many start retries too quickly") and never tried again — the container then sat there reporting "healthy" to Docker for the next ~23 hours with zero actual process running, because the healthcheck only checks the supervisor, not the real app.

- Another *arr-family container (also Mono/.NET) had a DLL that was consistently, 100%-reproducibly corrupted — every restart crashed identically at the exact same point loading the same DLL. A plain docker restart never fixed it (same corrupted bytes were already unpacked on disk); only a full "Force Update" (re-pulling the image layers fresh) resolved it. docker pull reporting "image up to date" was not sufficient evidence the on-disk bytes were intact — it only checks the registry manifest, not the already-extracted filesystem layers.

- A third container (Node.js, Mono-adjacent workflow-automation app) hit an equivalent persistent corruption, this time in libssl rather than app code — again clearing only after a full image re-fetch, not a restart.

- A media server (transcoding, uses an NVIDIA GPU passthrough) died with zero trace in its own application log or crash-uploader log — its internal logging simply stopped mid-session and never resumed even though the container reported "healthy" for the following ~2 days. Root cause turned out to be a startup-script bug triggered by the previous incident: after the Oops #1 hard-lockup and reboot, the app's startup script found a stale PID file left over from before the crash, misread it as "already running" inside the fresh container's new PID namespace, and skipped launching the real process entirely — so it just never started, silently, for two full days.

- A bookmarking/link-archiving container (Node.js + V8) has been segfaulting repeatedly in its web process (SIGSEGV in the Next.js/V8 runtime), auto-restarting every few minutes in a continuous crash-loop, correlating with a V8Worker-tagged segfault logged by the kernel's own segfault reporting around the same time as the crashes.

- Filesystem/service-layer weirdness, independent of any single container:

* Twice, a small set of core Samba binaries (rpcd_classic, rpcd_lsad, rpcd_epmapper, samba-dcerpcd) became unreadable ("No such file or directory" from the shell despite ls showing the files present) while sitting in the live, RAM-resident root filesystem — SMB shares became completely inaccessible both times, resolved only by a full reboot.

* Twice, a squashfs read (loop-mounted boot-time modules/firmware images from the boot USB) failed with zstd decompression errors on a specific block; both times, forcing a genuine fresh read from the physical USB (bypassing whatever was cached) came back clean, and checksums of the boot images themselves never changed across the whole investigation — pointing at transient corruption somewhere in the read-into-RAM path, not a failing USB drive.

* /var/log (also tmpfs/RAM-backed on Unraid) once filled to 100%, silently breaking several services that log heavily, until it was identified and cleared.

* Reproduced deliberately, once: stopping/restarting all ~30 Docker containers at once (via rc.docker) reliably triggers kernel messages "BUG: Bad rss-counter state" and "non-zero pgtables_bytes on freeing mm" tied to containerd-shim processes — i.e. the kernel's own process/mm teardown accounting gets confused under concurrent container teardown on this host. Doesn't crash anything by itself, but it's the same general kernel territory (mm/page accounting) as both Oopses above.

------------------------------------------------------------

PART 3 — WHY WE DON'T THINK THIS IS HARDWARE

------------------------------------------------------------

- EDAC (sb_edac) has been loaded and monitoring continuously for over a week, spanning several of the above incidents: correctable and uncorrectable error counts on both memory controllers have stayed at 0 the entire time.

- iDRAC SEL (Dell BMC), checked before and after every incident, shows zero memory/CPU/thermal events — only unrelated chassis-intrusion entries from physical maintenance and expected PSU redundancy-loss entries.

- Memtest86+ (2 full passes) and a ZFS scrub on both pools came back completely clean.

- All 15 disks have clean SMART attributes — no reallocated/pending sectors, no CRC errors.

------------------------------------------------------------

PART 4 — CONSIDERING A DOWNGRADE TO 7.2.8

------------------------------------------------------------

Everything above happened on the 6.18.x kernel line (7.3.0 through 7.3.2), and 6.18 is a regular upstream "stable" release rather than a Linux LTS kernel — i.e. it's had comparatively little real-world mileage compared to an LTS line. We're considering rolling back to 7.2.8 (2026-07-29), which is still on the actively-maintained 7.2.x branch (receiving its own security patches in parallel with 7.3.x) but runs kernel 6.12.98-Unraid — a genuine upstream LTS kernel with far more time in the field.

One more data point that makes us want outside input before doing this: other users reported a "[7.3.1] Kernel oops after zfs task exit" shortly after upgrading onto the 6.18.x line, with crashes several times a day in some cases. Different symptom than what we're seeing, but the same kernel branch and the same general shape of problem — a shrinker/reclaim-adjacent kernel path interacting badly with something (ZFS, in their case; list_lru and XFS inode lookup, in ours). We run two ZFS pools on this box as well, which makes that report stand out to us.

Before downgrading: any feedback on whether this is a sound move for this kind of issue, or knowledge of this specific bug being fixed in a later 6.18.x point release (making a downgrade unnecessary), would be genuinely appreciated.

------------------------------------------------------------

PART 5 — RELATED REPORTS WE FOUND

------------------------------------------------------------

- A near-identical case on Proxmox: GPF in list_lru_isolate_move() from kswapd0, also on a dual-socket NUMA host, leaving an orphaned list_lru spinlock and locking the whole machine — https://forum.proxmox.com/threads/kernel-7-0-14-9-pve-gpf-in-list_lru_isolate_move-kswapd0-leaves-an-orphaned-spinlock-whole-host-locks-up-dual-socket-numa-pbs-4-2-5.185582/

- An open Unraid 6.12.8 report of kernel NULL pointer dereferences correlated with ZFS operations (mutex_lock/arc_change_state/zfs_getattr_fast), where the reporter replaced RAM entirely and re-ran extensive memtest with no change — https://forums.unraid.net/bug-reports/stable-releases/consistent-kernel-errors-and-system-crashes-on-unraid-6128-with-zfs-operations-r2904/

- "[7.3.1] Kernel oops after zfs task exit" — https://product.unraid.net/p/7-3-1-kernel-oops-after-zfs-task-exit

------------------------------------------------------------

QUESTIONS FOR THE TEAM

------------------------------------------------------------

1. Is this a known issue with the list_lru/shadow-node shrinker path (and now XFS inode lookup too) on 6.18.x, particularly on multi-socket/NUMA hardware?

2. Is the ZFS-related oops reported on 7.3.1 the same underlying bug, or unrelated? We run ZFS pools too.

3. Is there a later 6.18.x point release, or an alternate kernel build, where this is known to be fixed?

4. Does downgrading a running array from 7.3.2 to 7.2.8 have any known gotchas (ZFS pool feature-flag compatibility, Docker/btrfs image compatibility, plugin compatibility) we should watch out for before attempting it?

Happy to provide full diagnostics, further log excerpts, or test a patched/alternate kernel if one is available.

Please post the diagnostics.

5 hours ago, JorgeB said:

Please post the diagnostics.

alpha-diagnostics-20260822-1351.zipUploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...

Here is the latest diagnostics report, pulled right before sending this message, along with the recent logs I've collected over the past few days :)

Logs - 08.zip
Sorry, it looks like it won't let me upload the logs as .txt files, so I've put the latest ones I have into a ZIP file.

The diagnostics confirm a genuine kernel oops in xfs_iget() while shfs was performing a filesystem lookup. However, shfs is only the process context in which the fault surfaced; this

does not currently establish an shfs or XFS-specific bug.

More importantly, the same boot contains 26 userspace segfaults and an invalid-opcode fault across several unrelated components, including Python, .NET, PHP, Node, smartctl, Radarr, and Sonarr. These faults occurred on 21 different logical CPUs and were split evenly across both CPU sockets. Some occurred before the xfs_iget() oops and others continued afterward.

There was plenty of available memory, no OOM event, and no accompanying XFS corruption, storage I/O, controller-reset, MCE, or EDAC error. The absence of MCE/ECC events does not completely rule out a marginal DIMM, CPU memory controller, motherboard, or power issue.

The separate log archive unfortunately does not contain the reported kswapd0/shadow_lru_isolate crash. Twelve of its thirteen daily files are empty, and the only populated file

contains routine activity rather than a crash.

At this point, it doesn't look to me like this is a 7.3.2 kernel regression. Hardware/platform instability is the leading possibility, although a broader kernel memory-management issue

remains possible.

My recommended next steps:

- Verify that remote syslog is actually receiving and retaining a test message. Use TCP instead of UDP if the receiver supports it.

- Export the iDRAC/Lifecycle Controller hardware and SEL logs.

- Run Dell’s extended CPU and memory diagnostics, followed by multiple complete passes of a boot-time memory test.

- If necessary, test matched RDIMM sets separately according to the T420 memory-population rules.

- Test Unraid Safe Mode with Docker and VM autostart disabled.

- Reboot promptly after any further kernel oops; avoid continuing write-heavy array, VM, or database activity on an already-corrupted boot.

Downgrading to 7.2.8 can be useful as a controlled A/B test, but stability on an older kernel alone would not prove a 7.3.2 bug because changing kernels also changes memory layout and timing. Ideally, hardware testing should be completed first, followed by a controlled 7.3.2 → 7.2.8 → 7.3.2 comparison with the same hardware, workload, and logging configuration.

Good morning, I was starting to go through the tests you suggested, and sorry in advance if some of this isn't phrased quite right, I'm not very experienced with this side of things. I'm working through it with the help of Claude (the AI assistant) reading and interpreting the logs for me, since on my own I wouldn't have known where to look.

It actually started throwing fresh errors while I was in the middle of getting ready to run your tests, so there's more to report than I expected sharing everything below, plus a fresh diagnostics zip (taken while some of this was still actively broken, so it should show the faulty state directly) and the two most relevant daily log files from our remote syslog archive.

---

alpha-diagnostics-20260824-1132.zip

Logs23-24.zip

Update had another incident overnight that materially changes the picture, so sharing the new data before deciding on next steps.

Turns out there wasn't just the one Oops from before going back through the full remote log archive, this same boot session had accumulated at least 12 kernel Oopses (the internal Oops counter read "[#12]" on the last one before the reboot), most of which we hadn't individually inspected until now. The most informative one we've captured so far:

kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000

kernel: Oops: Oops: 0000 [#12] SMP PTI

kernel: CPU: 9 UID: 0 PID: 224 Comm: khugepaged Tainted: P B D O 6.18.38-Unraid #1 PREEMPT(voluntary)

kernel: Tainted: [P]=PROPRIETARY_MODULE, [B]=BAD_PAGE, [D]=DIE, [O]=OOT_MODULE

kernel: RIP: 0010:mas_ascend+0x92/0x1c0

kernel: Call Trace:

kernel: mas_next_slot+0x13c/0x3e0

kernel: mas_find+0x44/0x110

kernel: khugepaged+0x199/0x600

kernel: kthread+0x1ce/0x1e0

kernel: ret_from_fork+0x24/0x170

khugepaged crashed inside the maple tree code (mas_ascend, the VMA-tracking data structure). The kernel tainted itself BAD_PAGE an explicit, direct signal that it found a corrupted page, not an inference on our part. Immediately following, as a nested fault with no new Oops header of its own:

kernel: RIP: 0010:xfs_iget+0x544/0x8f0

kernel: note: khugepaged[224] exited with irqs disabled

The exact same function and byte offset as the xfs_iget crash reported before. So that one wasn't a one-off; it's a recurring fault point, and this time it triggered as a secondary fault while the kernel was already unwinding from the khugepaged crash.

The system did not hard-lock this time and kept running for close to two hours afterward on an already BAD_PAGE-tainted kernel. That state cascaded: ~2 hours later shfs threw "fuse internal error: unable to unhash node" and the user-share mount died, Samba started failing client connections shortly after, and when we tried a normal array stop to recover, the stop sequence's own umount call died with the same "exited with irqs disabled" signature, leaving "sync" permanently stuck in D-state (unkillable).

Only a forced power cycle via BMC recovered it; a graceful shutdown was never going to complete given the stuck task.

So in one boot session we had three distinct core kernel subsystems each independently producing null-pointer faults: the shadow/LRU shrinker (shadow_lru_isolate), XFS inode lookup (xfs_iget, twice now), and VMA/maple tree management under khugepaged (mas_ascend), the last one with an explicit BAD_PAGE taint.

That's a lot of independent surface area for coincidence, and BAD_PAGE is about as close to a direct kernel statement of "a memory page got corrupted" as we're going to get from logs alone.

I'll fully grant that BAD_PAGE tells us a page was bad, not why consistent with either a kernel-side bug corrupting its own bookkeeping, or genuine hardware-level corruption that these memory-management structures happen to be sensitive enough to catch. It doesn't resolve which of those two it is, but it makes "several unrelated, coincidental single-app bugs" a much harder sell.

Still zero ECC/MCE events, before or after any of this, including right through the incident and the forced reboot. Array itself came back healthy 8 disks, all DISK_OK, 0 read/write errors, automatic post-unclean-shutdown parity check completed with only 1 correction (expected/normal for that kind of shutdown, not a red flag on its own).

On the syslog concern from before: confirmed it was a false alarm on my end the "empty" daily files are the current, not-yet-rotated file that logrotate resets to 0 bytes; the actual content lives in same-day files with a rotation date suffix. Found the original kswapd0/shadow_lru_isolate Oops there with its full trace intact.

Since then we've also moved the forwarder from UDP to TCP for reliability, confirmed via a live round-trip test with an established TCP connection to the receiver.

ADDITIONAL #1, discovered this morning after the reboot: Plex Media Server itself started segfaulting on every single launch attempt 100% reproducible, multiple consecutive identical crashes, each writing its own local crash report:

/home/nobody/start.sh: line 47: 119 Segmentation fault "${PLEX_MEDIA_SERVER_HOME}/Plex Media Server"

****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /config/Plex Media Server/Crash Reports/.../....dmp

The first sign was actually a segfault in "PMS GTP" (Plex's GPU transcode helper process) a few hours after the reboot; after that, the main Plex Media Server binary itself started dying on every start, reproducibly.

This matches a pattern we'd already seen with two other unrelated containers earlier in this whole saga (a persistently corrupted DLL in one, corrupted libssl in another) 100% reproducible crashes that a plain restart never fixes, only clearing after a full image re-pull, which strongly suggests bytes got corrupted on disk at some point rather than a transient in-memory glitch. Given the timing (a few hours after last night's BAD_PAGE-tainted kernel Oops), it's hard not to read this as another downstream casualty of the same event. Still working on getting it re-pulled cleanly.

ADDITIONAL #2, also this morning: the Unraid API service itself (the Node.js/GraphQL backend behind the modern WebGUI dashboard stats, live CPU graphs, etc.) had also silently died.

Its PM2-tracked PID no longer corresponded to any running process, and the dashboard had stopped updating CPU usage entirely. A plain service restart (rc.unraid-api restart) brought it back cleanly with no errors on the fresh start, so no persistent corruption there as far as we can tell but it's one more service that quietly dropped out sometime after the reboot, on top of Plex and the two other containers that needed manual restarts after the array came back (they'd been orphaned mid-crash when Docker itself got stopped during the failed array-stop sequence).

So the blast radius of last night's event, directly or indirectly, touched: the kernel three times over, the array/share layer, Samba, at least four Docker containers, and now the WebGUI's own backend service.

Attaching: a fresh diagnostics zip (captured while Plex and the Unraid API were still actively broken, so it should reflect the faulty state rather than the already-recovered one), plus the two daily log files from our remote syslog archive covering yesterday and last night/this morning (2026-08-23.log and 2026-08-24.log) these have the full timeline including everything quoted above with exact timestamps.

Given all of this, we're moving up the timeline on the 7.3.2 -> 7.2.8 -> 7.3.2 controlled comparison you suggested, and will also look at Dell's extended diagnostics and further memtest passes as we get a maintenance window.

Thank you :)

Thanks, the new logs are very useful. They confirm that this is broader memory-management corruption rather than an isolated shfs, XFS, BTRFS, or Plex problem.

Before the later crashes, the kernel reports bad RSS counters, non-zero page-table accounting, and BTRFS pages that are still mapped while being deleted. Oopses #2 through #12 then fault at the same location in mas_ascend(), with the repeated state strongly suggesting that lsof, another task, and later khugepaged are encountering the same damaged process address-space tree. The repeated trailing xfs_iget() register block matches the original August 20 oops and does not appear to be a new XFS crash each time.

The fault while unmounting /var/lib/docker occurs after the system has accumulated all this corruption, so it is more likely a secondary failure than a separate BTRFS unmount bug.

After the reboot, unrelated .NET and dynamic-loader processes start segfaulting again within approximately 50 minutes and on both CPU sockets. The later Plex message requesting about 94.5 TB is not genuine memory exhaustion, the diagnostics still show approximately 77.5 GB available, but likely another indication of a corrupted process state.

This suggests a real kernel/MM corruption issue, but it still does not establish whether the underlying cause is the 7.3.2 kernel, an optional/out-of-tree module, or hardware/ platform instability. There are still no recorded MCE, EDAC, PCIe AER, OOM, storage-reset, or filesystem-corruption errors.

The latest diagnostics show another normal-mode 7.3.2 boot, so the most useful next test would be:

1. Boot in Safe Mode with Docker and VM autostart disabled.

2. If that remains stable, enable Docker while still in Safe Mode and start the workload/container groups incrementally.

3. Also provide the results of Dell’s extended hardware/memory diagnostics, multiple complete memory-test passes, and the iDRAC/Lifecycle Controller SEL.

4. If those tests remain clean, compare 7.3.2 with 7.2.8 while keeping the BIOS settings, memory population and workload unchanged. Ideally, return to 7.3.2 afterward to confirm whether the corruption recurs.

Keep remote syslog enabled. After the first new kernel oops or bad-page/RSS warning, preserve the log and reboot rather than continuing workloads on the already-corrupted kernel.

3 hours ago, JorgeB said:

Thanks, the new logs are very useful. They confirm that this is broader memory-management corruption rather than an isolated shfs, XFS, BTRFS, or Plex problem.

Before the later crashes, the kernel reports bad RSS counters, non-zero page-table accounting, and BTRFS pages that are still mapped while being deleted. Oopses #2 through #12 then fault at the same location in mas_ascend(), with the repeated state strongly suggesting that lsof, another task, and later khugepaged are encountering the same damaged process address-space tree. The repeated trailing xfs_iget() register block matches the original August 20 oops and does not appear to be a new XFS crash each time.

The fault while unmounting /var/lib/docker occurs after the system has accumulated all this corruption, so it is more likely a secondary failure than a separate BTRFS unmount bug.

After the reboot, unrelated .NET and dynamic-loader processes start segfaulting again within approximately 50 minutes and on both CPU sockets. The later Plex message requesting about 94.5 TB is not genuine memory exhaustion, the diagnostics still show approximately 77.5 GB available, but likely another indication of a corrupted process state.

This suggests a real kernel/MM corruption issue, but it still does not establish whether the underlying cause is the 7.3.2 kernel, an optional/out-of-tree module, or hardware/ platform instability. There are still no recorded MCE, EDAC, PCIe AER, OOM, storage-reset, or filesystem-corruption errors.

The latest diagnostics show another normal-mode 7.3.2 boot, so the most useful next test would be:

1. Boot in Safe Mode with Docker and VM autostart disabled.

2. If that remains stable, enable Docker while still in Safe Mode and start the workload/container groups incrementally.

3. Also provide the results of Dell’s extended hardware/memory diagnostics, multiple complete memory-test passes, and the iDRAC/Lifecycle Controller SEL.

4. If those tests remain clean, compare 7.3.2 with 7.2.8 while keeping the BIOS settings, memory population and workload unchanged. Ideally, return to 7.3.2 afterward to confirm whether the corruption recurs.

Keep remote syslog enabled. After the first new kernel oops or bad-page/RSS warning, preserve the log and reboot rather than continuing workloads on the already-corrupted kernel.

Good afternoon! Understood, I'll get started on those tests and keep you updated.

In a couple of hours once I finish work, I'll boot it up in Safe Mode with Docker disabled (VMs are already off since I don't really use them). I'll leave it running like that overnight to see how it holds up, and tomorrow morning I'll start enabling the Dockers little by little.

Thanks a lot for the help! :)

Quick update from this morning.

The Safe Mode test survived overnight cleanly 13 hours, zero kernel Oopses of any of the three types we'd been tracking.

One segfault did occur, though, and it's telling: it was Unraid's own built-in "smartctl_type" SMART-monitoring helper, part of the base system, not a Docker container or a plugin. With everything else stripped away, something in the base OS still crashed once.

Separately, I made a mistake worth flagging: I didn't realize Safe Mode skips our custom boot script, so the sb_edac module (our ECC monitor) wasn't loaded for the entire overnight run. We had zero memory-error visibility during the exact window we most wanted it. Reloaded it now; keeping Safe Mode running for the rest of today with it active this time.

Also ran into something more significant this morning: the WebGUI became completely unusable every PHP page just hung indefinitely rather than erroring.

Root cause: php-fpm itself was broken at the binary level, throwing "undefined symbol" errors on startup (its master process was spinning with zero worker processes spawned). php-fpm lives in the read-only squashfs image that gets cached into RAM at boot the same image involved in the two squashfs decompression-error incidents we mentioned earlier in this thread.

Dropping the page cache (echo 3 > /proc/sys/vm/drop_caches) to force a clean re-read from the physical USB fixed it immediately no reinstall, no file changes, just forcing RAM to re-fetch the same bytes from disk.

That's about as direct a confirmation as we're likely to get that pages are getting corrupted in RAM specifically, since the exact same on-disk bytes came back clean on a forced re-read.

Separately (and not fixed the same way): the Unraid API service (dynamix.my.servers backend, powers the modern dashboard's live stats) is also throwing the identical "SyntaxError: Invalid or unexpected token" in one of its own node_modules files, but it lives in the writable overlay rather than the read-only squashfs, so the cache-drop trick didn't touch it. Left it stopped for now rather than let it crash-loop; will reinstall the plugin once we're done with Safe Mode testing.

Plan for today: staying in Safe Mode with Docker and VMs off for the rest of the day, now with EDAC properly active. If it stays clean, we'll start bringing Docker containers back up in small batches tomorrow rather than all at once, watching the log between each batch.

php-fpm-corruption-addendum.txt

alpha-safemode-diagnostics-20260825-1138.zip

2026-08-24.log

2026-08-25.zip

Sorry I didn't attach it in my previous message, the ZIP file was still being created. It wasn't letting me upload today's log because it's massive, so I had to compress it.

The new log confirms that sb_edac registered at 09:57:35, after both the smartctl_type fault at 01:39:58 and the PHP-FPM failures beginning at 09:49:59. Therefore, neither failure occurred while EDAC reporting was active, and the absence of EDAC errors during the overnight period cannot be used to exclude a hardware or memory-path problem.

I also would not describe the 13-hour Safe Mode run as completely clean. It is correct that none of the previously tracked kernel oopses occurred, but the smartctl_type segfault attempted to execute address zero. That is consistent with the wider corruption pattern and is significant because it happened with Docker disabled, third-party plugins skipped, and the NVIDIA modules absent. This shows that those components are not required to trigger the issue, but it does not necessarily establish a defect in the base OS—the base utility may simply have encountered already-corrupted state.

The PHP-FPM recovery after dropping caches is also important, but I do not think it proves specifically that RAM pages were corrupted. Dropping caches can evict the PHP executable, the dynamic loader, shared libraries, and other file-backed pages. Without checksums or copies from before and after the cache drop, we cannot confirm that identical bytes were read from the USB or identify which object was corrupt. The earlier SquashFS decompression errors make the boot image, USB/read path, and decompression path important possibilities alongside RAM, CPU/IMC, motherboard, and kernel memory-management corruption. If possible, please supply the complete earlier SquashFS errors and their surrounding log windows.

Regarding the API error, drop_caches is not limited to read-only SquashFS files; it can also evict eligible clean pages belonging to writable filesystems. The continued API failure could mean the file is persistently corrupt, the process retained an old mapping, or a different dependency is affected. Safe Mode also reports that API plugin discovery was skipped, while the failing file is under the core /usr/ local/unraid-api tree, so I would not yet assume that reinstalling the Connect plugin will correct it.

Before reinstalling anything, please capture:

findmnt -T /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js

sha256sum /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js

file /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js

xxd -l 128 /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js

If PHP-FPM fails again, please capture its filesystem, checksum, and ldd output before dropping caches.

Since the base runtime has already shown corruption and the API remains damaged, I would not start Docker containers on this same boot. I recommend preserving the evidence, rebooting cleanly, and then continuing the Safe Mode test with sb_edac loaded. After that, the highest-value comparisons would be:

- Safe Mode with the array stopped.

- Dell extended CPU and memory diagnostics plus multiple full memory-test passes.

- The same Unraid 7.3.2 release from known-good, freshly prepared boot media.

- If those tests remain clean, a controlled 7.3.2 → 7.2.8 → 7.3.2 comparison.

You're right on every point here, and I appreciate the correction. Addressing each one, plus the data you asked for, plus an update on where we are right now.

EDAC TIMING

Confirmed exactly. sb_edac registered at 09:57:35. The smartctl_type fault was at 01:39:58 and the php-fpm collapse started at 09:49:59 (syslog shows php-fpm's pool www children exiting with code 127 starting at that exact second, hundreds of them in the same burst).

Both predate EDAC coming up by hours/minutes respectively. You're correct: neither failure has any ECC visibility, and that overnight run can't be used to exclude a hardware/memory-path cause. That was a real mistake on my end, not a rounding error — I should have checked the module load time against the fault times before writing that up.

"NOT COMPLETELY CLEAN"

Agreed. smartctl_type executing address zero with Docker off, plugins skipped, and NVIDIA modules absent is a real, notable data point, not a null result. I shouldn't have flattened it into "0 Oops = clean." Also agreed it doesn't establish a base-OS defect on its own — could just be the utility hitting already-corrupted state rather than corrupting anything itself.

PHP-FPM "PROOF"

Agreed, dropping caches and having it come back clean does not prove RAM-page corruption specifically. I overstated that. It's equally consistent with the read path, decompression path, or boot media, as you said.

THE TWO EARLIER SQUASHFS INCIDENTS, IN FULL

Incident 1 (2026-08-13, ~03:05:40):

2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression error: 20

2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression failed, data probably corrupt

2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: Failed to read block 0x3374df0: -5

2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression error: 20

2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: zstd decompression failed, data probably corrupt

2026-08-13T03:05:40+02:00 Alpha kernel: SQUASHFS error: Failed to read block 0x3374df0: -5

Surrounding window: preceded by disks spinning down at 02:53-03:05 and a .NET segfault at 02:48:52. Followed by normal SMART reads and the scheduled mover run at 03:40:01 with no further errors. Same block (0x3374df0) failed identically twice in the same second.

Incident 2 (2026-08-14, two bursts at 08:36:37 and 08:36:50):

2026-08-14T08:36:37+02:00 Alpha kernel: SQUASHFS error: zstd decompression error: 20

2026-08-14T08:36:37+02:00 Alpha kernel: SQUASHFS error: zstd decompression failed, data probably corrupt

2026-08-14T08:36:37+02:00 Alpha kernel: SQUASHFS error: Failed to read block 0x206e1fb2: -5

(repeated 4x at 08:36:37, then 3x more at 08:36:50, same block 0x206e1fb2 every time)

Surrounding window: preceded by a smartctl_type segfault at 05:27:21 and a celeryd/libpython segfault at 07:20:00 earlier that morning. Nothing unusual immediately after — normal SSH session, no further errors in that window. Both incidents self-cleared on the next read without manual intervention at the time; this was noted days ago as "came back clean on retry," before we started saving raw excerpts, which is why we couldn't cite the original lines until now — this is what we had in the archive.

FORENSICS ON THE UNRAID-API FILE, AS REQUESTED

findmnt -T /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js

TARGET SOURCE FSTYPE OPTIONS

/usr /dev/loop0 squashfs ro,relatime,errors=continue

/usr overlay overlay rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr,uuid=on

sha256sum:

d66f16d9ba336f833ed6e19fc46388ea01ff8f27d33292874ea8f6b6d45b0c15 file-validator.interface.js

file:

file-validator.interface.js: JavaScript source, ASCII text

xxd wasn't available on this box (busybox/minimal userland), so I used od -A x -t x1z -v instead. First 128 bytes:

000000 22 75 73 65 20 73 74 72 69 63 74 22 3b 0a 4f 62 >"use strict";.Ob<

000010 6a 65 63 74 2e 64 65 66 69 6e 65 50 72 6f 70 65 >ject.definePrope<

000020 72 74 79 28 65 78 70 6f 72 74 73 2c 20 22 5f 5f >rty(exports, "__<

000030 65 73 4d 6f 64 75 6c 65 22 2c 20 7b 20 76 61 6c >esModule", { val<

000040 75 65 3a 20 74 72 75 65 20 7d 29 3b 0a 65 78 70 >ue: true });.exp<

000050 6f 72 74 73 2e 46 69 6c 65 56 61 6c 69 64 61 74 >orts.FileValidat<

000060 6f 72 20 3d 20 76 6f 69 64 20 30 3b 0a 2f 2a 2a >or = void 0;./**<

000070 0a 20 2a 20 49 6e 74 65 72 66 61 63 65 20 64 65 >. * Interface de<

000080 73 63 72 69 62 69 6e 67 20 46 69 6c 65 56 61 6c >scribing FileVal<

000090 69 64 61 74 6f 72 73 2c 20 77 68 69 63 68 20 63 >idators, which c<

That's clean, well-formed CommonJS nothing visibly garbled.

So I went a step further and tried loading it directly with a plain, uninstrumented require() outside of the unraid-api process. It loaded without error.

So right now, on disk, this specific file is not corrupted by any check available to me. That undercuts my "corrupted file" read of the API failure specifically.

The original failing require() went through require-in-the-middle (an instrumentation/hook library the API's tracing setup uses to patch Node's module loader) rather than a plain require.

Given the file itself checks out clean now, my best guess is either that hook doing something to the module at load time, or a transient read at the moment it was first loaded that's since cleared the same way the two SquashFS incidents did not persistent on-disk corruption of this particular file.

I don't have a "before" checksum to compare against, so I can't fully close this out either way.

I also don't have before/after evidence for the php-fpm binary — the cache-drop fix was already applied before I saw your evidence-preservation request, so there's no clean "before" state left to check now. If it recurs, I'll capture filesystem/checksum/ldd first, before touching anything.

WHERE WE ARE RIGHT NOW

Rebooted cleanly and back in Safe Mode. This time sb_edac was loaded manually 3 minutes after boot (14:18:02 boot, 14:21:05 EDAC up) rather than 8+ hours in like last time still not from second zero, since Safe Mode skips our persistent boot script entirely by design, so it has to be loaded by hand every time we boot this way.

Also, since the array happened to still be stopped after this boot, we're deliberately leaving it that way for a while rather than starting it — which covers your "Safe Mode with the array stopped" suggestion directly, as its own distinct phase, before we move on to Dell diagnostics and memtest.

ATTACHMENTS

Attaching four full daily log files from the remote syslog archive so you can check the surrounding context yourself rather than just my excerpts: the two covering the original SquashFS incidents (2026-08-13 and 2026-08-14), plus the two from the more recent overnight/reboot incident (2026-08-23 and 2026-08-24). Also attaching a fresh diagnostics zip taken just now, in the current state (Safe Mode, array stopped, sb_edac loaded).

alpha-safemode-diagnostics-20260825-1523.zip2026-08-25.zip

alpha_2026-08-13.logalpha_2026-08-14.log

The new files confirm both SquashFS incidents exactly and add one useful EDAC distinction.

The August 13 incident contains two identical failures against block 0x3374df0. The August 14 incident contains seven failures against a different block, 0x206e1fb2, across two bursts 13 seconds apart. Neither daily log contains an accompanying USB reset, lower-level block-I/O error, OOM event, or kernel oops. However, both incidents are surrounded by unrelated process faults, so they remain consistent with the wider corruption pattern.

The August 13 SquashFS incident occurred before that evening’s reboot and before sb_edac loaded. The August 14 incident occurred on the following boot after sb_edac registered both memory controllers at 20:27:31, with no intervening reboot or unload visible. No CE/UE was reported. That means at least the second SquashFS incident appears to have occurred with EDAC visibility. This makes an ordinary reportable DIMM-cell error less likely, but it does not exclude CPU cache, IMC, motherboard, DMA, power, or boot/read-path corruption.

The current diagnostics confirm that the new test is configured as intended:

- Safe Mode

- Array and filesystems stopped

- Docker disabled

- VM service disabled

- Only the stock OS plugin

- sb_edac loaded at 14:21, three minutes after boot

There are no new faults through the 15:23 diagnostic, but that is only approximately 65 minutes, continue this exact configuration beyond the previous 4-hour-38-minute first-fault interval and preferably beyond 13 hours before changing anything else. The API evidence now looks transient. The current file contents look valid, plain require() succeeds, and the post-reboot API log no longer contains the syntax error. The earlier require-in-the-middle stack frame only shows that it wrapped Node’s module-loading path; it does not currently establish that the hook modified or corrupted the source.

Also, the findmnt output shows that /usr is presented through an overlay with a SquashFS lower layer, but it does not prove that this specific file came from the writable upperdir. You can check that directly with:

test -e /var/local/overlay/usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js \

&& echo "present in upperdir" \

|| echo "not present in upperdir"

Before replacing the boot device, and preferably after the current observation phase so we do not alter this test, calculate the actual hashes of the boot archives and compare them with their supplied hash values:

sha256sum /boot/bzimage /boot/bzroot /boot/bzroot-gui /boot/bzmodules /boot/bzfirmware

The four newest files received here were the August 13 and 14 logs, the current diagnostics, and an updated August 25 log. We already had and reviewed the August 23/24 evidence from the previous upload.

For now, continuing the array-stopped Safe Mode test is the highest-value next step. After that, I would proceed with repeated boot-archive verification, known-good boot media using the same 7.3.2 release, and then the Dell hardware diagnostics and extended memory testing.

Update from the Safe Mode + array-stopped test. Ran for 20 hours 31 minutes before I checked in well past both the 13-hour target and the previous 4h38m first-fault interval.

ZERO KERNEL OOPSES

None of the three tracked fault types (shadow_lru_isolate, xfs_iget, mas_ascend) occurred in over 20 hours with the array stopped and Docker off.

FOUR MORE SMARTCTL_TYPE SEGFAULTS

17:08:01, 18:39:31, and 20:08:29 yesterday, then 01:40:59 this morning same recurring crash as before, three in php, one in the dynamic linker (ld-linux-x86-64-2.43.so) this time instead of php itself.

THE SIGNIFICANT ONE: A LIBC SEGFAULT IN ELOGIND-DAEMON

At 10:48:38 this morning, elogind-daemon (PID 2564, the persistent core session-management daemon, not a periodic script) segfaulted inside libc-2.43.so itself. In the same second, an incoming SSH login recorded: "pam_elogind(sshd:session): Failed to create session: Message recipient disconnected from message bus without replying" — the daemon died mid-handshake of an entirely routine session open.

It self-healed in 8 seconds: a fresh elogind-daemon (new PID) came up, re-registered the seat, and normal session handling resumed immediately after.

This is a segfault in the single most fundamental, most heavily linked shared library on the system, inside an always-running core daemon, with Docker, VMs, plugins, and the array all fully stopped. About as little as we can strip away from this box while it's still bootable.

EDAC COVERAGE THIS TIME

sb_edac was loaded at 14:21:05 yesterday, about 20 hours before this crash. So unlike the smartctl_type/php-fpm incidents from the first Safe Mode run, this one has full ECC visibility. Checked immediately after: 0/0/0/0 across both memory controllers, same as always.

ONE MORE THING WE FOUND WHILE PULLING LOGS

Going back through yesterday's full daily log to grab it for you, we found that the pool www child-respawn storm from the php-fpm corruption (Aug 25, starting 09:49:59, before the reboot) ran long enough and hard enough to fill Alpha's local /var/log (tmpfs) completely — over 4.1 million lines, ~594MB, in a single day. rsyslogd itself started throwing "No space left on device" write errors on the local syslog at 14:14:09, right as we were shutting down for the clean reboot. Remote logging to the archive kept working throughout since that's a separate destination, so nothing was lost, but worth flagging as its own side effect of that incident — the same disk-full-tmpfs failure mode we hit once earlier in this whole investigation, this time self-inflicted by the crash-loop's own log volume.

ATTACHMENTS

Two files this time, kept small on purpose: the pool www spam block from yesterday morning is redundant with what we already reported and was millions of lines, so I trimmed it out. Attaching only the post-reboot portion of yesterday's log (from the clean reboot at 14:14 onward, ~2,760 lines) plus today's full log so far, covering the elogind/libc crash. Diagnostics zip to follow separately.

alpha_2026-08-26.logalpha_2026-08-25_postreboot.logalpha-safemode-diagnostics-20260826-1110.zip

Array and Docker remain stopped. Will keep this configuration running further unless you'd like us to move on to the boot-archive hash verification next.

The diagnostics confirm that the array remained stopped, Docker and VM services were disabled, Safe Mode was active, and no third-party plugins or NVIDIA modules were loaded.

Although there were no further shadow_lru_isolate, xfs_iget, or mas_ascend kernel oopses, this was not a clean run. The four additional smartctl_type segfaults and the independent elogind-daemon segfault show that the broader userspace corruption continues without the array, shfs, Docker, VMs, or plugins. Those components can therefore be ruled out as necessary causes of the userspace failures.

One qualification: the elogind instruction pointer being inside libc tells us where the fault surfaced, but does not by itself prove that the libc file was corrupted. It could also result from a bad pointer passed into libc or damaged process state. It is still significant because this is a separate, persistent base-system daemon rather than another smartctl_type invocation.

sb_edac was active before all five faults and no ECC errors were logged. That weakens the likelihood of a conventional reportable DIMM error, but does not eliminate CPU cache, memory-controller, motherboard, firmware, power, DMA, or other platform faults that EDAC cannot detect.

The lack of kernel oopses may simply mean that leaving the array stopped removed the XFS, BTRFS and Maple Tree activity that previously exposed the corrupted state. At this point I still cannot distinguish platform corruption from a stock kernel, root-filesystem, boot-image or boot-media/read-path problem, but IMO there is not yet enough evidence to classify this as an Unraid 7.3.2 kernel bug.

The next steps I would recommend are:

1. Run Dell’s extended CPU and memory diagnostics and multiple complete boot-time memory-test passes using conservative BIOS defaults.

2. Repeatedly calculate the hashes of the actual boot archives and compare them with their expected hashes.

3. Repeat the same array-stopped Safe Mode test on 7.3.2 using freshly prepared, known-good boot media.

4. If those tests are clean, perform the identical stripped test on 7.2.8 beyond the observed fault window, followed by a return to 7.3.2 to confirm whether the failures recur.

The /var/log exhaustion in the earlier boot is most likely a consequence of the PHP-FPM respawn storm, not the cause of the failures in this new boot.

Sounds good, starting with these.

Reordering slightly for efficiency (nothing dropped, just resequencing) hashes captured now, then preparing fresh 7.3.2 boot media in parallel (doesn't require touching Alpha), then repeating the array-stopped Safe Mode test on that fresh media before committing to the Dell diagnostics + memtest window, since that one costs the most downtime and is worth running after the cheaper tests narrow things down. 7.2.8 comparison last, as you suggested. Will report back as each stage finishes.

Thank you a lot :)

Ran the exact same array-stopped Safe Mode test on the freshly created 7.3.2 media (identical boot-file hashes to both our original stick and the official release, verified beforehand).

Result: reproduced the same failure signature. In just under 20 hours: zero kernel oopses, but two smartctl_type segfaults in php, same pattern as before.

This weighs against the boot media itself being the source of corruption, since this stick was written fresh and verified byte-identical before the test started.

prueba_2026-08-26.logprueba_2026-08-27.logidrac_power.jsonidrac_sel.json

prueba-safemode-diagnostics-20260827-1555.zip

Also checked something we hadn't looked at directly yet: iDRAC power supply telemetry and the full SEL.

PSU1 (the one actually powering the system) reports normal voltage and OK health throughout;

PSU2 shows the same long-known "input lost" condition we'd already identified as unrelated.

Total system power draw is a steady ~120W.

The SEL itself has zero new entries of any kind since December 2025 nothing logged during this entire investigation, including through the physical USB swap.

One more note, expected rather than new: the WebGUI on this fresh install doesn't show live CPU/memory stats, and the Unraid API service is offline.

Confirmed this is the same Safe-Mode-disables-the-Connect-plugin behavior we already identified on the original stick, not a repeat of the php-fpm corruption incident php-fpm itself is healthy here (workers spawned, classic GUI pages load normally), it's specifically the API-backed dashboard widgets that are unavailable by design in Safe Mode.

Starting the extended Dell diagnostics and memtest passes now rather than scheduling for later have someone watching the console for this one. Will report back once those finish.

The fresh-media test is useful and confirms that the original flash drive and its configuration are not required for the problem to occur.

The logs actually contain three userspace failures:

- At 06:31, the PM2 supervisor for the core Unraid API trapped inside Node.

- At 07:37, smartctl_type segfaulted inside PHP at offset 0x6310c8.

- At 11:18, smartctl_type segfaulted at address zero inside PHP at offset 0x666303.

The last PHP offset is particularly notable because it exactly matches two faults from the previous array-stopped Safe Mode test, including occurrences across both CPU sockets.

The missing CPU and memory dashboard statistics are not expected Safe Mode behavior in this instance. The API log shows that the core API initially started successfully; Safe Mode only skipped plugin discovery. The PM2/Node failure appears to explain why the API was subsequently unavailable.

sb_edac was active for more than 12 hours before these failures, with no corrected or uncorrected ECC events. There are also no MCE, PCIe AER, SquashFS, storage, or OOM errors in this boot.

The iDRAC data suggests that PSU1 is healthy, and the voltage sensors are normal. PSU2 remains offline and redundancy is disabled, as already known. The SEL does contain PSU2 and chassis events through August 6, 2026, rather than December 2025, but there are no entries during this latest test and no corresponding CPU or memory errors.

Recommend completing the Dell extended diagnostics and the planned full memory-test passes. If those tests pass, the next useful discriminator would be to repeat this exact fresh-media, Safe Mode, array-stopped test on Unraid 7.2.8 for longer than the observed fault window. Afterward, return to fresh 7.3.2 and confirm whether the failures recur.

If the API fails again during another test, also preserve /var/log/.pm2/ before rebooting, as those logs are not included in the standard diagnostics and may contain more information about the PM2/Node failure.

Results from the extended Dell diagnostics and the memtest86+ run, plus something we should have surfaced earlier: an OpenZFS pool feature upgrade done before this thread started that may be relevant.

DELL EXTENDED DIAGNOSTICS (Thorough Test Mode, all devices)

First pass (standard, not thorough) came back "Fail" only because of the event-log-content check — actual hardware results were Pass=267, Warning=65, Fail=0, Critical=67, all 67 "Critical" being the already-known PSU2/chassis-intrusion history.

Re-ran with Thorough Test Mode enabled across all devices, including the Processor and Memory categories and full DST Long Tests on all six array disks:

• result was a clean "All tests passed", including the disk that took the longest (3h25m). System Health throughout: fans steady ~2640-2760 RPM, CPU1/CPU2 in the mid-30s C under load, PSU1 normal voltage (232-240V), system power draw 98-112W.

MEMTEST86+ (all cores, all 95.9GB)

• 25h31m elapsed, 5 complete passes, 0 errors. Screenshot attached. CPU stayed in the mid-30s to mid-50s C throughout, no thermal concerns.

ZFS HISTORY WE SHOULD HAVE MENTIONED SOONER

Before this thread started, we'd already made two changes trying to address the instability, both ZFS-related on the two pools (cache, seedeo, both raidz/mirror on top of the array):

1. 2026-07-28: ran zpool upgrade (feature flag upgrade) on both pools, followed by a full scrub (0 errors on both). This was prompted by an unrelated product-board report describing a similar Oops pattern tied to a pending ZFS feature upgrade. First night after was clean (~9.5h, no oops, previously-known segfaults temporarily absent), but the same failure pattern resumed about a week later (2026-08-04), so this wasn't the root cause, or at least not the only one.

2. 2026-08-06 17:08:59: raised zfs_arc_max to ~19.2GB (currently still set: 20615843020 bytes, confirmed just now). The same evening, two new segfault types appeared that hadn't been seen before — one in a node process, one in emhttpd against libc. Memory pressure was checked and ruled out at the time (94GiB total, 73GiB available), but whether the ARC size change itself was a factor was never confirmed either way.

Worth flagging: ZFS's ARC registers its own shrinker with the kernel's memory-reclaim subsystem — the same list_lru/shrinker path as the shadow_lru_isolate oops we reported early in this thread. We hadn't connected those two things until now. All of our more recent array-stopped Safe Mode observations (which is where essentially all of our strongest evidence has come from) happened with these pools unmounted, so they can't be a confound for those specific results — but the ARC/shrinker connection is a loose thread from earlier in the investigation that we don't think has been ruled out.

Checked just now, pools are healthy: both ONLINE, 0 errors, scrub timestamps match the 2026-07-28 event exactly.

Current active feature set on cache includes several from the OpenZFS 2.2+/2.3+ era: log_spacemap, zilsaxattr, head_errlog, block_cloning, vdev_zaps_v2, block_cloning_endian (seedeo has a similar but slightly smaller set — no zilsaxattr or block_cloning, presumably because those datasets never triggered them). Running OpenZFS 2.4.3-1.

The practical concern: the feature upgrade is irreversible, and 7.2.8 ships an older OpenZFS than 7.3.x. It's likely 7.2.8 will refuse to import these pools given the newer active features.

For the planned array-stopped Safe Mode comparison test this shouldn't matter, since the pools won't be touched. If we later want to test with the array/pools active on 7.2.8, we're expecting a possible import failure there and will treat that as a version-compatibility issue rather than new evidence, not a surprise.

Given Dell diagnostics and memtest are both clean so far, planning to proceed with the 7.2.8 array-stopped Safe Mode comparison next, same methodology as the fresh-7.3.2-media test.

These results are useful. The screenshots confirm that Dell’s all-device Thorough Test completed successfully and that Memtest86+ ran for 25 hours 31 minutes, completing five passes over all 95.9 GiB with zero errors.

This materially reduces the likelihood of an ordinary reproducible CPU, DIMM, disk, thermal, or monitored platform failure, although it cannot completely exclude intermittent CPU-cache, IMC, motherboard, power-delivery, DMA, or firmware issues.

The ZFS history is worth recording, but it does not currently establish ZFS as the cause. The pool feature upgrade was followed by only a short clean interval before the same failures returned, while the ARC change has a timing correlation but no accompanying memory-pressure evidence.

More importantly, the fresh-media 7.3.2 test reproduced independent Node and PHP faults with the array and pools stopped. Therefore, the upgraded on-disk pool features and active pool operations are not required for the broader fault pattern. The zfs and spl modules were still loaded, so a generic ZFS module interaction remains possible. However, shadow_lru_isolate is part of the kernel workingset shadow-LRU path; the fact that ZFS ARC also registers a shrinker does not by itself show that the fault occurred inside an ARC callback. The complete original trace is still needed to establish that connection.

Please proceed with the 7.2.8 array-stopped Safe Mode comparison. Keep the array and pools stopped and do not attempt to import the upgraded pools. Match the previous test conditions as closely as possible, including Docker and VM services disabled, early sb_edac loading, remote logging, and verified boot archives. Please also record whether zfs and spl are loaded.

Run it beyond the known approximately 20.5-hour failure window—preferably at least 48 hours—then return to fresh 7.3.2 under the same conditions:

- If 7.2.8 also faults, that weighs against a 7.3-specific regression.

- If 7.2.8 remains clean and 7.3.2 faults again, that would provide substantially stronger evidence of version dependence.

I suspect the pool will still work with 7.21, but if 7.2.8 refuses to import the upgraded pools, treat that only as an OpenZFS feature-compatibility result, not as evidence from this crash comparison.

Hi again, still running Unraid on 7.2.8 on other flash drive I used for the 7.3.2 test, this time unlicensed since it isn't offering me the free trial option anymore.

Still have the original USB completely untouched with the original OS on it, for what it's worth.

After all these tests, honestly not sure what else to try beyond what you've already suggested.

On the ARC/shrinker point: fair correction, and I was overreaching. Re-checked the complete original trace, and it does not touch anything ZFS-specific it's the generic workingset shadow-LRU path throughout, no ARC callback frame anywhere in it. Full trace for reference:

kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000

kernel: #PF: supervisor read access in kernel mode

kernel: #PF: error_code(0x0000) - not-present page

kernel: Oops: Oops: 0000 [#1] SMP PTI

kernel: CPU: 24 UID: 0 PID: 352 Comm: kswapd0 Tainted: P O 6.18.38-Unraid #1 PREEMPT(voluntary)

kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE

kernel: RIP: 0010:_raw_spin_trylock+0x13/0x50

kernel: Call Trace:

kernel: shadow_lru_isolate+0x1f/0x150

kernel: __list_lru_walk_one+0xb8/0x140

kernel: list_lru_walk_one_irq+0xc/0x20

kernel: do_shrink_slab+0x199/0x2d0

kernel: shrink_slab+0x16e/0x2a0

kernel: shrink_node+0x515/0x7a0

kernel: balance_pgdat+0x456/0x670

kernel: kswapd+0x292/0x2e0

kernel: kthread+0x1ce/0x1e0

kernel: ret_from_fork+0x24/0x170

kernel: note: kswapd0[352] exited with irqs disabled

kernel: note: kswapd0[352] exited with preempt_count 2

Agreed the zfs/spl modules being loaded without an active pool proves nothing either way — recording it as requested, not claiming it as evidence.

7.2.8 ARRAY-STOPPED SAFE MODE TEST — IN PROGRESS

Matching the previous test conditions: array and pools stopped (not importing the upgraded pools, per your note), Docker and VM services disabled, sb_edac loaded 3 minutes after boot (earliest yet), remote logging confirmed working throughout.

At 17h11m so far: zero kernel oopses, but four smartctl/smartctl_type segfaults:

kernel: smartctl_type[129171]: segfault at 0 ip 0000000000000000 sp 00007ffff7b96708 error 14

kernel: smartctl[631166]: segfault at bfa60 ip 00000000000bfa60 sp 00007ffc58a65228 error 14

kernel: smartctl_type[1003123]: segfault at c ip 00000000009be224 sp 00007ffc4afa9dc0 error 4 in php[5be224,600000+48d000]

kernel: smartctl_type[1476102]: segfault at 0 ip 0000000000a66303 sp 00007ffd5e931128 error 4 in php[666303,600000+48d000]

The second entry (fault address bfa60, ip bfa60, error 14) is byte-for-byte identical to a fault we saw on the original 7.3.2 stick. The fourth (offset 0x666303 in php) is the same offset you already flagged as matching between two earlier 7.3.2 occurrences — now a third occurrence, this time under 7.2.8's kernel (6.12.98) instead of 7.3.2's (6.18.38).

Continuing to run this past the 20.5h mark, targeting at least 48h as you suggested, before moving to the fresh-7.3.2 retest under matching conditions.

prueba728_2026-08-29.logprueba-7-2-8-safemode-diagnostics-20260829-2106.zip

Thank you :)

Edited August 29Aug 29 by Thiago Alexander

At this point, the results no longer support this being an Unraid 7.3 regression or something that can be resolved by downgrading.

The stripped test has now reproduced on both Unraid 7.3.2 with Linux 6.18.38 and Unraid 7.2.8 with Linux 6.12.98

In both cases, faults occurred with the array and pools stopped, Docker and VMs disabled, Safe Mode active, and no third-party plugins involved. They also reproduced from separately prepared boot media.

The failures affect unrelated components and execution paths—including PHP, smartctl, Node, the dynamic loader, libc, SquashFS, XFS, BTRFS and core kernel memory-management code, and occur across both CPU sockets. This is not the pattern expected from a single Unraid service or a regression confined to one kernel version.

The repeated PHP offset may represent a separate defect in a software path common to both releases, since 7.2 and 7.3 share some userspace components. However, that cannot explain the complete cross-runtime, filesystem and kernel corruption history.

Taken together, the evidence now points primarily to intermittent platform-level corruption. The remaining likely areas include:

- CPU cache or execution instability

- Integrated memory controller instability

- Motherboard or chipset problems

- Power delivery or transient PSU/board regulation problems

- DMA or PCIe-related memory corruption

- Firmware or another platform-level issue

Passing Dell diagnostics and five Memtest86+ passes reduces the likelihood of an ordinary reproducible CPU, DIMM or monitored hardware failure, but it does not rule out intermittent faults in the CPU caches, memory controllers, motherboard, power delivery or DMA paths. Those failures can corrupt memory without producing an EDAC, MCE or Dell event.

You may continue the current 7.2.8 run to 48 hours for completeness, but it is already a failed test because four processes have faulted. I do not think another return-to-7.3 test is necessary unless you specifically want to compare the frequency of faults or kernel oopses.

The next useful work should be hardware isolation rather than further Unraid-version testing:

1. Load conservative BIOS defaults and remove any non-default performance or power settings.

2. Reseat the processors, RDIMMs, risers and relevant power connections, or have this performed by a qualified technician.

3. Test reduced matched RDIMM populations, following Dell’s T420 socket/channel population rules.

4. If practical, test with one CPU/socket configuration or substitute known-good compatible CPUs, memory or motherboard hardware.

5. The most decisive control would be running the same stripped Unraid configuration on different known-good hardware.

IMO there is no evidence that upgrading, downgrading or changing the Unraid kernel branch resolves the overall problem. The broad failure pattern is now much more consistent with an intermittent hardware/platform fault.

Understood, and that matches what we were starting to suspect ourselves given the offset repeating across kernel versions. Appreciate you laying out the reasoning clearly.

7.2.8 FINAL NUMBERS (stopped the count here, past your 20.5h reference point)

32h29m elapsed: zero kernel oopses, five smartctl/smartctl_type segfaults total (one more since the last update, at 12:42:18, offset 0x666314 in php — same family as the 0x666303 one, one byte apart). EDAC stayed active and clean the entire time (0/0/0/0 across both memory controllers). Array, pools, Docker and VMs stayed stopped/disabled throughout, zfs and spl modules loaded but pools never imported.

Not running a fresh-7.3.2 retest per your note, since you don't think it's necessary at this point.

MOVING TO HARDWARE ISOLATION

Putting the original, untouched boot USB back in for regular use while we work through this, since the fault clearly isn't tied to which media or which Unraid version is booted.

Going to start with your suggestion of substituting known-good compatible parts. Ordering a couple of replacement CPUs first (they're inexpensive for this platform) to rule those in or out before touching memory — with the current state of things, going down the path of buying replacement RDIMMs as well if the CPUs don't resolve it is going to get expensive fast, so we're prioritizing the cheaper substitution first.

Also checking the PSU2 situation now that you mention power delivery as a candidate — it's been sitting in a "input lost" state for a while, which we'd previously assumed was unrelated, but given the pattern reset given by this reply we're going to make sure it's actually getting power before ruling it out.

Will keep this thread updated as we work through the hardware substitutions.

Quick update.

After the BIOS factory-reset + PSU2 swap, the system ran completely clean for about 25.5 hours (previous failures were typically showing up within just a few hours of boot) — first sign of trouble was a segfault at 23:12 on 31/08. Since then we've had 6 userspace segfaults total over about 12 hours, plus one incident that had a more visible impact: at 09:24 today, Unraid's own "shfs" User Share daemon (the FUSE process backing /mnt/user) segfaulted and took the whole /mnt/user mount down ("Transport endpoint is not connected") until I manually unmounted/remounted it — several containers with an open handle into the old mount (two Postgres instances, Vaultwarden, Paperless) needed a restart afterward to recover cleanly.

All 6 segfaults share the same signature: fault address near-zero (0x0–0xa9), error code 4 (user-mode read on a not-present page) — a pointer that should point to a valid struct coming back null in every case, across 6 completely unrelated processes (a Plex updater, smartctl, shfs, node, and Celery/Python twice). Two of them — smartctl_type and shfs — faulted at the exact same offset (0xb34c) inside two totally unrelated shared libraries, which reads more like something corrupting memory at a consistent relative position than a bug in either piece of software. EDAC and the MCE log stayed silent through all of it (0/0 CE/UE, no machine-check entries), so whatever this is still isn't tripping the standard ECC/MCE detection path.*

For what it's worth, 5 of the 6 crashes landed on CPU socket 0, only 1 on socket 1 — not enough to draw a conclusion, but noting it since we're about to try swapping CPUs.

So: longer time-to-first-failure than before, but once it started again the pattern and frequency look similar to what we'd seen previously, just with a new symptom this time (a core Unraid service crashing instead of an application). Continuing to monitor, and will report back once the CPU swap is done.