Thiago Alexander

Good afternoon! Understood, I'll get started on those tests and keep you updated. In a couple of hours once I finish work, I'll boot it up in Safe Mode with Docker disabled (VMs are already off since I don't really use them). I'll leave it running like that overnight to see how it holds up, and tomorrow morning I'll start enabling the Dockers little by little. Thanks a lot for the help! :)

Good morning, I was starting to go through the tests you suggested, and sorry in advance if some of this isn't phrased quite right, I'm not very experienced with this side of things. I'm working through it with the help of Claude (the AI assistant) reading and interpreting the logs for me, since on my own I wouldn't have known where to look. It actually started throwing fresh errors while I was in the middle of getting ready to run your tests, so there's more to report than I expected sharing everything below, plus a fresh diagnostics zip (taken while some of this was still actively broken, so it should show the faulty state directly) and the two most relevant daily log files from our remote syslog archive. --- alpha-diagnostics-20260824-1132.zip Logs23-24.zip Update had another incident overnight that materially changes the picture, so sharing the new data before deciding on next steps. Turns out there wasn't just the one Oops from before going back through the full remote log archive, this same boot session had accumulated at least 12 kernel Oopses (the internal Oops counter read "[#12]" on the last one before the reboot), most of which we hadn't individually inspected until now. The most informative one we've captured so far: kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 kernel: Oops: Oops: 0000 [#12] SMP PTI kernel: CPU: 9 UID: 0 PID: 224 Comm: khugepaged Tainted: P B D O 6.18.38-Unraid #1 PREEMPT(voluntary) kernel: Tainted: [P]=PROPRIETARY_MODULE, [B]=BAD_PAGE, [D]=DIE, [O]=OOT_MODULE kernel: RIP: 0010:mas_ascend+0x92/0x1c0 kernel: Call Trace: kernel: mas_next_slot+0x13c/0x3e0 kernel: mas_find+0x44/0x110 kernel: khugepaged+0x199/0x600 kernel: kthread+0x1ce/0x1e0 kernel: ret_from_fork+0x24/0x170 khugepaged crashed inside the maple tree code (mas_ascend, the VMA-tracking data structure). The kernel tainted itself BAD_PAGE an explicit, direct signal that it found a corrupted page, not an inference on our part. Immediately following, as a nested fault with no new Oops header of its own: kernel: RIP: 0010:xfs_iget+0x544/0x8f0 kernel: note: khugepaged[224] exited with irqs disabled The exact same function and byte offset as the xfs_iget crash reported before. So that one wasn't a one-off; it's a recurring fault point, and this time it triggered as a secondary fault while the kernel was already unwinding from the khugepaged crash. The system did not hard-lock this time and kept running for close to two hours afterward on an already BAD_PAGE-tainted kernel. That state cascaded: ~2 hours later shfs threw "fuse internal error: unable to unhash node" and the user-share mount died, Samba started failing client connections shortly after, and when we tried a normal array stop to recover, the stop sequence's own umount call died with the same "exited with irqs disabled" signature, leaving "sync" permanently stuck in D-state (unkillable). Only a forced power cycle via BMC recovered it; a graceful shutdown was never going to complete given the stuck task. So in one boot session we had three distinct core kernel subsystems each independently producing null-pointer faults: the shadow/LRU shrinker (shadow_lru_isolate), XFS inode lookup (xfs_iget, twice now), and VMA/maple tree management under khugepaged (mas_ascend), the last one with an explicit BAD_PAGE taint. That's a lot of independent surface area for coincidence, and BAD_PAGE is about as close to a direct kernel statement of "a memory page got corrupted" as we're going to get from logs alone. I'll fully grant that BAD_PAGE tells us a page was bad, not why consistent with either a kernel-side bug corrupting its own bookkeeping, or genuine hardware-level corruption that these memory-management structures happen to be sensitive enough to catch. It doesn't resolve which of those two it is, but it makes "several unrelated, coincidental single-app bugs" a much harder sell. Still zero ECC/MCE events, before or after any of this, including right through the incident and the forced reboot. Array itself came back healthy 8 disks, all DISK_OK, 0 read/write errors, automatic post-unclean-shutdown parity check completed with only 1 correction (expected/normal for that kind of shutdown, not a red flag on its own). On the syslog concern from before: confirmed it was a false alarm on my end the "empty" daily files are the current, not-yet-rotated file that logrotate resets to 0 bytes; the actual content lives in same-day files with a rotation date suffix. Found the original kswapd0/shadow_lru_isolate Oops there with its full trace intact. Since then we've also moved the forwarder from UDP to TCP for reliability, confirmed via a live round-trip test with an established TCP connection to the receiver. ADDITIONAL #1, discovered this morning after the reboot: Plex Media Server itself started segfaulting on every single launch attempt 100% reproducible, multiple consecutive identical crashes, each writing its own local crash report: /home/nobody/start.sh: line 47: 119 Segmentation fault "${PLEX_MEDIA_SERVER_HOME}/Plex Media Server" ****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /config/Plex Media Server/Crash Reports/.../....dmp The first sign was actually a segfault in "PMS GTP" (Plex's GPU transcode helper process) a few hours after the reboot; after that, the main Plex Media Server binary itself started dying on every start, reproducibly. This matches a pattern we'd already seen with two other unrelated containers earlier in this whole saga (a persistently corrupted DLL in one, corrupted libssl in another) 100% reproducible crashes that a plain restart never fixes, only clearing after a full image re-pull, which strongly suggests bytes got corrupted on disk at some point rather than a transient in-memory glitch. Given the timing (a few hours after last night's BAD_PAGE-tainted kernel Oops), it's hard not to read this as another downstream casualty of the same event. Still working on getting it re-pulled cleanly. ADDITIONAL #2, also this morning: the Unraid API service itself (the Node.js/GraphQL backend behind the modern WebGUI dashboard stats, live CPU graphs, etc.) had also silently died. Its PM2-tracked PID no longer corresponded to any running process, and the dashboard had stopped updating CPU usage entirely. A plain service restart (rc.unraid-api restart) brought it back cleanly with no errors on the fresh start, so no persistent corruption there as far as we can tell but it's one more service that quietly dropped out sometime after the reboot, on top of Plex and the two other containers that needed manual restarts after the array came back (they'd been orphaned mid-crash when Docker itself got stopped during the failed array-stop sequence). So the blast radius of last night's event, directly or indirectly, touched: the kernel three times over, the array/share layer, Samba, at least four Docker containers, and now the WebGUI's own backend service. Attaching: a fresh diagnostics zip (captured while Plex and the Unraid API were still actively broken, so it should reflect the faulty state rather than the already-recovered one), plus the two daily log files from our remote syslog archive covering yesterday and last night/this morning (2026-08-23.log and 2026-08-24.log) these have the full timeline including everything quoted above with exact timestamps. Given all of this, we're moving up the timeline on the 7.3.2 -> 7.2.8 -> 7.3.2 controlled comparison you suggested, and will also look at Dell's extended diagnostics and further memtest passes as we get a maintenance window. Thank you :)

Logs - 08.zip Sorry, it looks like it won't let me upload the logs as .txt files, so I've put the latest ones I have into a ZIP file.

alpha-diagnostics-20260822-1351.zipUploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment...Uploading Attachment... Here is the latest diagnostics report, pulled right before sending this message, along with the recent logs I've collected over the past few days :)

Hi, good evening everyone! Apologies in advance for my English, it’s not my native language, im using a translator :) I'm no tech expert, I really enjoy it, but I have to admit some of this stuff is a bit tough for me hahahah. I have a modest server setup: a Dell T420 with dual Xeon E5-2450 v2 CPUs. For over a month now, I've been running into a lot of issues. A friend recommended I use Claude in Visual Studio Code to help diagnose what's going on with the server. I even connected an old laptop to capture the Syslog in real time, just in case it froze completely again like before, leaving me with no SSH access or anything. After analyzing the error logs for a couple of weeks, it seems the conclusion is a kernel panic/error related to the dual-CPU setup. This started right when I updated to version 7.3.2, so I'm considering rolling back to the previous version, but I'd love to get your opinion first. I wanted to have at least some context before coming here so I wouldn't be completely clueless ;-; Here is the summary Claude gave me. Thank you so much in advance for your help! :) SYSTEM - Unraid 7.3.2, kernel 6.18.38-Unraid - Dell PowerEdge T420, 2x Intel Xeon E5-2450 v2 (dual-socket, dual NUMA node), 96GB DDR3 ECC RAM - Broadcom/LSI SAS2308 HBA in IT mode (mpt3sas), 15 disks total across array + 2 ZFS pools (cache, seedeo) - NVIDIA Quadro P2200 for Plex transcoding - ~30 Docker containers, mix of custom bridge and host networking SUMMARY Over the past several weeks this host has shown a recurring pattern: individual containers dying or hanging one at a time, occasional filesystem-level weirdness, at least one full hard lockup with a clean kernel Oops captured, and — most recently — a second, different kernel Oops plus a batch of application segfaults, all while every hardware indicator (ECC, BMC SEL, SMART, memtest) stays completely clean. Writing up the whole pattern rather than a single incident, because no individual piece looks conclusive on its own but together they point pretty consistently at the kernel's memory-management path. We're now seriously considering downgrading from the 7.3.x/6.18.x line to 7.2.8/6.12.x and would like feedback before doing it. ------------------------------------------------------------ PART 1 — THE TWO CAPTURED KERNEL OOPSES (full stack traces) ------------------------------------------------------------ OOPS #1 — full host hard lockup, kswapd0, shadow_lru_isolate Captured via a remote syslog server (this host's own local syslog is tmpfs and gets wiped on crash/reboot, so without the remote copy this would have been lost entirely). The host became completely unresponsive — no SSH, no WebGUI, no SMB, and the BMC's remote KVM showed no video signal at all, as if powered off (fans were still spinning). Only a hard power cycle via the BMC recovered it; the console showed a normal "clean shutdown" style prompt frozen mid-boot-flow, suggesting the crash happened very early/deep rather than mid-userspace. kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 kernel: #PF: supervisor read access in kernel mode kernel: #PF: error_code(0x0000) - not-present page kernel: Oops: Oops: 0000 [#1] SMP PTI kernel: CPU: 24 UID: 0 PID: 352 Comm: kswapd0 Tainted: P O 6.18.38-Unraid #1 PREEMPT(voluntary) kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE kernel: RIP: 0010:_raw_spin_trylock+0x13/0x50 kernel: Call Trace: kernel: shadow_lru_isolate+0x1f/0x150 kernel: __list_lru_walk_one+0xb8/0x140 kernel: list_lru_walk_one_irq+0xc/0x20 kernel: do_shrink_slab+0x199/0x2d0 kernel: shrink_slab+0x16e/0x2a0 kernel: shrink_node+0x515/0x7a0 kernel: balance_pgdat+0x456/0x670 kernel: kswapd+0x292/0x2e0 kernel: kthread+0x1ce/0x1e0 kernel: ret_from_fork+0x24/0x170 kernel: note: kswapd0[352] exited with irqs disabled kernel: note: kswapd0[352] exited with preempt_count 2 kernel: pstore: backend (erst) writing error (-38) Reading: kswapd0 died mid-reclaim inside shadow_lru_isolate() while holding a non-preemptible spinlock, with IRQs disabled (CPU 24, i.e. on the second NUMA node). That's consistent with an orphaned list_lru spinlock — every other CPU that subsequently enters memory reclaim piles up trying to take the same lock, and the whole host goes completely unresponsive. This matches a pattern others have reported on different platforms (see "related reports" below). OOPS #2 — shfs / XFS inode lookup, non-fatal (host survived) Happened roughly two days later, same uptime session (i.e. after the reboot that followed Oops #1). This one did NOT take the whole host down — only one shfs worker task died; the shfs master process and the /mnt/user share stayed accessible throughout, confirmed immediately after by listing shares successfully. kernel: BUG: kernel NULL pointer dereference, address: 0000000000000a18 Comm: shfs, PID 19197 RIP: xfs_iget+0x544/0x8f0 Call trace (top to bottom): xfs_lookup xfs_vn_lookup __lookup_slow lookup_slow walk_component path_lookupat filename_lookup vfs_statx vfs_fstatat __do_sys_newfstatat Reading: a plain stat() syscall (from shfs, Unraid's user-share FUSE layer) walked into the XFS inode-lookup path and hit a NULL pointer inside xfs_iget(). Different subsystem than Oops #1 (XFS inode cache vs. page/shadow LRU reclaim), but the same general area — kernel-side cache/lookup structures misbehaving under this workload — and the same "no hardware fault reported anywhere" signature. ------------------------------------------------------------ PART 2 — APPLICATION-LEVEL SYMPTOMS, IN ROUGH CHRONOLOGICAL ORDER ------------------------------------------------------------ Several Docker containers, running completely different software stacks (Mono/.NET, Python, PHP, Node/V8), have shown segfaults or silent deaths over the same multi-week window. Individually each looks like "that app has a bug"; together, the pattern (identical faulting offsets recurring over days, containers reporting Docker "healthy" while the real process is dead, corruption that clears on a fresh read but not on a normal restart) reads more like something underneath all of them is occasionally corrupting state than N unrelated app bugs. - A media-management container (.NET/Mono runtime) was seen fully hung consuming 100% of a single CPU core with an unresponsive web UI; separately, at another point, we captured a tight burst of 4 segfaults in ~7 seconds, all in libclrjit.so (the .NET JIT compiler) at the identical crash offset: /home/nobody/start.sh: line 10: Segmentation fault /usr/lib/radarr/bin/Radarr -nobrowser -data=/config (repeated 4x within 7 seconds, error 6, fault address 0xffffffff each time) Its process supervisor gave up retrying after the 4th crash ("entered FATAL state, too many start retries too quickly") and never tried again — the container then sat there reporting "healthy" to Docker for the next ~23 hours with zero actual process running, because the healthcheck only checks the supervisor, not the real app. - Another *arr-family container (also Mono/.NET) had a DLL that was consistently, 100%-reproducibly corrupted — every restart crashed identically at the exact same point loading the same DLL. A plain docker restart never fixed it (same corrupted bytes were already unpacked on disk); only a full "Force Update" (re-pulling the image layers fresh) resolved it. docker pull reporting "image up to date" was not sufficient evidence the on-disk bytes were intact — it only checks the registry manifest, not the already-extracted filesystem layers. - A third container (Node.js, Mono-adjacent workflow-automation app) hit an equivalent persistent corruption, this time in libssl rather than app code — again clearing only after a full image re-fetch, not a restart. - A media server (transcoding, uses an NVIDIA GPU passthrough) died with zero trace in its own application log or crash-uploader log — its internal logging simply stopped mid-session and never resumed even though the container reported "healthy" for the following ~2 days. Root cause turned out to be a startup-script bug triggered by the previous incident: after the Oops #1 hard-lockup and reboot, the app's startup script found a stale PID file left over from before the crash, misread it as "already running" inside the fresh container's new PID namespace, and skipped launching the real process entirely — so it just never started, silently, for two full days. - A bookmarking/link-archiving container (Node.js + V8) has been segfaulting repeatedly in its web process (SIGSEGV in the Next.js/V8 runtime), auto-restarting every few minutes in a continuous crash-loop, correlating with a V8Worker-tagged segfault logged by the kernel's own segfault reporting around the same time as the crashes. - Filesystem/service-layer weirdness, independent of any single container: * Twice, a small set of core Samba binaries (rpcd_classic, rpcd_lsad, rpcd_epmapper, samba-dcerpcd) became unreadable ("No such file or directory" from the shell despite ls showing the files present) while sitting in the live, RAM-resident root filesystem — SMB shares became completely inaccessible both times, resolved only by a full reboot. * Twice, a squashfs read (loop-mounted boot-time modules/firmware images from the boot USB) failed with zstd decompression errors on a specific block; both times, forcing a genuine fresh read from the physical USB (bypassing whatever was cached) came back clean, and checksums of the boot images themselves never changed across the whole investigation — pointing at transient corruption somewhere in the read-into-RAM path, not a failing USB drive. * /var/log (also tmpfs/RAM-backed on Unraid) once filled to 100%, silently breaking several services that log heavily, until it was identified and cleared. * Reproduced deliberately, once: stopping/restarting all ~30 Docker containers at once (via rc.docker) reliably triggers kernel messages "BUG: Bad rss-counter state" and "non-zero pgtables_bytes on freeing mm" tied to containerd-shim processes — i.e. the kernel's own process/mm teardown accounting gets confused under concurrent container teardown on this host. Doesn't crash anything by itself, but it's the same general kernel territory (mm/page accounting) as both Oopses above. ------------------------------------------------------------ PART 3 — WHY WE DON'T THINK THIS IS HARDWARE ------------------------------------------------------------ - EDAC (sb_edac) has been loaded and monitoring continuously for over a week, spanning several of the above incidents: correctable and uncorrectable error counts on both memory controllers have stayed at 0 the entire time. - iDRAC SEL (Dell BMC), checked before and after every incident, shows zero memory/CPU/thermal events — only unrelated chassis-intrusion entries from physical maintenance and expected PSU redundancy-loss entries. - Memtest86+ (2 full passes) and a ZFS scrub on both pools came back completely clean. - All 15 disks have clean SMART attributes — no reallocated/pending sectors, no CRC errors. ------------------------------------------------------------ PART 4 — CONSIDERING A DOWNGRADE TO 7.2.8 ------------------------------------------------------------ Everything above happened on the 6.18.x kernel line (7.3.0 through 7.3.2), and 6.18 is a regular upstream "stable" release rather than a Linux LTS kernel — i.e. it's had comparatively little real-world mileage compared to an LTS line. We're considering rolling back to 7.2.8 (2026-07-29), which is still on the actively-maintained 7.2.x branch (receiving its own security patches in parallel with 7.3.x) but runs kernel 6.12.98-Unraid — a genuine upstream LTS kernel with far more time in the field. One more data point that makes us want outside input before doing this: other users reported a "[7.3.1] Kernel oops after zfs task exit" shortly after upgrading onto the 6.18.x line, with crashes several times a day in some cases. Different symptom than what we're seeing, but the same kernel branch and the same general shape of problem — a shrinker/reclaim-adjacent kernel path interacting badly with something (ZFS, in their case; list_lru and XFS inode lookup, in ours). We run two ZFS pools on this box as well, which makes that report stand out to us. Before downgrading: any feedback on whether this is a sound move for this kind of issue, or knowledge of this specific bug being fixed in a later 6.18.x point release (making a downgrade unnecessary), would be genuinely appreciated. ------------------------------------------------------------ PART 5 — RELATED REPORTS WE FOUND ------------------------------------------------------------ - A near-identical case on Proxmox: GPF in list_lru_isolate_move() from kswapd0, also on a dual-socket NUMA host, leaving an orphaned list_lru spinlock and locking the whole machine — https://forum.proxmox.com/threads/kernel-7-0-14-9-pve-gpf-in-list_lru_isolate_move-kswapd0-leaves-an-orphaned-spinlock-whole-host-locks-up-dual-socket-numa-pbs-4-2-5.185582/ - An open Unraid 6.12.8 report of kernel NULL pointer dereferences correlated with ZFS operations (mutex_lock/arc_change_state/zfs_getattr_fast), where the reporter replaced RAM entirely and re-ran extensive memtest with no change — https://forums.unraid.net/bug-reports/stable-releases/consistent-kernel-errors-and-system-crashes-on-unraid-6128-with-zfs-operations-r2904/ - "[7.3.1] Kernel oops after zfs task exit" — https://product.unraid.net/p/7-3-1-kernel-oops-after-zfs-task-exit ------------------------------------------------------------ QUESTIONS FOR THE TEAM ------------------------------------------------------------ 1. Is this a known issue with the list_lru/shadow-node shrinker path (and now XFS inode lookup too) on 6.18.x, particularly on multi-socket/NUMA hardware? 2. Is the ZFS-related oops reported on 7.3.1 the same underlying bug, or unrelated? We run ZFS pools too. 3. Is there a later 6.18.x point release, or an alternate kernel build, where this is known to be fixed? 4. Does downgrading a running array from 7.3.2 to 7.2.8 have any known gotchas (ZFS pool feature-flag compatibility, Docker/btrfs image compatibility, plugin compatibility) we should watch out for before attempting it? Happy to provide full diagnostics, further log excerpts, or test a patched/alternate kernel if one is available.