JorgeB

The new files confirm both SquashFS incidents exactly and add one useful EDAC distinction. The August 13 incident contains two identical failures against block 0x3374df0. The August 14 incident contains seven failures against a different block, 0x206e1fb2, across two bursts 13 seconds apart. Neither daily log contains an accompanying USB reset, lower-level block-I/O error, OOM event, or kernel oops. However, both incidents are surrounded by unrelated process faults, so they remain consistent with the wider corruption pattern. The August 13 SquashFS incident occurred before that evening’s reboot and before sb_edac loaded. The August 14 incident occurred on the following boot after sb_edac registered both memory controllers at 20:27:31, with no intervening reboot or unload visible. No CE/UE was reported. That means at least the second SquashFS incident appears to have occurred with EDAC visibility. This makes an ordinary reportable DIMM-cell error less likely, but it does not exclude CPU cache, IMC, motherboard, DMA, power, or boot/read-path corruption. The current diagnostics confirm that the new test is configured as intended: - Safe Mode - Array and filesystems stopped - Docker disabled - VM service disabled - Only the stock OS plugin - sb_edac loaded at 14:21, three minutes after boot There are no new faults through the 15:23 diagnostic, but that is only approximately 65 minutes, continue this exact configuration beyond the previous 4-hour-38-minute first-fault interval and preferably beyond 13 hours before changing anything else. The API evidence now looks transient. The current file contents look valid, plain require() succeeds, and the post-reboot API log no longer contains the syntax error. The earlier require-in-the-middle stack frame only shows that it wrapped Node’s module-loading path; it does not currently establish that the hook modified or corrupted the source. Also, the findmnt output shows that /usr is presented through an overlay with a SquashFS lower layer, but it does not prove that this specific file came from the writable upperdir. You can check that directly with: test -e /var/local/overlay/usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js \ && echo "present in upperdir" \ || echo "not present in upperdir" Before replacing the boot device, and preferably after the current observation phase so we do not alter this test, calculate the actual hashes of the boot archives and compare them with their supplied hash values: sha256sum /boot/bzimage /boot/bzroot /boot/bzroot-gui /boot/bzmodules /boot/bzfirmware The four newest files received here were the August 13 and 14 logs, the current diagnostics, and an updated August 25 log. We already had and reviewed the August 23/24 evidence from the previous upload. For now, continuing the array-stopped Safe Mode test is the highest-value next step. After that, I would proceed with repeated boot-archive verification, known-good boot media using the same 7.3.2 release, and then the Dell hardware diagnostics and extended memory testing.

The new log confirms that sb_edac registered at 09:57:35, after both the smartctl_type fault at 01:39:58 and the PHP-FPM failures beginning at 09:49:59. Therefore, neither failure occurred while EDAC reporting was active, and the absence of EDAC errors during the overnight period cannot be used to exclude a hardware or memory-path problem. I also would not describe the 13-hour Safe Mode run as completely clean. It is correct that none of the previously tracked kernel oopses occurred, but the smartctl_type segfault attempted to execute address zero. That is consistent with the wider corruption pattern and is significant because it happened with Docker disabled, third-party plugins skipped, and the NVIDIA modules absent. This shows that those components are not required to trigger the issue, but it does not necessarily establish a defect in the base OS—the base utility may simply have encountered already-corrupted state. The PHP-FPM recovery after dropping caches is also important, but I do not think it proves specifically that RAM pages were corrupted. Dropping caches can evict the PHP executable, the dynamic loader, shared libraries, and other file-backed pages. Without checksums or copies from before and after the cache drop, we cannot confirm that identical bytes were read from the USB or identify which object was corrupt. The earlier SquashFS decompression errors make the boot image, USB/read path, and decompression path important possibilities alongside RAM, CPU/IMC, motherboard, and kernel memory-management corruption. If possible, please supply the complete earlier SquashFS errors and their surrounding log windows. Regarding the API error, drop_caches is not limited to read-only SquashFS files; it can also evict eligible clean pages belonging to writable filesystems. The continued API failure could mean the file is persistently corrupt, the process retained an old mapping, or a different dependency is affected. Safe Mode also reports that API plugin discovery was skipped, while the failing file is under the core /usr/ local/unraid-api tree, so I would not yet assume that reinstalling the Connect plugin will correct it. Before reinstalling anything, please capture: findmnt -T /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js sha256sum /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js file /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js xxd -l 128 /usr/local/unraid-api/node_modules/@nestjs/common/pipes/file/file-validator.interface.js If PHP-FPM fails again, please capture its filesystem, checksum, and ldd output before dropping caches. Since the base runtime has already shown corruption and the API remains damaged, I would not start Docker containers on this same boot. I recommend preserving the evidence, rebooting cleanly, and then continuing the Safe Mode test with sb_edac loaded. After that, the highest-value comparisons would be: - Safe Mode with the array stopped. - Dell extended CPU and memory diagnostics plus multiple full memory-test passes. - The same Unraid 7.3.2 release from known-good, freshly prepared boot media. - If those tests remain clean, a controlled 7.3.2 → 7.2.8 → 7.3.2 comparison.

Were you able to confirm if this still happens in safe mode and/or get new logs of more events?

This is a known bug; restart the array and they will go away.

That's not an error; it just gets highlighted because of the word "error" showing.

Disable Docker and VM service and reboot in safe mode, start the array and see if it stops, if yes, retest again after booting in normal mode to rule out a plugin issue, leave the services still disabled, if still OK enable one of the services and retest, then the other, note that if the issue happens with the docker service enabled only, you may need to try enabling one or a few containers at a time to find the culprit.

Aug 17 04:40:24 EggsBenjamin kernel: SQUASHFS error: Unable to read directory block [2ddf6a97:a72] Aug 17 04:40:24 EggsBenjamin kernel: SQUASHFS error: Unable to read metadata cache entry [2ddf6a97] Aug 17 04:40:24 EggsBenjamin kernel: SQUASHFS error: Unable to read directory block [2ddf6a97:a72] Aug 17 04:40:24 EggsBenjamin kernel: SQUASHFS error: Unable to read metadata cache entry [2ddf6a97] This typically means flash drive problems; you can try recreating it first. If issues continue, replace it.

If the latest version shows as an update from 6.9 you can update directly, but create a full flash drive backup first. Also, recommend reading the release notes for all the minor versions in between, 6.10.0, 6.11.0, etc

When you create a snapshot, a local qcow2 file is created to reference it. Getting rid of those files, and restoring only the vdisk should resolve the issue.

We seem to be going in circles. I already told you before how to add a slot to an existing pool, but the pool you posted a screenshot from already had 2 members; you can create a new one if you like using the wizard (after rebooting from a flash drive)

SHould be in the docs below, and you would only lose the snapshot, not the current data. https://docs.unraid.net/unraid-os/using-unraid-to/create-virtual-machines/vm-setup/#vm-snapshots

This is not the correct procedure to create a boot pool, you don't start the array or format anything, just run the wizard and let Unraid configure the internal boot pool, and it will then automatically reboot.

Thanks, the new logs are very useful. They confirm that this is broader memory-management corruption rather than an isolated shfs, XFS, BTRFS, or Plex problem. Before the later crashes, the kernel reports bad RSS counters, non-zero page-table accounting, and BTRFS pages that are still mapped while being deleted. Oopses #2 through #12 then fault at the same location in mas_ascend(), with the repeated state strongly suggesting that lsof, another task, and later khugepaged are encountering the same damaged process address-space tree. The repeated trailing xfs_iget() register block matches the original August 20 oops and does not appear to be a new XFS crash each time. The fault while unmounting /var/lib/docker occurs after the system has accumulated all this corruption, so it is more likely a secondary failure than a separate BTRFS unmount bug. After the reboot, unrelated .NET and dynamic-loader processes start segfaulting again within approximately 50 minutes and on both CPU sockets. The later Plex message requesting about 94.5 TB is not genuine memory exhaustion, the diagnostics still show approximately 77.5 GB available, but likely another indication of a corrupted process state. This suggests a real kernel/MM corruption issue, but it still does not establish whether the underlying cause is the 7.3.2 kernel, an optional/out-of-tree module, or hardware/ platform instability. There are still no recorded MCE, EDAC, PCIe AER, OOM, storage-reset, or filesystem-corruption errors. The latest diagnostics show another normal-mode 7.3.2 boot, so the most useful next test would be: 1. Boot in Safe Mode with Docker and VM autostart disabled. 2. If that remains stable, enable Docker while still in Safe Mode and start the workload/container groups incrementally. 3. Also provide the results of Dell’s extended hardware/memory diagnostics, multiple complete memory-test passes, and the iDRAC/Lifecycle Controller SEL. 4. If those tests remain clean, compare 7.3.2 with 7.2.8 while keeping the BIOS settings, memory population and workload unchanged. Ideally, return to 7.3.2 afterward to confirm whether the corruption recurs. Keep remote syslog enabled. After the first new kernel oops or bad-page/RSS warning, preserve the log and reboot rather than continuing workloads on the already-corrupted kernel.

Please use the wizard to create the pool, not the CLI