teiresias

Update: root cause found Thanks for the quick replies. “Memtest immediately” was the right call. I ran Memtest and it started showing errors very quickly — over 1200 errors already at around 4% / Test 2. I then tested the RAM sticks individually in the same slot: - Stick 1 in slot 1: no errors - Stick 2 in slot 1: errors So the issue appears to be a bad RAM stick. This also explains why BTRFS RAID1 could not correct the checksum errors: bad data was likely already being produced in memory before being written to both mirrored NVMe devices. Unfortunately RAM prices are very high at the moment, so for now I will run the server with only the good 32 GB stick. Thanks everyone for the help. This seems to have been a RAM issue, not primarily an Unraid/BTRFS issue.

Small update: I checked both NVMe drives with SMART. Both drives report: - SMART overall-health: PASSED - Critical Warning: 0x00 - Available Spare: 100% - Percentage Used: 0% - Media and Data Integrity Errors: 0 - Short self-tests completed without error The only notable values are around 46 Unsafe Shutdowns on both drives, which matches the hard resets I mentioned earlier. There are also some “Invalid Field in Command” entries in the NVMe error log, but no media/data integrity errors. I also checked dmesg for NVMe/controller problems: - no NVMe timeouts - no controller resets - no “controller is down” - no obvious I/O errors The only current errors in dmesg are BTRFS checksum errors pointing to specific files, not NVMe resets or device failures. So at this point the NVMe drives do not look obviously bad from SMART/dmesg. I will run Memtest overnight to rule out RAM issues. Current question: If Memtest passes and the cache pool scrub stays clean after deleting the affected files, would you still recommend rebuilding/reformatting the BTRFS cache pool, or would you monitor it first?

Hi everyone, after updating to Unraid 7.3.x, I started seeing recurring BTRFS read/checksum errors on my cache pool. I am not sure whether this is related to the update, existing corruption becoming visible, or a hardware issue. The cache pool consists of two NVMe devices mounted at: /mnt/cache The errors showed up as BTRFS checksum/corruption errors, for example: btrfs scrub start -Bd /mnt/cache Error summary: csum=... Corrected: 0 Uncorrectable: ... btrfs device stats /mnt/cache showed corruption_errs on both NVMe devices, but no read/write/flush/generation errors. We repeatedly did this: btrfs device stats -z /mnt/cache btrfs scrub start -Bd /mnt/cache btrfs device stats /mnt/cache After deleting the files reported in syslog, the scrub became clean again. But after larger write operations or copying data, new checksum errors appeared again. The affected files were mostly replaceable files, for example: Crafty backup ZIP files Docker layer files Minecraft/Paper/ViaVersion jar files a Nextcloud .js.map file in a temporary backup/staging copy So far, no Minecraft world/ region files were reported as damaged. We copied the Minecraft world folders to a Windows machine using Robocopy and got 0 errors. We also found that the Crafty SQLite database was corrupted: database disk image is malformed So Crafty was reinstalled fresh instead of reusing the old database. Docker was also rebuilt because one corrupted file appeared to be inside old Docker BTRFS subvolumes/layers. Current situation: After removing the last reported damaged file, scrub is clean again. btrfs device stats stays at 0 for now. But because new checksum errors appeared after larger write/copy operations, I am unsure whether this is only old corruption from previous hard resets, or whether there is still an active issue. Important context: I had several hard resets before, because after a router restart I lost access to the server and forced reboots. So unsafe shutdowns are definitely possible. Questions: Does this pattern look like old corruption from unsafe shutdowns, or more like RAM/NVMe/PCIe/PSU problems? Has anyone seen similar BTRFS cache pool errors after updating to Unraid 7.3.x? Would you rebuild/reformat the cache pool even if the current scrub is clean? Besides memtest and NVMe SMART logs, what else should I check? My current plan is: finish backups run memtest overnight check NVMe SMART/error logs monitor btrfs device stats after every larger write rebuild the cache pool if any new csum errors appear again Any advice appreciated.