Hi everyone,

after updating to Unraid 7.3.x, I started seeing recurring BTRFS read/checksum errors on my cache pool. I am not sure whether this is related to the update, existing corruption becoming visible, or a hardware issue.

The cache pool consists of two NVMe devices mounted at:

/mnt/cache

The errors showed up as BTRFS checksum/corruption errors, for example:

btrfs scrub start -Bd /mnt/cache
Error summary: csum=...
Corrected: 0
Uncorrectable: ...

btrfs device stats /mnt/cache showed corruption_errs on both NVMe devices, but no read/write/flush/generation errors.

We repeatedly did this:

btrfs device stats -z /mnt/cache
btrfs scrub start -Bd /mnt/cache
btrfs device stats /mnt/cache

After deleting the files reported in syslog, the scrub became clean again. But after larger write operations or copying data, new checksum errors appeared again.

The affected files were mostly replaceable files, for example:

Crafty backup ZIP files
Docker layer files
Minecraft/Paper/ViaVersion jar files
a Nextcloud .js.map file in a temporary backup/staging copy

So far, no Minecraft world/ region files were reported as damaged. We copied the Minecraft world folders to a Windows machine using Robocopy and got 0 errors.

We also found that the Crafty SQLite database was corrupted:

database disk image is malformed

So Crafty was reinstalled fresh instead of reusing the old database.

Docker was also rebuilt because one corrupted file appeared to be inside old Docker BTRFS subvolumes/layers.

Current situation:

• After removing the last reported damaged file, scrub is clean again.

• btrfs device stats stays at 0 for now.

• But because new checksum errors appeared after larger write/copy operations, I am unsure whether this is only old corruption from previous hard resets, or whether there is still an active issue.

Important context: I had several hard resets before, because after a router restart I lost access to the server and forced reboots. So unsafe shutdowns are definitely possible.

Questions:

1. Does this pattern look like old corruption from unsafe shutdowns, or more like RAM/NVMe/PCIe/PSU problems?

2. Has anyone seen similar BTRFS cache pool errors after updating to Unraid 7.3.x?

3. Would you rebuild/reformat the cache pool even if the current scrub is clean?

4. Besides memtest and NVMe SMART logs, what else should I check?

My current plan is:

finish backups
run memtest overnight
check NVMe SMART/error logs
monitor btrfs device stats after every larger write
rebuild the cache pool if any new csum errors appear again

Any advice appreciated.

Edited May 25May 25 by teiresias

Memtest immediately.

Small update:

I checked both NVMe drives with SMART.

Both drives report:

- SMART overall-health: PASSED

- Critical Warning: 0x00

- Available Spare: 100%

- Percentage Used: 0%

- Media and Data Integrity Errors: 0

- Short self-tests completed without error

The only notable values are around 46 Unsafe Shutdowns on both drives, which matches the hard resets I mentioned earlier. There are also some “Invalid Field in Command” entries in the NVMe error log, but no media/data integrity errors.

I also checked dmesg for NVMe/controller problems:

- no NVMe timeouts

- no controller resets

- no “controller is down”

- no obvious I/O errors

The only current errors in dmesg are BTRFS checksum errors pointing to specific files, not NVMe resets or device failures.

So at this point the NVMe drives do not look obviously bad from SMART/dmesg. I will run Memtest overnight to rule out RAM issues.

Current question:

If Memtest passes and the cache pool scrub stays clean after deleting the affected files, would you still recommend rebuilding/reformatting the BTRFS cache pool, or would you monitor it first?

3 hours ago, teiresias said:

Small update:

3 hours ago, trurl said:

Memtest immediately.

You shouldn't be doing anything else except memtest. Each moment you run your server with suspect RAM you may be corrupting things.

You must never attempt to run any computer unless memory is working perfectly. Everything goes through RAM. The OS and other executable code, your DATA! EVERYTHING! The CPU can't do anything with anything until it is loaded into RAM.

13 hours ago, teiresias said:

But after larger write operations or copying data, new checksum errors appeared again.

run memtest

Update: root cause found

Thanks for the quick replies. “Memtest immediately” was the right call.

I ran Memtest and it started showing errors very quickly — over 1200 errors already at around 4% / Test 2.

I then tested the RAM sticks individually in the same slot:

- Stick 1 in slot 1: no errors

- Stick 2 in slot 1: errors

So the issue appears to be a bad RAM stick. This also explains why BTRFS RAID1 could not correct the checksum errors: bad data was likely already being produced in memory before being written to both mirrored NVMe devices.

Unfortunately RAM prices are very high at the moment, so for now I will run the server with only the good 32 GB stick.

Thanks everyone for the help. This seems to have been a RAM issue, not primarily an Unraid/BTRFS issue.