Hi everyone,

 

Unraid 6.12.10, AMD 7800x, 32GB RAM. One cache (now ZFS, previously BTRFS), one Array (BRTFS in process of converting to ZFS, one drive converted, one to go, dual parity). We use our Unraid as VM storage for an ESXi server connecting via NFS. A few minor dockers, no (built-in) VMs.

 

We've been having failures on our Unraid for months now (about once a week) that seemed to be related to BTRFS. Something about IO failures on the cache device.. I don't have the exact message atm, but we seem to have moved "past" it. This started seemingly randomly after Unraid running for about a year without issue. Originally, we tried replacing some of the cache nVME drives.

 

Part of my research seemed to indicate it could be hardware failure, and while it's still possible there's a bad motherboard or CPU, we've run the RAM through multiple memtest86 passes over hours and weekends and it's fine. [edit: We did swap the motherboard for another of the same part and the issue persists] Part of my research also indicated it could be file system corruption or other software related issue, so we went through the process of moving files and recreating the cache as ZFS. That was completed two weeks ago, and then our above-mentioned BTRFS error seemed to have "migrated" from the cache to the array. So the next decision was to transition the array from BTRFS to ZFS.

 

However, last week while I was away, after having had rebuilt the cache to ZFS for about 2 weeks, we are now getting a new error, and it's failing almost immediately, and seems to impact Unraid itself (unable to stop the array due to unable to umount /mnt/user).

 

After rebooting Unraid and ESXi, and turning some of our VMs on, the VMs die as they did before, but instead of the BTRFS error from before we now get what looks like some sort of kernel panic/crash dump. At this point, Unraid is still mostly usable, but we cannot stop the array, or reboot it since it hangs trying to unmount /mnt/user

 

Attached is the "crash dump" when the event occurs, as well as the diagnostics.

I should mention the BTRFS errors we had before started when we were on 6.11.5, and upgrading didn't solve. We spoke with an Unraid paid expert and he didn't the cause but thought it's possible the motherboard was going bad, hence the swap, and thought my approach of changing to ZFS was prudent.

 

[edit: also found a picture of the prior BTRFS error we were getting. This one would about half the time hang the entire Unraid server requiring a hard reboot.]

 

Appreciate any insight as it seems our situation has gone from bad (failures about once a week) to worst (where booting up our VMs causes the crash immediately).

 

Thanks,

Unraid error new.txt tower-diagnostics-20240715-1721.zip

zfs is detecting data corruption, suggest start by running memtest.

Thanks for the reply. We've run memtest several times, I think even over an entire weekend without issue.

memtest is only definitive if it finds errors, fix the pool and if you have multiple sticks try using the server with just one, if more corruption is found try with a different one, that will basically rule out bad RAM, and if corruption keeps happening there will be another issue.