JorgeB

Don't format anything or it will also format disk1.

Best bet is to reboot, see if you can get the diagnostics before doing it.

Btrfs is detecting data corruption, this is usually RAM related, and Ryzen with overclocked RAM like you have is known to corrupt data in some cases, start by correcting that then run a scrub.

this suggest a device problem, or wrong capacity, post new diags after running it.

Server remains online during a rebuild.

Check filesystem on the emulated disks 11 and 15, run it without -n or nothing will be done, if it asks for -L use it.

If the upgrade was done with v6.11.2 you must rebuild again with the current release to correctly partition the disk: stop array unassign disk1 start array post new diags.

Nov 21 07:47:02 Tower emhttpd: error: share_luks_status, 6173: Operation not supported (95): getxattr: /mnt/user/appdata Nov 21 07:47:02 Tower emhttpd: error: share_luks_status, 6173: Operation not supported (95): getxattr: /mnt/user/local Nov 21 07:47:02 Tower emhttpd: error: share_luks_status, 6173: Operation not supported (95): getxattr: /mnt/user/mergerfs Nov 21 07:47:02 Tower emhttpd: error: share_luks_status, 6173: Operation not supported (95): getxattr: /mnt/user/mount_rclone These paths should not exist before array start, do you know what is creating them?

Disk was already disabled at boot time so we can't see what happened, but SMART looks good so most likely a power/connection problem.

NOCOW disables checksums, so with raid1 if one of the devices drops offline and then comes back online btrfs has no way of knowing which device has the latest and correct data, and it will just read form both alternately, and since the dropped device has wrong data it can result in data corrutpion, e.g:

It's the same: X7SBE with AOC-SIM1U+ and firmware v1.64: Don't know why the IPMI device shows an X8

Also, since it's a Ryzen server take a look here: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

Update the LSI to latest firmware (20.00.07.00), all other p20 releases have known issues.

One of the devices dropped offline, you should run a scrub to bring it up to date, corruption errors are normal in this case for every synced block, you can re-set them when done. Make sure system share is set to COW, old default was NOCOW, and that cannot be corrected. The below might help with the dropping device. On the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 pcie_aspm=off e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off Reboot and see if it helps.

Enable the syslog server and post that after a crash.

Monthly scrub should be enough, regular balance might not be needed, depends on how the pool is used, but if you want to schedule one a monthly balance using the default block usage should be good.

Unassign the unmountable disk Start array Stop array Re-assign the disk Start array, disk will be rebuilt again but it should now mount

Yes, but by unassigning them parity together with the other drives should emulate the old contents, assuming parity is still valid, and it might not be since you mentioned there were writes to the array.

Some SMR disks work fine if it's mostly large file transfers, some models are always much slower, in my experience most Toshiba SMR models, Seagate are usually fine, again for large files, but that specific model seems to be hit and miss.

The emulated disks should still mount, before starting the rebuild.

That makes sense, like I mentioned disk 15 was logged as a disk problem. Filesystem was not detected on disks 11 and 15, not a very good sign, but click on both disks with the array stopped and change the filesystem from auto to xfs, then post new diags after array start.

I would recommend running mentest, though it's only conclusive if it finds errors.

Problem is that might make recovery not possible, or at least cause some data loss, you can still try but keep old disk11 intact, that way you can also try and use ddrescue on it. To re-add old disk 15 and try to rebuild both disks you can do this: -Tools -> New Config -> Retain current configuration: All -> Apply -Check all assignments and assign any missing disk(s) if needed, including old disk 15 and new disks 11 and 21, replacement disks should be same size or larger than the old ones -IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked) -Stop array -Unassign disks 11 and 21 -Start array (in normal mode now) Grab and post new diags.

Yes, that disk looks OK, nothing written to the serve since you started the replacement?

Post a SMART report from old disk15, looks like it might be failing also, if you didn't write to the array since the initial replacement don't for now.