This afternoon I started to get massive BTRFS corruption for the second time, for no known reason on my 2 disk cache pool. I moved everything I could to the array, wiped the pool and restarted the 2 drives as a ZFS RAID1. I then went to restore everything. Was going well, until I got to my HA VM and things went off the rails during the HA restore backup. Now I am getting massive problems trying to start the array on the XFS drives. Ran the fix on the 3 drives, disk 1 seems ok, but disk 2 has the below failure during fix and 3 are currently going through fixes stuck on superblock issues. Those 2 drives wont unencrypt even into maintenance mode. The 2 disk pool is not mounted so I cannot run anything on that at this point. I think I have backups within the last week of most things, but I really dont want to lose any data and/or have to rebuild the server, HAOS etc.

Diagnostic logs attached. Anything that can help!

tower-diagnostics-20260110-0142.zip

Edited January 10Jan 10 by apsmith12
added screenshot

xfs_repair is refusing to write corrupt metadata. If you had other filesystem issues, I would recommend running memtest first.

29 minutes ago, JorgeB said:

xfs_repair is refusing to write corrupt metadata. If you had other filesystem issues, I would recommend running memtest first.

Def seems to be some memory errors, that I assume would be the possible cause of the corruption, but doesn't help me recover the data correct? I also assume I gotta figure out which DIMM is at fault if not both?

First you need to fix that, and don't try to run the server until that is resolved. Once it's fixed, try running xfs_repair again; it may or may not then be able to fix it.

If it can't, best bet is using a file recovery app, like UFS Explorer.

Can parity not help if I manage to fix 2 of the 3 drives? Can the one drive be considered failed?

Rebuilding a corrupt filesystem from parity will result in a corrupt filesystem

8 hours ago, JorgeB said:

don't try to run the server until that is resolved

No point in speculating about what else can be done until we see how things are working without your memory issues.

You should never attempt to run any computer unless memory is working perfectly. Everything goes through RAM. The OS and other executable code, your DATA. Everything. The CPU can't do anything with anything until it is loaded into RAM.

So I replaced both DIMMS, and no memory errors. When I got back into the server, drive 3 fixed, so Drive 1 and 3 are working. Drive 2 still gives that same fix error, where it doesnt want to actually do anything. It also wont mount, says its not encrypted (it was and the rest of the array is) and the cache pool wont mount.

Post new diagnostics

1 hour ago, trurl said:

Post new diagnostics

tower-diagnostics-20260110-2138.zip

Diags after array start, please.

8 hours ago, JorgeB said:

Diags after array start, please.

tower-diagnostics-20260111-1122.zip

What happens if you change disk2 to xfs:encrypted? Does it think you need to format or does it recognize it?

1 hour ago, trurl said:

What happens if you change disk2 to xfs:encrypted? Does it think you need to format or does it recognize it?

I really don't see anyway to do that, but yes the one way I know how it wanted to format.

I took the disk out and hooked it to a Linux system via USB thinking it would see it and let me put in pw, but it just tells me there are errors. Im wondering if the memory issues caused corruption to the encryption and that's why it wouldn't repair

Disk2 still has filesystem issues:

Jan 11 03:05:16 Tower kernel: XFS (md2p1): Filesystem needs repair. Please run xfs_repair.

But if xfs_repair still cannot repair it, best option is a file recovery app, like UFS Explorer. The free trial should show if it can recover anything.

It cant find anything to repair. I think the corruption while encrypted caused issues. Most of the data was on Disk 1, and I had backups of enough of what I am missing to be content for now.