drmalcolm

Gotcha, thanks! Is it safe to stop the array to add a disk for parity? I'd like to avoid triggering the bug again if possible.

This worked! Thank you greatly for this! Disks are up and looks like the data is intact. My question now is: I am assuming that now these disks are "fixed", but how should I proceed? I have 3 disks that have 1GB left and I want to hear your recommendations on how best to balance that out. Assuming that the lack of free space likely contributed to the corruption, I definitely want to handle this before I stop the array next. I used the unbalance plugin the last time to accomplish this (mostly to scatter all data from one disk to other disks), but I wanted to see if there is a better unraid-native way I can do this. I've attached diagnostics just in case. nasa-diagnostics-20240131-1248.zip

The original set of diagnostics should have that information. I started the process around Jan 30 12:00:00. I went through the syslog and can see the series of unmounts from bringing the array down. Later at 12:00:45 is where I can see logs in the syslog about wrong disks being found, and then around 12:14 begins my series of attempts to fix using new config, and the subsequent mount failures. I hope that's able to help. As for the repairs, they have all failed with "Sorry, could not find valid secondary superblock. Exiting now." Is there any further action I can take on these drives, or are these effectively dead? If they are dead, I'll start the process of formatting and restoring data, but I want to know what I can do about preventing this in the future. I have two disks that are full, what would be the best way to scatter some of that data so that there's more breathing room on the drives?

I appreciate the warning! I am learning, and while it's quite different from a "normal" Linux nas setup, it's quite a clever system. As for xfs_repair output, I haven't had a single one finish yet. They've been running for several hours so far, and not even my small 4TB drive has finished scanning, and I suspect they might be going for quite some time. However, the longer they run, the less hopeful I am that a superblock can be recovered. Output so far has been: Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!! attempting to find secondary superblock... ...found candidate secondary superblock... unable to verify superblock, continuing... ........... Are disks that are too full causing the issues with corruption? I'm interested to hear more about this. I used the unbalance plugin to scatter the data from 1 drive to the rest of the drives (i made sure all other drives were selected), but it only really transferred to 3 other drives, which completely filled 2 of them (the two that are actually fine). The unbalance plugin has a default of 1GB min free space, which I suspect does not actually play nicely with unraid. I also would like to hear your thoughts about what should be done with my two full drives. Whether or not these other drives are repaired, they will eventually be added back to the array as a working or freshly wiped xfs drive. When that happens, what would be the best way to balance the data so that this doesn't happen again? I am assuming that once I am settled, unraids default data allocation strategy of High Water should be enough to never cause this issue on its own, correct?

Sorry my wording was a bit unclear: I ran xfs_repair on the failed disks (the xfs_repair hadn't failed yet, they were still running). I was running xfs_repair -v /dev/mdxp1 where x corresponds to the disk number I am checking. This was being run in the terminal just because I am usually more comfortable with cli work. I rebooted, started the array and saved the diagnostics. You will no doubt see the invalid superblock magic number for a number of the disks in the syslog. While I do hope I can recover, I do want to make sure this doesn't happen again (be it user, hardware, or software error). nasa-diagnostics-20240130-1746.zip

So quick update: i've read a number of articles and unraid documentation to initiate some repair attempts using xfs_repair on the various disks that seem to have failed. Those are still running, and have been for a couple hours now. While I hopefully await recovery, is any way of seeing why this happened in the first place? Nothing lost power (I am on an EATON ups and there was no power fluctuations), there wasn't even a reboot, I was using the disks just fine just before I stopped the array to add a parity, and the failed disks are scattered amongst devices (I have a disk that's directly attached to the motherboard via the built in sata backplane, and disks connected to 2 different MD1200 SANs via a LSI sas card in IT mode). Digging into the logs myself, I could see the various operations I performed, as well as the unmounts from stopping the array, but I couldn't find anything out of the ordinary that would potentially cause any sort of fs corruption.

I just recently migrated to Unraid this past week. I had an existing NAS running OMV that was connected to a 12 bay SAN, with drives formatted ext4 and pooled together with mergerfs (using snapraid for parity). I added a few temporary drives to the new Unraid machine to transfer stuff over. The new drives were formatted properly with XFS, and the transfer happened successfully. I then switched over my SAN to the new unraid machine, had unraid format all the new drives, and then used unbalance to scatter filed from one of my drives I wanted to use as a parity. That, too, happened successfully. Where it went wrong was after spinning down my array to re-enable the cache and to move that one disk to parity, and I got a bunch of errors that disks were wrong, even though the disks looked right. I used the new config tool, reassigned the drives, and now a all but 2 of my drives are listed as "Unmountable: Unsupported or no file system". This is strange considering I had Unraid properly format the drive after they were originally added to the array. If I unassign a disk and try to mount it using Unassigned Devices, the mount button is disabled. Posted the diagnostics. I am hoping for recovery here, and am hoping to understand what happened. Edit: From the logs, looks like there are bad superblocks somehow? Any way to recover from this? nasa-diagnostics-20240130-1230.zip