I tried running xfs_repair -L (via the Check button in the UI) and it seemed like it finished successfully.  Unfortunately, the drive isn't coming back, and I'm reluctant to do anything with the parity drive (I have two).  

 

For context, I had an array of 16tb & 8tb drives - two 16TB parity drives, one SSD cache drive.  I removed a dead 8TB, replaced with a 16TB, added it to the array, and let it rebuild for two days.   This morning, I logged into my machine and found a bunch of missing files, so I started investigating and saw all the xfs corruption errors.  That's when I attempted the repair after reading some forum threads.

Diagnostic zip attached.

tortuga-diagnostics-20250304-0749.zip

Start the array in normal (not maintenance) mode and post new diagnostics.

Here you go. Thanks.

tortuga-diagnostics-20250304-1127.zip

Looks like emulated disk4 is mounted now after you did xfs_repair. It also has a lost+found share from the repair.

 

Before rebuilding on top, let's see how much got put in lost+found.

 

On the User Shares page, click Compute... on the lost+found share and post a screenshot of the result.

Sorry, I'm not clear here.  I can click "Compute All", which seems to churn for a very, very long time.  When I click on lost+found, I do indeed see many, many folders in there for which the location is "Disk 4".  Is there a way to recover these?

Edit:  Nevermind - Compute All is completed:
 

Edited March 4, 20251 yr by Juliet

You repaired emulated disk4, which is actually calculated from parity and all other disks. Physical disk4 is not involved at all. That is why I wanted to see how the repair went before rebuilding it onto that same disk.

 

Unassign disk4 and see if Unassigned Devices thinks it may be mountable.

Unassigned devices seems to have mounted it.

Don't know if it's relevant/useful, but with drive 4 merely emulated, a ton of data is missing from my Music folder in my Audio share.  However, that data does appear to be present on the mounted, unassigned Drive 4.

Thank you for all your help thus far, btw.  I'm a DevOps Engineer by trade, but I spend my days in the clouds, and I don't get this close to the metal too often.

Edited March 5, 20251 yr by Juliet

New Config disk4 back into the array and rebuild parity, instead of rebuilding disk4.

 

https://docs.unraid.net/unraid-os/manual/tools/#new-config

Well, I'm pretty sure I just boned it.  I tried to stop the array, but had a terminal open in the background where the CWD was within /mnt/user, so the array never stopped.  I didn't realize this, so I tried to go ahead with the New Config, but as the docs state "After doing this there is no obvious feedback that anything has been done.", I assumed the "apply" button had done its job.  So I headed back to Main, kept trying to remember where I needed to be to reassign Disk 4 to the array, but then realized it was still stuck in "Stopping".  

After a bit of googling, I saw the answer, realized I had the terminal open, and I exited it - the array stopped.  I went ahead and assigned Disk4 and started it again, only to see Disk4 was rebuilding (which you mentioned we were trying to avoid).  I panicked and stopped the rebuild.  Then I went and tried the "New Config" again, and it was more obvious that something had actually happened.  When I returned to Main, I tried starting the array with Disk4 assigned, and then it started trying to rebuild Parity (which we originally wanted) - but Disk4 was now unmountable.  By starting that rebuild, I am pretty sure I trashed the filesystem.  I tried running xfs_repair on it, and it completed successfully, but Disk4 shows "Unmountable: Unsupported or no file system" and the disk log shows XFS corruption errors. 

So the FS on that drive is in a bad way, and my parity has been wiped - so I'm pretty sure I'm boned.  None of it is your fault...just one of those "oh sh!t" moments and I did a stupid in my panic. 

Here's a fresh diagnostics zip in case theres any chance of salvaging the data on Drive4.  Don't mind the new "sdj" - that's just a 16tb USB drive I plugged in to backup whatever I'm able of my most critical data in case things go further south.  😕

tortuga-diagnostics-20250304-2109.zip

On a related note, initially both my parity drives showed rebuilding, but assigned - however it seems Parity1 just went disabled again 😞 

Post new diagnostics.

I did with my last update, but since then, the parity rebuild failed, and the current state is that the array is stopped and all drives are unassigned so I could try and mount them individually to recover whatever I could.  If you think there's any chance of fixing it, let me know, I guess.

tortuga-diagnostics-20250305-0809.zip

Just now, Juliet said:

all drives are unassigned so I could try and mount them individually to recover whatever I could

No point to mount them individually, they can just be assigned to the array, with or without parity.

Does order matter?  Do they have to be assigned the same drive numbers?  I took a screenshot, so I can reassemble it.

Order only matters if you have parity2. Make sure you don't assign any data disk to any parity slot.

I've reassembled the array and restarted it.  Here's the current Diagnostic 

tortuga-diagnostics-20250305-0832.zip

After parity finishes rebuilding we can try check filesystem on disk4 again, but might not be very successful.

 

Do you have backups of everything important and irreplaceable?

Yeah, I figured as much.  Short of some sort of forensic recovery, I doubt I'm getting anything off of Disk4.  No, I didn't have any of the stuff on there backed up.  Currently rsyncing the most important stuff from the array to an external drive (which was something on my list of things to do this week, before all the trouble started), but I lost all my docker containers and a bunch of media.  

We didn't lose power or anything...I have no idea what happened to cause this initially, but I'm sure I trashed the FS when I inadvertently started a rebuild of Disk 4.