Replaced a bad drive in my array. It said it was unmountable so I formatted it. Now the data is gone. Can I fix it using the old drive?

[s449]

I had a bad drive in my array. Whole thread about it here. Essentially writing data was causing XFS Filesystem corruption and it was making a mechanical chirping sound.

 

So I purchased a new drive to replace it, a WD EasyStore 12TB. I attached the external drive to Unraid through USB to do a preclear, it went fine. I shucked the drive and replaced the bad drive with it. When I started Unraid it said disk1 was missing, as expected, so I chose the new disk to replace disk1. It immediately started a data rebuild however I noticed it said the new disk was "unmountable". I don't remember ever seeing that when I've done disk replacements before. I figured I don't think the data rebuild was doing anything if the disk was unmountable, maybe it was rebuilding parity as if the disk was gone. It also said I needed to format the new disk to use it. I'm pretty sure formatting the disk so it can be used in Unraid is a normal step in the process but I was just surprised adding the disk immediately went to rebuild and unmountable. So I formatted the disk. But now the disk is completely empty yet it kept doing a data rebuild.

 

I immediately stopped the array. I have the old drive still. It works, it just corrupts when new data is added. But the files are still there.

 

I'm thinking what I need to do is put the old drive back in, then have the parity drive rebuild itself from scratch, once that's done, I can attach the new drive back into the array. Is that correct? How would I do that exactly? Any help is appreciated. I'm worried of messing things up more.

 

Edit: Put the old drive back in. Unraid had a blue square next to that disk and it said "new device". I started the array in maintenance mode and then it said disk 1's "device contents emulated" and I got a notification saying "Disk 1, drive not ready, content being reconstructed". But no parity sync/data rebuild was happening. I tried to see if I could start it without the parity drive, hoping I could rebuild the parity drive as if I never removed the old drive to begin with, but it said invalid configuration.

 

I'm thinking tools > new config is what I might need. That way it'll force a rebuild of parity, if I'm understanding correctly. Because as my server is right now, all my data drives are okay. It's just my parity is now wrong and confused.

 

Edit 2: Did New Config. Started the array. It started a parity rebuild. Disk 1 came back as "Unmountable: not mounted" and the file system was "auto". I changed the file system back to "xfs" and ran xfs_repair with the -n flag and now it's just stuck at:

 

Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!

attempting to find secondary superblock...
.......................................................................

 

I'll just let it keep running I guess.

 

UPDATE: Thread 3:

 

Edited March 2, 2021 by s449

[Vr2Io]

1 hour ago, s449 said:

I figured I don't think the data rebuild was doing anything if the disk was unmountable

Not true, parity have no idea of filesystem / file, it just a XOR result in byte by byte of member disk. So any corrupt in member disk or unsafe array stop could corrupt the parity.

 

Pls forget anything relate rebuild, the most importance thing is copy out the file to a normal disk.

 

- If your file already corrupt in original disk, you shouldn't expect rebuild could correct for you.

- If original disk still mountable, pls copy out its content ASAP

- I am not sure does XFS apply UUID stuff, as emulate disk and original disk have same UUID, in normal practice you need change the UUID in original disk.

 

Once copy out the content, then you actually need was "new config" without parity assign, then add back the parity and rebuild parity.

[trurl]

Really wish you had asked for advice before doing anything. Pretty much everything you did was wrong. NEVER format a disk you think should have data on it. Format is a write operation. Unraid treats that write operation exactly as it does any other, by updating parity. So after you format a disk in the parity array it is empty and rebuilding an empty disk can only result in an empty disk.

 

Would have been better if you had tried to work with the original disk as an Unassigned Device. Not sure what state it is in now since you put it back in the array.

 

Maybe too late for anything and since you must have rebooted a lot of what you did will not be in syslog anymore. But we will know more about your situation if you post your diagnostics.

 

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

 

 

[s449]

3 minutes ago, trurl said:

Really wish you had asked for advice before doing anything. Pretty much everything you did was wrong. NEVER format a disk you think should have data on it. Format is a write operation. Unraid treats that write operation exactly as it does any other, by updating parity. So after you format a disk in the parity array it is empty and rebuilding an empty disk can only result in an empty disk.

 

Would have been better if you had tried to work with the original disk as an Unassigned Device. Not sure what state it is in now since you put it back in the array.

 

Maybe too late for anything and since you must have rebooted a lot of what you did will not be in syslog anymore. But we will know more about your situation if you post your diagnostics.

 

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

 

Attached diagnostics.

 

Yeah you're right. I mean I've done this process a few times before of replacing a disk for a bigger disk and it's always gone through swimmingly. When I saw the "unmountable" disk this time after replacing I just was stupid and didn't think to ask anyone if it was a bad idea or not to format it. I just did it. From there I've just been impatient and talking with people on the Unraid Discord in regards to what I've done since then in my edits.

 

As it stands right now I have an unsynced parity and a data disk with an even more broken(?) xfs file system. I only found one thread about my current xfs_repair logged error:

From what it looks like it's just something I need to wait out.

 

I really appreciate any help.

apollo-diagnostics-20210227-1751.zip

[trurl]

I don't think there is any point in trying to repair the disk in the parity array when parity is invalid. Either rebuild parity then try to repair the disk after parity is rebuilt, or New Config without that disk, rebuild parity, and try to repair it outside the array.

[trurl]

That thread you linked is pretty old, but go ahead and let it continue for a while to see if anything works out.

[s449]

5 minutes ago, trurl said:

I don't think there is any point in trying to repair the disk in the parity array when parity is invalid. Either rebuild parity then try to repair the disk after parity is rebuilt, or New Config without that disk, rebuild parity, and try to repair it outside the array.

 

It looks like in maintenance mode while I let it run the xfs_repair I can click "sync" start a parity sync. So I can at least let the parity start to build up again while it tries to repair the file system. Just not sure if that's a bad idea to do at the same time.

 

That's also a good idea of another option of just letting the data be lost and get my server back up and running with the new drive but then try to get the data back by plugging the old drive into my desktop or something, I assume. Just no clue how I'd go about that honestly.

[trurl]

1 hour ago, s449 said:

plugging the old drive into my desktop

Probably can't work with XFS filesystem in your desktop. Unassigned Devices is what I had in mind, but doing it in or out of the array won't matter much as far as the repair goes, you just don't have valid parity until parity is rebuilt.

 

1 hour ago, trurl said:

go ahead and let it continue for a while to see if anything works out.

Any progress with finding a superblock?

[s449]

18 minutes ago, trurl said:

Probably can't work with XFS filesystem in your desktop. Unassigned Devices is what I had in mind, but doing it in or out of the array won't matter much as far as the repair goes, you just don't have valid parity until parity is rebuilt.

 

Any progress with finding a superblock?

 

I have that plugin, so yeah I can keep this drive in my array at least and look into it more if I go the route of just replacing with the new drive and starting from zero. I RMA'd it but I have 30 days to send it back. I was thinking like a software UFS Explorer: https://www.ufsexplorer.com/ as far as trying to recover the files on my desktop. Or a similar software. I mean, at the very least I want to just see what files were on it so I know what to redownload. Like I mentioned my important "backup" share is on Backblaze B2 at least so the big loss is just media I can find online again.

 

No luck on superblock. Same exact error log, stuck on "attempting to find secondary superblock...". But I went ahead and let the parity rebuild run. Might as well. I also ran a SMART short self-test which has been stuck at 90% for like an hour. I can't imagine that's a good sign. Normally it takes 10 minutes max.

 

Thank you for the help so far!

Edited February 28, 2021 by s449

[s449]

2 hours ago, trurl said:

Any progress with finding a superblock?

 

It found a secondary superblock.

 

verified secondary superblock...
writing modified primary superblock

 

What proceeded was over 10,000 lines of logs. But it completed without any error. I stopped the array and started it again. The disk amazingly mounted just fine. However there's one issue...the entire disk is in the lost+found. I have the logs and it does reference some files with inode numbers but I really don't know how feasible this is to try to save since we're talking a few thousand files.

 

Luckily I have a backup of my most important stuff. But I don't think there's any way to rebuild this file system to be what it once was.

 

Welp. Now we know the data is lost for sure. I'm going to let the parity finish rebuilding then re-add the new drive. I guess I could slap the new drive in and do a New Config since I have nothing to lose but why bother with another sketchy maneuver. Might as well keep the lost+found for a while just in case I can do something with it.

 

Thank you for the help everyone. I hope this thread and my last thread can be helpful to anyone who finds them at the very least on some lessons on what not to do lol.

[itimpi]

If you do try and sort out the Lost+Found folder then the Linux ‘file’ command can at least tell you what type of content (and thus the likely file extension) each file contains.