Failed drive, unmountable, contents not emulated

[johnnyb613]

Hello!

 

I am running a Unraid server with 1 pairty drive, 6 data drives, and 1 cache.

 

I had a drive fail, it seems to have been a hardware failure, a SMART report shows 2814 "uncorrectable error in data" errors,  

ER -- ST COUNT  LBA_48  LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 51 00 00 00 01 7b 64 94 37 40 00  Error: UNC at LBA = 0x17b649437 = 6365156407

 

And the recover software I am using gave me this: 

Warning: Storage defect was detected in "read" operations of:

This error indcates the storage device failed to complete a read request due to error.

 

The power blinked and for whatever reason the UPS didn't hold, when it came back up disk 2 was showing as unmountable. I didn't do things the correct way, I am so used to UnRaid being foolproof that I didn't run dianostics to post when I saw the error, I just shut down and hoped for the best. (I've been up and running since 2015 with no issues other than the occasional drive failures where is disables the drive, emulates it, and I run a couple of days until the new drive comes in and I just swap and it rebuilds and I am fine)

 

I referenced the "check file systesm" wiki and ran through the steps to check an XFS drive. It couldn't find the superblock or any secondary superblock.

 

Currently the array is started with the one disk missing. Not all files are showing up on the shares (but do exist on the remaining disks) but I have read this is normal when you have corruption. For example, I have a blu-ray directory where I store an ISO backup of all my blu-rays. If I browse to l33tnas\data\blu-ray the directory shows as empty. If I browse to \\disk1 (or 3, 6, 7, or 8\blu-ray the data is there.

 

I have all data from the other 5 disks backed up to external drives. I am running a recovery on the dead drive with UFS Explorer to get any of the data I can.

 

Where do I go from here? Is there any way to recover what I have without starting from scratch? I see a document about replacing the boot drive and adding existing data drives back in to the array, all my drives are clearly labeled so I know which one is the parity disk, but I am more concerned about the config and all the shares, I don't really want to try to reinvent the wheel on all of those.

 

I am attaching the current diagnostics, if there is any other info I need or if I need to pop the failed drive back in and run diagnostics again let me know. 

 

ps. If it looks odd in my config that I have disks 1 2 and 3 and then 6 7 and 8 there is a reason. When the array started getting too large I stole a couple of drives from it and started ANOTHER Unraid server  I just never took the time to renumber the drives (although it does bother me)

 

Thanks!

 

 

 

l33tnas-diagnostics-20230422-1008.zip

[JorgeB]

Start the array in maintenance mode and post the output of:

xfs_repair -v /dev/md2

 

[johnnyb613]

Here is the xfsrpair log as requested. I also attached new diagnostics since the drive is back in the system.

 

Thanks!

xfsrepair.txt l33tnas-diagnostics-20230426-2136.zip

[JorgeB]

xfs_repair is unable to fix the filesystem, you can use a file recovery util like UFS explorer or re-format the disk and restore from a backup if available.

[johnnyb613]

Yes. As I mentioned in my earlier post, I have already backup up all the available data on all the remaining drives in the array and used UFS Explorer to recover all of the data I could off of the failed drive. My question is, where do I go from here?

 

Currently the array is started with the one disk missing. Not all files are showing up on the shares (but do exist on the remaining disks) but I have read this is normal when you have corruption. For example, I have a blu-ray directory where I store an ISO backup of all my blu-rays. If I browse to l33tnas\data\blu-ray the directory shows as empty. If I browse to \\disk1 (or 3, 6, 7, or 8\blu-ray the data is there.

 

I'm guessing my first step would be to replace the failing drive since it had thousands of SMART errors. But once I do that how do I fix it so the shares can see all the files on the other drivers? Do I need to wipe everything and copy all files from scratch like I was building a brand new system?

[itimpi]

2 minutes ago, johnnyb613 said:

If I browse to l33tnas\data\blu-ray the directory shows as empty. If I browse to \\disk1 (or 3, 6, 7, or 8\blu-ray the data is there.

This implies to me that in the first case you are looking in the ‘data’ share, whereas the other one is looking in the ‘blue-ray’ share?

 

4 minutes ago, johnnyb613 said:

But once I do that how do I fix it so the shares can see all the files on the other drivers?

Shares are effectively the aggregate of  any top level folder with a given name across all array drives and pools so you should not need to do anything if the files are in the correct location on each drive.

 

if you think it all files are in the correct location then it can be worthwhile to run a filesystem check across all your drives as corruption at the filesystem level has been known to upset user shares being visible.

[JorgeB]

8 minutes ago, johnnyb613 said:

My question is, where do I go from here?

If data from disk2 was already recovered next step is to format it, all array data should become accessible then, except any data that was on disk2, then restore that data from the backup.

[johnnyb613]

17 minutes ago, itimpi said:

24 minutes ago, johnnyb613 said:

If I browse to l33tnas\data\blu-ray the directory shows as empty. If I browse to \\disk1 (or 3, 6, 7, or 8\blu-ray the data is there.

This implies to me that in the first case you are looking in the ‘data’ share, whereas the other one is looking in the ‘blue-ray’ share?

 

My fault. Typo. If I browse to l33tnas\data\blu-ray I see an empty directory, if I browse to disk1\data\blu-ray the data is there.

 

20 minutes ago, itimpi said:

if you think it all files are in the correct location then it can be worthwhile to run a filesystem check across all your drives as corruption at the filesystem level has been known to upset user shares being visible.

 

So I would run the xfs_repair -v /dev/(drive number) for each disk correct?

 

21 minutes ago, JorgeB said:

If data from disk2 was already recovered next step is to format it, all array data should become accessible then, except any data that was on disk2, then restore that data from the backup.

 

After I have checked all my other disks (I might as well at this point) should I reformat the bad drive and let it rebuild and the array and THEN replace it, or can I skip that step and just pop a new drive in that slot?

 

 Thanks everybody for helping me work through this!

[trurl]

Format the emulated, missing disk to make it an empty filesystem. Make very sure it is the only disk listed for formatting. Then assigning a new disk to that slot will rebuild the newly formatted filesystem onto the new replacement disk.

 

Then restore data from backup.

 

[itimpi]

1 hour ago, johnnyb613 said:

So I would run the xfs_repair -v /dev/(drive number) for each disk correct?

You should it via the GUI as doing it from the CLI is error prone.

[johnnyb613]

4 hours ago, johnnyb613 said:

if you think it all files are in the correct location then it can be worthwhile to run a filesystem check across all your drives as corruption at the filesystem level has been known to upset user shares being visible.

 

You were exactly right! I ran a check on all the other drives, disk 7 had some errors and as soon as those were fixed the data in my shares is showing up again!