2 data drive failures and parity swap

[m1a8x2]

Hi everyone,

 

I'm having some issues with my array. My dog knocked over my server while I was working on it. When I started the server back up, it said disk 4 was not installed. I checked all cables, etc. and it still wasn't showing up. I removed the drive and tried seeing if another computer would recognize it and it made some terrible sounds. When I started the server back up with this drive removed, I now see that disk 5 says "Unmountable: No file system". Disk 4 now also says this as well.

 

I've ordered two 8tb drives, so I'll need to do a parity swap as well I believe? Can anyone help me figure out how to fix this file system error and what order I should be doing everything?

 

Thanks in advance!

krieger-diagnostics-20201228-1233.zip

[JorgeB]

22 minutes ago, m1a8x2 said:

I removed the drive and tried seeing if another computer would recognize it and it made some terrible sounds.

Then you have a problem, because disk5 is also failing, single parity can only tolerate one failure.

[m1a8x2]

I was still able to access everything a couple days ago and disk5 seemed fine. Does it look like it's gone too? At that point there's no recourse?

[JorgeB]

For disk5 you can try ddrescue, depending on how bad the damage is it might recover a lot or most of the data, you could also then try rebuilding disk4 with the clone in place, but there always be some corruption.

[m1a8x2]

3 minutes ago, JorgeB said:

For disk5 you can try ddrescue, depending on how bad the damage is it might recover a lot or most of the data

Would I need to do a parity swap first? My parity is 5tb and my data drives are all 4tb. I have two new 8tb drives arriving tomorrow.

[JorgeB]

To use ddrescue and then re-use the clone in the array you'd need to use a disk of the same size, or Unraid won't accept it.

[m1a8x2]

Well crap. So I should order another 4tb and use ddrescue... then parity swap and then swap disk4 and rebuild?

[JorgeB]

Yes, with the clone you can then do a new config, you still need a temp disk4, but you can use the old disk5, then disable it and do the parity swap, when the clone is ready ask and I can post detailed instructions, but like mentioned rebuild will be as corrupt as there are errors in the clone operation.

[m1a8x2]

Okay, thank you. I'm going to go buy a 4tb locally and run ddrescue. I'll report back when that's done.

[m1a8x2]

3 hours ago, JorgeB said:

Yes, with the clone you can then do a new config, you still need a temp disk4, but you can use the old disk5, then disable it and do the parity swap, when the clone is ready ask and I can post detailed instructions, but like mentioned rebuild will be as corrupt as there are errors in the clone operation.

I have the new drive in the server and am ready to use ddrescue... I'm not entirely sure I understand the command yet so I haven't started it until I read the instructions again.

[m1a8x2]

@JorgeB I've had ddrescue running for 11 hours now. 

 

pct rescued:   36.09%, read errors:1150916909,  remaining time:         n/a
                              time since last successful read:      5h 40s

[JorgeB]

Bad news, that's a lot of errors, and it will also affect the other disk rebuild.

[m1a8x2]

4 minutes ago, JorgeB said:

Bad news, that's a lot of errors, and it will also affect the other disk rebuild.

Is it possible to know what data is still intact and copy it manually to the new drive? Would I be unable to rebuild disk4 if I did that?

[JorgeB]

8 minutes ago, m1a8x2 said:

Is it possible to know what data is still intact and copy it manually to the new drive?

After the clone you can do that, assuming the disk mounts.

 

9 minutes ago, m1a8x2 said:

Would I be unable to rebuild disk4 if I did that?

Unrelated, problem with rebuilding disk4 is all the errors on that disk will result in the same corrupt sectors on the rebuilt disk, so more errors, more corruption.

[m1a8x2]

18 minutes ago, JorgeB said:

Unrelated, problem with rebuilding disk4 is all the errors on that disk will result in the same corrupt sectors on the rebuilt disk, so more errors, more corruption.

Disk4 is the one that's completely dead, so wouldn't it be able to be emulated/rebuilt from parity? I don't know what I'm talking about, I'm sure... but just want to make sure I'm on the same page.

[JorgeB]

25 minutes ago, m1a8x2 said:

Disk4 is the one that's completely dead, so wouldn't it be able to be emulated/rebuilt from parity?

Parity plus all the other disks.

[JorgeB]

Just to expand, parity disk itself don't have any data, it needs all the other data disks to be read successfully for a rebuild.

[m1a8x2]

So what's my best approach here? What's my endgame - will I be able to restore my array in any way or am I completely screwed here?

[JorgeB]

Best chance to recover any data from disks 4 and 5 is to let ddrescue finish and then rebuild, though there will always be some data loss on both, possibly a lot due to all those errors, if you just want to bring the array online and forget about those disks you can do a new config.

 

Also in the future consider adding a second parity, small price to pay for the added redundancy, even for small arrays, but still not a substitute for backups.

[trurl]

Do you have Notifications setup to alert you immediately by email or other agent as soon as a problem is detected? If you let things go until you have multiple problems then it can be difficult or impossible to fully recover.

[m1a8x2]

1 hour ago, trurl said:

Do you have Notifications setup to alert you immediately by email or other agent as soon as a problem is detected? If you let things go until you have multiple problems then it can be difficult or impossible to fully recover.

No I don't have any email notifications unfortunately. I need to get those setup once I'm out of the woods here.

 

2 hours ago, JorgeB said:

Best chance to recover any data from disks 4 and 5 is to let ddrescue finish and then rebuild, though there will always be some data loss on both, possibly a lot due to all those errors

I can live with some data loss I suppose. I'm okay with losing media files, but it's pictures and files from school/work I don't want to lose. So far, when I connect to my shares on another PC it appears most all of my important files are safe.

 

2 hours ago, JorgeB said:

Also in the future consider adding a second parity, small price to pay for the added redundancy, even for small arrays, but still not a substitute for backups.

I'm assuming this parity would need to be the same size as the other parity drive? Right now I have a 5tb parity, and five 4tb data drives. I just added a 4tb drive for this recovery and I have two new 8tb drives to swap into the array/parity.

[itimpi]

Parity drives do not need to be the same size - but they must be at least as large as the largest data drive.

[JonathanM]

1 hour ago, m1a8x2 said:

So far, when I connect to my shares on another PC it appears most all of my important files are safe.

Since you have access, NOW is the time to back them up to another location, before you start changing things around and possibly getting in a worse situation.

 

Anything important should always be backed up elsewhere, preferably 2 other places if it's truly irreplaceable and valuable. Unraid or any RAID is not backup, it's high availability so you can still access your files when a drive goes down, and rebuild to a new drive.

[m1a8x2]

15 minutes ago, jonathanm said:

Since you have access, NOW is the time to back them up to another location, before you start changing things around and possibly getting in a worse situation.

Good call. I'm going to dump as much as I can onto another external drive.

[m1a8x2]

ddrescue has finished.

 

rescued: 1444 GB, tried: 2556 GB, bad-sector: 1175 GB, bad areas: 1765

Current status
     ipos:    4000 GB, non-trimmed:        0 B,  current rate:       0 B/s
     opos:    4000 GB, non-scraped:        0 B,  average rate:       0 B/s
non-tried:        0 B,  bad-sector:    2556 GB,    error rate:  27596 kB/s
  rescued:    1444 GB,   bad areas:     1764,        run time: 11h 43m 30s
pct rescued:   36.09%, read errors:2697716168,  remaining time:         n/a
                              time since last successful read:         n/a
Finished      

 

What's next? I now have SMART warnings for disk 2 about current pending sectors. Is that drive failing now too or is that read errors from running ddrescue or something else?