2 failed drives

[rogerza]

Hi,

 

I have a system running with 7 X 2TB drives + 1 128gb ssd, all older drives from a previous server.

All drives are running on onboard sata.

 

One of the drives, drive 4, failed so I ordered a new one, due to no stock in area drive took 2 weeks to deliver, set for delivery tomorrow.

The problem is that in that time another drive has failed, joys of using older hardware.

 

So the problem now is drive 4 is actually back as an unassigned drive, if I try to assign it to drive 4 it says it's a new drive, so it seems to have forgotten that it was drive 4.

Drive 3 just says format so that seems to be totally broken, before dying it was also showing errors.

 

I can mount drive 4 using a plugin and browse the data on it so should be able to get the data off, the problem is that I can't start the array.

 

What would be the best way to get things working without losing data?

Is it possible?

 

I ordered another 2TB drive today so will have 2 fresh drives to replace the old ones but not sure if adding 2 blank drives is enough to start the array with 2 failed?

 

Thank you.

[trurl]

Tools - Diagnostics. Attach complete zip.

[christopherrutherford]

Been having the same issues, with the exception that my drives are new, and I've had no less than four drives to fail... Two of which are being replaced, and One drive was recently replaced with one that has just now failed......

 

I can't upload the diagnostics as there are file size limitations. what could be the issue? why are my drives failing? seriously? I've never had drives fail like this!

[Squid]

Been having the same issues, with the exception that my drives are new, and I've had no less than four drives to fail... Two of which are being replaced, and One drive was recently replaced with one that has just now failed......

 

I can't upload the diagnostics as there are file size limitations. what could be the issue? why are my drives failing? seriously? I've never had drives fail like this!

You really should open up a new thread as your issues may be different than the OP's.  But, when your diagnostics are larger than the maximum, upload them to drop box or something and then supply a link.

 

As an aside, if you're only used to Windows, etc then you will find that it ignores hard drive errors left, right and center.  Ever wonder why you've got to reinstall the odd program on windows because it just doesnt work right anymore?  Or windows has started with some strange issues and miraculously a reformat / reinstall of it fixes the problem?

 

 

[S80_UK]

I had two drives "fail" today on my backup server.  Actually, they didn't.  As happens sometimes when we try to connect so many drives to a power supply, one of my power splitters had become intermittent.  unRAID couldn't  access the drives properly, even though they had been seen at boot up and given a /dev/sdx assignment.  After some poking around in the wiring, one of the drives came back.  The other was also visible, but unRAID had already marked it as faulty and was emulating it from parity data with the other drives.  unRAID is now rebuilding the drive that was worst affected by the faulty power splitter.

 

This may be different from the OP's case, but when multiple drives fail at the same time or nearly so, there's sometimes a reason other than the drives themselves.  Just a thought...

[trurl]

I had two drives "fail" today on my backup server.  Actually, they didn't.  As happens sometimes when we try to connect so many drives to a power supply, one of my power splitters had become intermittent.  unRAID couldn't  access the drives properly, even though they had been seen at boot up and given a /dev/sdx assignment.  After some poking around in the wiring, one of the drives came back.  The other was also visible, but unRAID had already marked it as faulty and was emulating it from parity data with the other drives.  unRAID is now rebuilding the drive that was worst affected by the faulty power splitter.

 

This may be different from the OP's case, but when multiple drives fail at the same time or nearly so, there's sometimes a reason other than the drives themselves.  Just a thought...

Not only multiple drive problems, but even single drive problems, are usually not the drive at fault. Diagnostics are what we need to decide.

[rogerza]

Here you go.

 

Getting 2 new drives tomorrow so can then use them if needed.

slyder-diagnostics-20160516-1858.zip

[trurl]

Syslog has disk3 (sdg?) missing, disk4 removed. Lots of errors on ata6, which I think is sdg. Several of your SMART don't look so good. Are you not getting notifications on any of these? I think you will need more than just the 2 disks on order to get back to a reliable server. Do you have any backups? The first priority would be to copy anything you can that you consider irreplaceable to another computer.

 

disk0 (parity)

Serial Number:    5YD298CK
  5 Reallocated_Sector_Ct   0x0033   092   092   036    Pre-fail  Always       -       5376
183 Runtime_Bad_Block       0x0032   040   040   000    Old_age   Always       -       60
187 Reported_Uncorrect      0x0032   068   068   000    Old_age   Always       -       32
197 Current_Pending_Sector  0x0012   100   097   000    Old_age   Always       -       24
198 Offline_Uncorrectable   0x0010   100   097   000    Old_age   Offline      -       24

 

disk1

Serial Number:    6YD0456V
183 Runtime_Bad_Block       0x0032   040   040   000    Old_age   Always       -       60

 

disk5

Serial Number:    5YD294VF
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       64
183 Runtime_Bad_Block       0x0032   040   040   000    Old_age   Always       -       60
187 Reported_Uncorrect      0x0032   034   034   000    Old_age   Always       -       66

 

disk6

Serial Number:    5YD296X0
183 Runtime_Bad_Block       0x0032   040   040   000    Old_age   Always       -       60

[rogerza]

Ok, so disk 3 (sdg) is the one that is totally broken, i.e. can see it in list but can't mount it at all, it was the second one to fail.

 

Disk 4 is I am assuming sdc, if it's showing as removed how would I go about adding it again?

 

In terms of getting the server back to full functionality, I will be ordering the remaining drives over the next week or so but just wanted to get the array online as I currently can't access it to copy any important stuff off. Is there another way of getting stuff off the drives without getting the array back online?

 

SMART wise Disk 1 and 5 both have warning so will be next replacements, rest still have green thumb.

 

I am able to mount sdc and see data so would I be able to either assign it back as disk 4 and put a new disk onto disk 3 to get array online, copy data off it then replace it too or is that not possible?

Could I put in both new drives to get array online and then just copy back data from the sdc drive?

Assuming though that with 2 new drives the parity is broken so the array can't be rebuilt properly?

 

Then for Disk 1, if I put a 4TB drive in to replace that for parity would that mean a higher level of redundancy as it's larger than the other drives or is redundancy always based on 1 drive max failing?

[trurl]

Post a screenshot of Main showing your Array Devices.

[rogerza]

Here you go.

[trurl]

sdc - do you know if this was the serial of Drive 4?

Serial Number:    5YD28DZQ
183 Runtime_Bad_Block       0x0032   040   040   000    Old_age   Always       -       60

I don't think the Runtime Bad Block numbers on any of these mean anything.

 

Do you have backups? If not you should try to copy anything really needed to another computer before trying anything else.

 

You might be able to trick the system into rebuilding disk3. Not clear if it would work since it is not clear what state parity is likely to be in with 2 drives out. It may not really be in sync with disk4 since that was the first one out and it may have been writing parity while emulating disk4. Once disk3 went out parity wouldn't have been used anymore. Also, SMART for parity is perhaps the worst of the drives. If you want to try it anyway you could:

 

New Config - Assign a new drive to disk3. Assign all other drives exactly as before. Check the box saying parity is already valid, then start the array.

 

Stop the array, unassign disk3, start the array. This will make it think disk3 is missing and needs to be rebuilt.

 

Stop the array, reassign disk3, start the array and let it rebuild disk3.

[rogerza]

I just want to double check, if I go Tools ->New config is says that the drives will be erased, that sounds bad?

Will it actively erase the disks if I do that or is that why I need to tick that parity is ok so that it doesn't?

 

I tried just assigning a new drive to disk 3, the old disk 4 to drive 4 but that still wont let me start the array, too many failed drives.

[trurl]

I just want to double check, if I go Tools ->New config is says that the drives will be erased, that sounds bad?

Will it actively erase the disks if I do that or is that why I need to tick that parity is ok so that it doesn't?

 

I tried just assigning a new drive to disk 3, the old disk 4 to drive 4 but that still wont let me start the array, too many failed drives.

I have never seen New Config say it is going to erase anything. Normally all it does is rebuild parity. If you check the box it won't even do that.

 

If it is saying you have too many failed drives you must not be doing New Config.

 

Post a screenshot.

[rogerza]

Sorry no I didn't read the page right.

 

I tried the new config but it still kept saying it can't start so I took the plunge and put my 2 new drives as parity and drive 1 and started then mounted other drives with mount plugin and first copied data off 1 drive, unmounted and added to array then next etc and as I went removed the other drives that had smart issues so am now up and running with just healthy drives.

 

Whatever data was on disk 3 is gone but guessing that with parity having kicked in on disk 4 already disk 3's data was gone regardless so all up and running.

 

Thank you for your help!