Disk failure during rebuild :(

[alex92]

Hello, 

 

I am running unraid at the latest version with 5 disk (1 parity disk). My 2 older disks (4TB, disk1 and disk3) started having read errors at a high rate. I replaced one of them (disk1)  with a new 6TB and started the rebuild. The rebuild was VERY slow at 600KB/s. I paused the rebuild, checked the syslog which were littered with read errors from disk3 (the other older drive which had issues). At this point i tried to restart the rebuild but it never started again. Reboot command did not work either.  The server was pretty much unresponsive. I rebooted the server with the power button. Started the array again to start the rebuild, but disk3 never mounted. Now the server is stuck on "mounting" the array. It's been hours. The reboot function does not work either. 

 

Now it looks like disk3 is fried. Disk1 (the one that i removed and was rebuilding) is still working (with some read errors). I was hoping to be able to put disk1 back in the array, and rebuild disk3 first with the hopes that disk 1 is still good enough to last the rebuild. But since it was replaced, the array is expecting the new disk. Can i manually update the configs to get the original array back ? I have a unraid backup 9/22 which has the DISK_ASSIGNMENTS.txt file.

 

attached is diagnostics file from last night during the rebuild and one from this morning once disk3 wouldn't mount anymore. 

 

Any advice is welcome. 
 

THanks

Alex.

 

nas-diagnostics-20191002-0655.zipnas-diagnostics-20191002-1452.zip

[trurl]

Parity disk doesn't look so good either. I would put disk3 and parity replacement ahead of disk1.

 

What you need is the invalidslot command. Instead of 4 in the command in this linked post you would put 3 to rebuild disk3:

 

https://forums.unraid.net/topic/77771-multiple-errors/?do=findComment&comment=720061

 

Take a look at that but don't proceed for a while to see if someone else has anything to add. I always like to ping @johnnie.black at times like this.

 

Do you have another copy of anything important and irreplaceable?

[alex92]

Thanks for the reply.

What log indicate that the parity disk is having issues ? The parity disk is ST6000VN0041-2EL11C_ZA17Z98A (sdb) (disk0), looking at the logs, i don't see any errors from that device. 

Either way step number 1 is to rebuild disk3. But to do that i need to put disk1 back in the array, however, the array is expecting the new 6TB disk as disk1 . Do i need to restore the unraid backup from 9/22 to get the original config, and then issue the invalidslot command ? 

 

wrt the content, it is mostly a large plex library. Would be a huge bummer to lose it. There is also some photos and a phd thesis on there. 

[trurl]

Looking again, I think it is probably just a connection problem with parity. Lots of errors in the syslog on disk0, and parity isn't showing SMART in your 2nd diagnostics. I think it looks OK in the 1st diagnostics though.

[trurl]

17 minutes ago, alex92 said:

Do i need to restore the unraid backup from 9/22 to get the original config, and then issue the invalidslot command ? 

When you do New Config before issuing the invalidslot it will let you assign the disks as you want.

 

17 minutes ago, alex92 said:

Would be a huge bummer to lose it. There is also some photos and a phd thesis on there. 

You must always have at least one other copy of anything important and irreplaceable. Parity is no substitute for backups. Plenty of ways to lose files besides disk failure, including user error. You don't have to backup everything, but you do have to decide what you can't afford to lose.

[trurl]

Looks like nobody else want to chime in.

 

I guess you might as well proceed with the invalidslot. Be sure to follow the instructions in that post exactly, except for disk3 instead of 4 as noted. Be sure to ask if you have any problems or questions.

[alex92]

Hey trurl ! 

 

Sorry for the long silence. I was away from home for a while. I thought i should replace all the SATA cables, before doing invalidslot just in case some of these errors were due to a connection issue. So last night, i replaced all cables, reseated the controller cards, left the new 6TB drive in disk1, fired up the server, mounted the disks, which mounted right away, and started rebuild the array. The rebuild has been running at over 100MB/s, no errors, and will finish in 17 hours. It is 97% done now.  Woot ! 

 

Thanks again for your help trurl ! much appreciate it

I might just add a second parity drive for added redundancy and piece of mind. 

 

Alex.