1 hour ago, Pyro said:

if I can get my array back to the broken-but-still-usable state it was in, can I go back to my original plan?

I don't remember if we actually covered this in this thread and a quick skim didn't turn it up, but are you aware that the parity disk by itself doesn't contain any of your data?

 

Every bit of parity plus ALL other array disks must be reliably read to reliably reconstruct every bit of a missing disk.

Actually there are a couple of data disks you have that are OK so you could also include them in the new array.

You don't currently have any disks emulated since you don't have any disks assigned in the new install. If you can't resurrect that other disk we will have to work through this a bit at a time to try to get back to that emulated situation. Unfortunately we have no idea what in your config was preventing you from going with the old install. We should be able to get you back to that situation though with some tricks that you will have to follow exactly.

 

Probably though, if you can resurrect the actual disk, it is going to give you a better chance of getting its data than the emulated disk will.

New diagnostic file after I fixed the key and checked all of the SATA and SATA power cables.

48 minutes ago, trurl said:

Actually there are a couple of data disks you have that are OK so you could also include them in the new array.

I also have another 1tb 2.5" drive that I could use at least temporarily.

55 minutes ago, trurl said:

Every bit of parity plus ALL other array disks must be reliably read to reliably reconstruct every bit of a missing disk.

Ah, I did not know that.

tower-diagnostics-20181128-1051.zip

We could try your original plan and maybe get what we could that way, then go to my plan as a fallback to get whatever we couldn't get your way. It would stress all the already stressed disks though, since as mentioned all other disks must be read to emulate a disk.

 

Whatever we do is going to take some time and effort with no guarantee of complete success.

 

Just waiting for a SMART report on the missing disk if you can get it.

 

I see you posted just as I was. I'll take a look.

Looking at the drop-down menu, it looks like the emulated drive is missing. I'm assuming dead. At this point I'm completely at your mercy I think. If I can copy everything from the failed drive via emulation then everything else should be recoverable as well, right?

 

I do understand there's no guarantee. Anything saved is better than nothing saved.

Edited November 28, 20187 yr by Pyro

This is the disk that wasn't reporting SMART in your previous diagnostics.

Serial Number:    9VS05DG9
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  5 Reallocated_Sector_Ct   PO--CK   100   100   036    -    4
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0

That must be the other disk showing the Dashboard warning but it isn't too bad. Probably OK to use for now.

 

4 minutes ago, Pyro said:

Looking at the drop-down menu, it looks like the emulated drive is missing. I'm assuming completely dead. At this point I'm completely at your mercy I think. If I can copy everything from the failed drive via emulation then everything else should be recoverable as well, right?

 

Yes we are indeed missing a hard drive in the SMART. I will start a new post for the next step.

Shut down and put your flash in your PC. There are a couple of files we need to get from that backup config you took earlier.

 

One file is named disk.cfg. Take a look at it and if it has Autostart as yes change it to no.

 

The other file is named super.dat. It contains your disk assignments. It isn't plain text so don't try to open it. I think if you can boot with that we may be back to the missing disk emulated.

3 hours ago, trurl said:

I think if you can boot with that we may be back to the missing disk emulated.

Progress!

 

I'm guessing that my next step should be starting the array and copying everything on disk 3 to my PC since I have room? (2.5tb free on my desktop)

 

Just now, Pyro said:

Progress!

 

I'm guessing that my next step should be starting the array and copying everything on disk 3 to my PC since I have room? (2.5tb free on my desktop)

 

I was just about to ask for a screenshot. Looks good!

 

Yes start the array and copy the emulated disk3 to your PC.

I'll back in about an hour.

Small problem... 

WDC_WD2003FYPS-27Y2B0_WD-WCAVY6439141 - 2 TB (sde) (Disk 1) shows unmountable, no file system.

1 hour ago, Pyro said:

Small problem... 

WDC_WD2003FYPS-27Y2B0_WD-WCAVY6439141 - 2 TB (sde) (Disk 1) shows unmountable, no file system.

We can deal with that separately later. Can you get the files on the emulated disk?

Give me a screenshot. And whatever you do don't agree to FORMAT anything.

Also good to post diags to see why the disk isn't mounting.

1 minute ago, johnnie.black said:

Also good to post diags to see why the disk isn't mounting.

Glad to see you on this thread. Hope you will stick around for the fun😉

Yes, I am copying disk3 now. 3 failed files so far, all replaceable. 37% and counting. Definitely not going to format anything.

Here's the screen, and diagnostics.

 

tower-diagnostics-20181128-1552.zip

Disk1 is failing, so some files will fail to copy, copy what you can for now.

6 minutes ago, johnnie.black said:

Disk1 is failing, so some files will fail to copy, copy what you can for now.

So is disk2, I posted the SMART on the other page before we got his disk assignments back. But its all we have to work with to try to get the data for the unresponsive disk3.

Yes, OP should try to copy what it can from the emulated disk3 first, with some luck disk2 is OK for now since last read errors on the SMART report are from a long time ago, and some times WD disks report false pending sectors, if not disk2 together with disk1 are good candidates for ddrescue after disk3 copy is done, disk1 will need a filesystem check but that can only be done after it's cloned to a new disk, since xfs_repair would abort on a read error.

disk3 copied successfully, with a few minor exceptions.

 

Where should I be headed from here? Deal with the unmountable drive or start the new array with the 10tb and 8tb?

If we aren't going to try to rebuild disk3 then a New Config is definitely going to be done. Parity wouldn't help us with the unmountable disk anyway.

 

@johnnie.black suggested cloning disk1 before trying to repair its filesystem. Might be just as well to get the rest of the array protected again and maybe even copy the data you saved from disk3 back before trying to deal with that. Maybe he will have another idea.

Were there any read errors on disk2? If not I would concentrate on disk1 first, if there's data you want there you can use ddrescue to then try to fix the filesystem.

Disk 2 also has read errors. Should I attempt to back it up first before proceeding?

Since disk2 is not emulated you can do it in the order you like, and since disk2 mounts you don't need to use ddrescue, you can just copy the data like you did from the emulated disk3, some files will fail to copy because of the read errors, with ddrescue those files will be still copied, though with some corruption.