This evening I have begun to upgrade my parity drive to a larger one so I can expand my array. I precleared my new disk, stopped the array, then changed the parity assignment from my previous 3tb to my new 6tb. The original parity is still in the server but now unassigned.

 

During the parity rebuild onto the larger drive, another data disk has failed. It showed no symptoms or smart errors prior to this, it simply dropped offline with the log showing an I/O error. 

 

The rebuild seems to be continuing despite the number of errors being in the millions.

 

What are my options now? I have tried pulling and reinserting the failed disk and it briefly shows the activity light for around 3 seconds but doesn't appear to be detected by Unraid when I reinsert it. 

 

Have I lost the data on that drive? Or the whole array? What will happen once the parity rebuild completes? If worst comes to the worst, I have my previous parity drive, can I use that somehow to rebuild the failed array drive (although my only spare drives are larger than the parity, so I cannot replace that without doing the parity first). 

 

Few screenshots attached. Disk 4 is the failed one, ignore the other 2 showing errors (due to a previous dodgy cable).

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your next post.

Hi again sorry I should have thought of that before

unraid-diagnostics-20190901-0815.zip

Disk4 dropped offline, disk itself looks fine, shutdown replace/check cables and try again, but you do appear to have a problem with disk8, so keep old parity intact, but it can only help in there weren't any writes to the array since it was removed.

34 minutes ago, johnnie.black said:

Disk4 dropped offline, disk itself looks fine, shutdown replace/check cables and try again, but you do appear to have a problem with disk8, so keep old parity intact, but it can only help in there weren't any writes to the array since it was removed.

Thanks Johnnie that's reassuring

 

There were some minor writes, only an automatic update to some dockers. Does that ruin the whole thing? My cache disk holds the appdata folder so not sure if those updates would have affected the array?

 

 

Edited September 1, 20196 yr by fitzy89

29 minutes ago, fitzy89 said:

My cache disk holds the appdata folder so not sure if those updates would have affected the array?

If appdata and docker image are on cache it won't affect parity.

All of the dockers that I use are set to store on the cache drives however I've just looked at some older dockers that were shutdown, they are still set to the array, and I can't guarantee that none of them updated at the same time so I think I'll assume that the previous parity is now useless.

 

However just been reading some recovery threads elsewhere, you mention that the disk dropped offline rather than failed. In which case there's a good chance that the data on it is still readable via other means. In which case, if I unassign the disk from the array, rebuild parity without it, and then manually copy the data from this disk back into the array that could potentially save it?

1 hour ago, johnnie.black said:

shutdown replace/check cables and try again,

If you do the above data on disk4 should be immediately accessible, the problem might then be disk8, which is likely to have some read errors during the parity sync, but if it does you can deal with that later, though it would be best if old parity was still valid.

1 hour ago, fitzy89 said:

There were some minor writes, only an automatic update to some dockers. Does that ruin the whole thing? My cache disk holds the appdata folder so not sure if those updates would have affected the array?

Your docker image is in the system share, but that share has files on cache and disk1. And your appdata share has files on lots of disks. After you get your data safe again we can discuss how to get those shares completely on cache where they belong.

 

As I'm sure you know, most of your disks are completely full or nearly so.

4 minutes ago, trurl said:

Your docker image is in the system share, but that share has files on cache and disk1. And your appdata share has files on lots of disks. After you get your data safe again we can discuss how to get those shares completely on cache where they belong.

 

As I'm sure you know, most of your disks are completely full or nearly so.

Thanks for the info. Yes the appdata was a bit of a mess caused by the Sqlite corruption bug where I moved everything to the cache disk to prevent that from happening again. Somehow I made a mess of moving the data to cache and ended up with duplicates and then didn't know which copy it was using so left them all in place at the time, I guess there's some flag for rsync that can copy/keep the newer files and consolidate them all into one place?

 

I'm hoping to separate some of my data onto dedicated disks once this gets sorted, I'm adding 4x 6tb drives (1 for parity, other 3 to replace existing array drives). Once the current parity rebuild has finished I'll see what state it's all in and then shut down the system to check cables on disk4

Hi again

 

The parity rebuild has now completed at last. Disk4 is also listed as disabled/emulated now and I can see the emulation does show some data although I have no way of telling whether this is everything that should be there or not. 

 

As I was going to replace this drive anyway, can I just swap it out as normal and assign one of my new drives? I'm nervous to power down the system with the state it's in at the moment so can't check cabling yet but the new drives are installed, precleared and ready for use.

 

I wonder whether it may be best to add one of the new drives as an additional drive and copy the contents of the emulated disk4 and the erroring disk8 over to a new drive? My thinking is that we wouldn't want the load of a rebuild on the already dodgy disk8 which could push it over the edge. Disk4 being emulated at present isn't protected anyway and needs the data moving anyway so I'm thinking this would put the least strain on everything.

 

Latest logs attached

unraid-diagnostics-20190901-1701.zip

Diags posted don't show nothing new, have the same info from the previous one, without a reboot.

On 9/1/2019 at 1:12 PM, fitzy89 said:

I wonder whether it may be best to add one of the new drives as an additional drive and copy the contents of the emulated disk4 and the erroring disk8 over to a new drive? My thinking is that we wouldn't want the load of a rebuild on the already dodgy disk8 which could push it over the edge. Disk4 being emulated at present isn't protected anyway and needs the data moving anyway so I'm thinking this would put the least strain on everything.

In order to emulate disk4, parity PLUS ALL other disks must be read, so not much difference really between emulating the disk to copy it and just rebuilding it. Disk8 will be used either way. And you can't add another disk to the array in this state anyway.

 

Do you have backups?

I think the previous logs were the same because the log partition was full so new ones weren't being stored. 

 

Attached my current logs. I put the old "failed" disk4 into another machine but it's definitely dead sadly. It spins up and sounds fine but is not detected under Ubuntu or Windows. I lost around 1.7TB out of the 2TB used. None of it is irreplacable, but most of it being a Plex library it's a bit of a pain to download things again. I do have a partial/incomplete online backup with Crashplan, but given the size of the array I don't have the budget for a local backup.

 

For the disk8, I swapped that out and added an additional disk at the same time so I had the extra capacity of the new disk to use whilst the disk8 was rebuilding. 

 

I think I'm reasonably happy with the current state of it now. I'll add a second parity in the future I think but for the timebeing I'll have to stick with one. 

 

Is it possible to re-order disks in the array at all? I'd love to have them sorted in alphabetical order (just an OCD thing lol).

 

unraid-diagnostics-20190903-1802.zip

19 minutes ago, fitzy89 said:

Is it possible to re-order disks in the array at all?

Yes, but with dual parity it will require parity2 to be re-synced, while still single parity you can do a new config and trust parity.

3 hours ago, fitzy89 said:

I'd love to have them sorted in alphabetical order

If you are talking about the sdX identifier, no. That is determined by the hardware and the order they are initialized. If you are talking about brand and serial number, see johnnie's answer.