I had disk 6 of 21 fail.  it was 4tb and I was running low so I replaced it with a 5tb.  I started the rebuild and then I noticed the writing had stopped and disk 5 3tb was showing 127 million read errors and the temperature reading on that disk was *.  So I stopped the rebuild, rebooted and thinking I should remove that new drive and run some tests or reseat the cables, I did a new config and started putting back all the drives in their correct place planning to leave disk 6 out and clear it to try the rebuild again.  but when I went to assign the parity disk it wouldn't assign.  I see it in the drop down but the browser refreshes and it's not assigned.  I tried assigning some of the main array disks and only the first three drives which are 4tb drives would assign, then next one drive 4 3tb did the same as the parity drive.  I'd select it and then the browser would refresh and it wasn't selected again.  I looked at the log and see the entry where it took the 4tb disks but theres no entry for the other disks which wouldn't assign as though nothing happened so nothing was entered in the log. 

any ideas?

syslog.txt

hate to bump but im down hard and nothing seems to be working.

I'll take a look at your syslog, but just a reminder, it's the diagnostics we want.  Please see Need help? Read me first!, and attach the diagnostics zip.

I read a diff post that made mention of usb drives possibly affecting writing to the flash drive.  I have a usb external that I use for transferring to air gapped backup.  I'm going to get that removed and see if I can get drive assignment to stick.

tower-diagnostics-20161028-1053.zip

Also, DO NOT CLEAR DISK 6!  Don't touch it until we have had a look, you may need it intact.

 

And please tell us what was wrong with Disk 6 that made you decide it had failed, what errors and symptoms did it have.

Some of your disks are have the SAS address at the end, you may need to change a setting:

 

http://lime-technology.com/forum/index.php?topic=48508.msg500313#msg500313

disk 6 showed up as not found and was emulated.  I removed it and have it safe.  I put in a new disk which was discovered fine leading me to believe that the issue was in fact a disk failure and not a different pc issue.  Then the read errors on disk 5 prompted me to put the kibosh on the rebuild and check things out before starting it again.  If I can get the system back to normal I can restore the data from the missing drive by syncing it with my backup server so I'm not terribly concerned about data loss.  Id prefer not to have to resync the entire machine which would take a week or two.

johnnie.black, that worked!  I can assign them now!  thanks  I'll post back and let you know how the rebuild goes.

ok the array came up and I left disk 6 unassigned.  the replacement for disk 6 is showing in the unassigned devices list.  I need to clear it and then assign it back to disk 6 slot for the rebuild?  or have a ruined the option to rebuild and I should just restore from backups?

New config should not be done when you need to rebuild a disk, if you have a spare the same size as the old disk there's "trick" that usually works.

johnnie.black already spotted the key issue, the WWN thing.  That was affecting 16 of your drives!  And he spotted the 'New config' you mentioned, which is what worried me about rebuilding Disk 6.

 

Disks can drop off for a number of reasons, not all of which are the drive's fault, so your original Disk 6 may be perfectly fine.  You should try obtaining a SMART report for it, and let us look.  You may be able to put it back online, and only have to worry about Disk 5.

the rebuild started again and right away starting throwing read errors on disk 5.  I'll get the old disk 6 into a cradle and plug it into my mac to run some tests.  Also need to look at disk 5 cables etc.  something may have pulled.  At least I can bring up my windows vm and disk 6 doesnt have anything on it that I don't have copies of.  disk 5 however has a few things I never felt the need to keep a copy of untili now.

the rebuild started again and right away starting throwing read errors on disk 5.  I'll get the old disk 6 into a cradle and plug it into my mac to run some tests.  Also need to look at disk 5 cables etc.  something may have pulled.  At least I can bring up my windows vm and disk 6 doesnt have anything on it that I don't have copies of.  disk 5 however has a few things I never felt the need to keep a copy of untili now.

Don't mess with disk6 unless you are ok with giving up on recovering disk5. If disk6 or the other array disks haven't been written to after this mess started, you should be able to use disk6 to recover disk5, but if you write to disk6 or the other array disks, that will corrupt any emulation of disk5.

ok so I reseated the disk 5 cable and started the array.  it's doing a parity sync and is reading away just fine from disk 5, no errors so far.  it seemed it didn't like reading from the disk on a rebuild but doesn't seem to mind reading for a parity check.  disk 6 shows emulated data since it appears 60% full even though it's blank. 

 

should i just  wipe out the data from a command prompt on disk 6 and attempt to reload the data from my old disk 6?  If I can't use the old disk which I'd prefer, I'll reload from my air gapped backup machine.  Do I need to wait for the parity to finish?  It's going to take a long time and not be good anyway, right?

I feel like a jackass.

 

I put everything back the way it was and the old drive 6 seems fine. I'm still nervous because I reseated the cables and did mupltiple reboots before thinking it was dead. Putting it in the enclosure and feeling the way it spun up made me think it was a cable somewhere after all and so far so good. I'm having to do a new parity but I'm seeing no read errors.  Thanks for your help though, when the drives wouldn't even assign I was scared. 

 

Once the parity is done I'd like to take out a 4tb and put in the new 5tb, what's the best way to do that?

The normal way, shutdown server, replace disk, power on, assign new disk and start array to begin rebuild.