I have 11 disks in the array, 1 parity and 10 data drives. 
Data disk 1 has had a smart error and because it's not the first, I am compelled to change the disk.

 

Even though I have hotswap, I shut down the server, removed data disk 1, replaced it with a new identical disk. Array wasn't turned off when I did my shutdown.

 

Server back on, my array is completely different-- my disks were all mixed up and not able to start because somehow the parity disk had been moved to the data disks...???

 

I powered down forcefully. Removed the disk I had justt replaced and put back the same drive I had prior - data disk 1, tthe one that had a smart error. 

Now, my array is in the same config, only the Parity drive is showing as a NEW DEVICE. It's exactly the same drive in the same spot! Am I going to lose everything here? the array doesn't want to start anymore because It's say "Too many wrong and/or missing disks!"

 

Please help me fix this. This is 29 TB of personal stuff I stand to lose. 

10 hours ago, couzin2000 said:

Server back on, my array is completely different-- my disks were all mixed up and not able to start because somehow the parity disk had been moved to the data disks...???

That suggests a flash drive problem.

 

10 hours ago, couzin2000 said:

Now, my array is in the same config, only the Parity drive is showing as a NEW DEVICE. It's exactly the same drive in the same spot! Am I going to lose everything here? the array doesn't want to start anymore because It's say "Too many wrong and/or missing disks!"

If you have all the original disks assigned to the correct slots, you can do a new config and check "parity is already valid" before array start, then start array and you can try replacing the bad disk again.

 

10 hours ago, couzin2000 said:

Data disk 1 has had a smart error and because it's not the first, I am compelled to change the disk.

Are you sure the SMART error indicates a disk problem?   Some errors (e.g. CRC errors) are nothing to do with the drive having a problem but are due to external factors.

I'm sure the drive is faulty, as this smart error is occuring for a third time now. The first two times self-tests revealed a functioning drive. This time the drive is "marked" and I'm unable to use it, self-tests fail everytime at 10%.

 

The parity drive did show a smart error when the other drive failed. However repeated self-testing suggests it's not at fault. 
 

The SAS to SATA cable has been replaced with a brand new, better quality cable. Triple checked the cabling is connected to the proper connectors.

 

I will recheck again to make sure no data is lost.

 

As far as the flash drive problem, I doubt this is the case as my flash drive has been pretty reliable and the odds of it failing at the same time as a hard drive (let alone possibly 2 drives) are astronomical. There was no issue prior to this, no power outages no transfer problems, nothing.