I'll start by saying I likely did everything wrong in this scenario.
The Story:
I have been acquiring upgrades for my server over the past year, and finally had a chance to add them all. Several 14T drives, an HBA card, and a cache drive. Drives were pre-cleared a long time ago and have just been sitting on my desk until I had time.
I took a picture of my array before I shut it down so I would know what order drives were supposed to be in because also as part of this I had planned on labeling my drives in the system for easy identification. When looking at the picture I noticed there were a number of errors on one of the drives and thought that the drive is failing it's good that I am doing this maintenance. Parity at this point was good and all I needed to do is reconfigure the drives and add all the new stuff. This is also about the time I noticed my crashplan backups were not working as expected so I now have one copy of data that is parity protected ... I'm freaking out a bit at this point but feel comfortable knowing the parity is good.
So in the process I added all the things and configured things to be correct as I thought they were. I selected the WRONG 14T drive as the parity drive. This started a parity rebuild, but it was going at a glacial pace ~700kb/s. So now I have no backup, no parity and this is where true panic sets in. After about 19 array restarts I was finally able to get the drive to function normally. I then used Unbalance to move all of my VM's and some of my ROM data to a safe drive since I was getting full write speed. This action caused those drives to get written to cache not the drive out right so now I have a 1TB cache drive full and mover won't move content over.
My array is safe now, and the bad drive has been replaced, however now when I try to use mover to move data over it says that the cache is full and it can't move files.
My question:
So now I have good parity, a good drive in place of the bad drive, and my cache is full of actual data, how can I get that data moved over so that the system knows it is on the right drive? Every time I try to start mover it just says Cache is full, those shares are all set to Prefer cache but it shouldn't fail when it can't use cache. I'm kind of lost at this point.
Lessons Learned:
Use Hard drive serial numbers NOT it's place on the /dev/ file system to designate what it is if you are using an HBA controller.