I'll start by saying I likely did everything wrong in this scenario.

 

The Story:

I have been acquiring upgrades for my server over the past year, and finally had a chance to add them all.  Several 14T drives, an HBA card, and a cache drive.  Drives were pre-cleared a long time ago and have just been sitting on my desk until I had time.

 

I took a picture of my array before I shut it down so I would know what order drives were supposed to be in because also as part of this I had planned on labeling my drives in the system for easy identification.  When looking at the picture I noticed there were a number of errors on one of the drives and thought that the drive is failing it's good that I am doing this maintenance.  Parity at this point was good and all I needed to do is reconfigure the drives and add all the new stuff.  This is also about the time I noticed my crashplan backups were not working as expected so I now have one copy of data that is parity protected ... I'm freaking out a bit at this point but feel comfortable knowing the parity is good.

 

So in the process I added all the things and configured things to be correct as I thought they were.  I selected the WRONG 14T drive as the parity drive.  This started a parity rebuild, but it was going at a glacial pace ~700kb/s.  So now I have no backup, no parity and this is where true panic sets in.  After about 19 array restarts I was finally able to get the drive to function normally.  I then used Unbalance to move all of my VM's and some of my ROM data to a safe drive since I was getting full write speed.  This action caused those drives to get written to cache not the drive out right so now I have a 1TB cache drive full and mover won't move content over.

 

My array is safe now, and the bad drive has been replaced, however now when I try to use mover to move data over it says that the cache is full and it can't move files.

 

My question:

So now I have good parity, a good drive in place of the bad drive, and my cache is full of actual data, how can I get that data moved over so that the system knows it is on the right drive?  Every time I try to start mover it just says Cache is full, those shares are all set to Prefer cache but it shouldn't fail when it can't use cache.  I'm kind of lost at this point.

 

Lessons Learned:

Use Hard drive serial numbers NOT it's place on the /dev/ file system to designate what it is if you are using an HBA controller.

1 minute ago, Wachuwamekil said:

those shares are all set to Prefer cache

Which means you Prefer to keep them on cache, and move them to cache if any are on the array. Cache-yes is what you want.

2 minutes ago, Wachuwamekil said:

Use Hard drive serial numbers NOT it's place on the /dev/ file system to designate what it is if you are using an HBA controller.

Regardless of controller, Unraid only cares about serial numbers when deciding where a disk is assigned.

5 minutes ago, Wachuwamekil said:

I selected the WRONG 14T drive as the parity drive. 

So the data on that disk was overwritten. Do you have backups? Even if you do everything right parity is no substitute for backups. Plenty of ways to lose data that parity won't help with. You don't have to backup everything, just everything important and irreplaceable. You get to decide.

8 minutes ago, trurl said:

Regardless of controller, Unraid only cares about serial numbers when deciding where a disk is assigned.

Brilliant thank you, I'm going to make this change once mover fails and see what magic happens.

 

8 minutes ago, trurl said:

Regardless of controller, Unraid only cares about serial numbers when deciding where a disk is assigned.

I found this out the hard way, I've updated the labels to have the Serial number moving forward.

 

5 minutes ago, trurl said:

So the data on that disk was overwritten. Do you have backups? Even if you do everything right parity is no substitute for backups. Plenty of ways to lose data that parity won't help with. You don't have to backup everything, just everything important and irreplaceable. You get to decide.

I made a massive assumption that crashplan was able to backup everything needed, however when I was in the middle of this and I looked at the console it wasn't backed up correctly. I am revisiting my backup strategy and ensuring that crashplan is configured correctly.  I'm also going to be doing an off site backup at a friends house and adding others as we get additional external drives.  We are building a bit of a mesh backup network between friends and I'm going to take advantage of it.

 

I appreciate the replies and feedback!

If you want us to take a look to see if you have anything else setup less than ideally, attach diagnostics to your NEXT post in this thread.

Thank you fro the help trurl, that fixed the issue I really appreciate it.

 

If you can review the diagnostics blob that would be awesome!

thetwelve-diagnostics-20211126-2208.zip

Your docker.img is ridiculously large. 20G is often more than enough. Have you had problems filling it? I run 15-20 dockers and only use about half of 20G allocated.

 

Your appdata, domains, system shares have files on the array. In fact, your system share is configured to only have files on the array. You want these shares to be on pool (cache), so dockers/VMs won't have performance impacted by slower array, and so they won't keep array disks spunup since these shares always have open files.

 

Didn't examine SMART for you disks. Do any have SMART warnings on the Dashboard page?

 

Do you have Notifications setup to alert you immediately by email or other agent as soon as a problem is detected?

23 minutes ago, trurl said:

Your docker.img is ridiculously large. 20G is often more than enough. Have you had problems filling it? I run 15-20 dockers and only use about half of 20G allocated.

I was getting warnings that i was at 95% utilization of docker.img, so I increased it a ton to avoid the issue.  I'm going to drop that back down to 30G and see if I can't find out what is chewing up space.

 

23 minutes ago, trurl said:

Your appdata, domains, system shares have files on the array. In fact, your system share is configured to only have files on the array. You want these shares to be on pool (cache), so dockers/VMs won't have performance impacted by slower array, and so they won't keep array disks spunup since these shares always have open files.

This is likely related to all the moving around I had to do with system data.  Is there a safe way to ensure that all array data/info gets moved to to appdata and domians on my cache drive?

 

24 minutes ago, trurl said:

Didn't examine SMART for you disks. Do any have SMART warnings on the Dashboard page?

What started this whole thing was a 3GB drive that had errors, the drive was similar to the other one and I grabbed the wrong one.  I verified SMART status to ensure I was acting on the correct drive.

 

25 minutes ago, trurl said:

Do you have Notifications setup to alert you immediately by email or other agent as soon as a problem is detected?

I do not, adding notifications is next on my list to tackle so I won't be in this position again.

1 hour ago, Wachuwamekil said:

95% utilization of docker.img, so I increased it a ton

Making it larger won't fix anything, it will only make it take longer to fill. Of course, with that 250G docker.img it might take a very long time, but there are other likely problems with filling docker.img.

 

The usual cause of filling docker.img is an application writing to a path that isn't mapped. Any path specified in an application must correspond to a container path, case-sensitive. And if, for example, you have a downloading application that is downloading to a path that isn't mapped, not only will it fill docker.img, but you can't easily access the downloaded files since they are not in any user share.

 

And, completely filling docker.img will probably corrupt it.

 

1 hour ago, Wachuwamekil said:

moved to to appdata and domians on my cache drive?

Nothing can move open files, and mover won't move duplicates.

 

To make sure Docker and VM Manager have no open files, you have to disable them in Settings.

appdata, domains, system shares must all be set to cache-prefer so their files will be moved to cache.

Then you can run mover, wait for it to complete, and post new diagnostics so we can see if there are duplicates that need to be sorted out.