OK, so I'm not having a good day. I've noticed all my shares were set to use cache and they didn't just use the cache for new files. It kept using the cache and refused to empty the cache drive. I've suspected something was going on for a while, but couldn't put a finger on it. I figured I was just improperly utilizing cache due to constant complaints of cache being full or near full. I decided it was time to start marking shares as don't use cache. I began rsyncing files over. from /mnt/cache to /mnt/user/  . I had one VM being used as a transparent bridge firewall, I decided rsyncing was taking too long and checked both /mnt/user and /mnt/cache to make sure they both had the VM. Then proceeded with deleting the VM once I noticed they were both there. NOPE! now the VM is gone completely. I'm not sure what was happening with the mover. even with the VM halted, the mover refused to empty all the files off the cache drive.

 

Now I'm praying that the entire library of videos I used for my plex docker actually decided to move when I rsynced from the cache to the user (because invoking mover didn't move the files off). wish me luck guys. I'm thinking I'm in for a nightmare.

Unfortunately your attempt to rsync the files from the cache disc to /mnt/user will have probably caused the rsynced files to be truncated to zero size.    If you wanted to do this you should have copied them to /mnt/user0 just as mover usually does.

 

The /mnt/user share already includes the cache disc, while the /mnt/user0 specifically does not include the cache disk.

 

i suspect that if you used the option to rsync to delete the original file then it would have then deleted the file as well.

 

 

Edited June 25, 20179 yr by remotevisitor

The /mnt/user location is a consolidated view of all the top level folders on the array disks AND /mnt/cache.   The problem is that /mnt/cache is a disk Share and /mnt/user is a user share.   Copying files from /mnt/cache to /mnt/user can result in you trying to copy a file to itself and result in the file being truncated (this leading to data loss).   

Nope, didn't even give the zero size. 500G worth of crap gone due to mover refusing to move the files. SSD is a 1TB. I guess its not all bad. at least I know whats missing. I could probably just acquire it all again and reinstall pfsense on a new VM and start over there. its only a couple weeks worth of work. I still feel like I could crawl in a hole right now.

Manually moving stuff is risky, not going into details since it has already been mentioned.

 

Could the initial problem be that you had the shares cache settings set to Prefer? That would cause mover to move files from the array to the cache and not the other way round.

Nope was not set to prefer, was set to yes.

The real answer would have been to work out why mover was not behaving as expected.    The way to investigate this would be to involve mover manually from the Main screen and then post the diagnostics zip file (obtained via Tools->Diagnostics) so others can try and spot what is set up incorrectly.

I'll keep that in mind next time. I'm planning on setting my VMs folder to Prefer for cache so Win10Steam machine will quick running so bad when HDDs have a bit of a load on them.

OK, I still had 500G worth of stuff that needed moved. I manually started the mover. now the mover appears to have completed, but nothing was moved.

cold-rewind-diagnostics-20170625-1703.zip

Just to confirm, it is the DATAZ share that contains those 500G, right?

 

Edit: Oh, 6.4.0-rc6?

Edited June 25, 20179 yr by gubbgnutten

Yep, you would be correct.  the issue occurred before moving to 6.4-rc6. I was previously on 6.3 when the issue began. I'm also not the only one who uses the shares so I'd rather not make the friends and family mad by loosing their data lol. Luckily the stuff I lost was only mine, but the rest starts getting into shared territory.

Edited June 25, 20179 yr by Darksurf

1 hour ago, Darksurf said:

OK, I still had 500G worth of stuff that needed moved. I manually started the mover. now the mover appears to have completed, but nothing was moved.

cold-rewind-diagnostics-20170625-1703.zip

At this point according to the diagnostics the DATAZ share is set to Use Cahe=No.   Mover will only move data for a share if it is set to Yes.

1 hour ago, itimpi said:

Mover will only move data for a share if it is set to Yes.

and the files are not open. Files currently in use when the mover is running will stay on the cache drive as well.

Also, your VMs share is set to cache prefer, so the mover is trying to move it to cache:

 

Jun 25 16:41:53 cold-rewind root: move: file /mnt/disk1/VMs/Windows10Nvidia.cfg
Jun 25 16:41:53 cold-rewind root: move: skip /mnt/disk1/VMs/Windows10/vdisk1.img
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs/Windows10
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs/Windows10: Directory not empty
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs: Directory not empty
Jun 25 16:41:54 cold-rewind root: move: skip /mnt/disk1/system/libvirt/libvirt.img
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system/libvirt
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system/libvirt: Directory not empty
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system: Directory not empty
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk2/VMs/ISOs
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk2/VMs/Windows10
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk2/VMs
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk3/VMs/ISOs
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk3/VMs

 

2 hours ago, johnnie.black said:

Also, your VMs share is set to cache prefer, so the mover is trying to move it to cache:

 

Jun 25 16:41:53 cold-rewind root: move: file /mnt/disk1/VMs/Windows10Nvidia.cfg
Jun 25 16:41:53 cold-rewind root: move: skip /mnt/disk1/VMs/Windows10/vdisk1.img
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs/Windows10
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs/Windows10: Directory not empty
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs
Jun 25 16:41:53 cold-rewind root: move:  dir /mnt/disk1/VMs: Directory not empty
Jun 25 16:41:54 cold-rewind root: move: skip /mnt/disk1/system/libvirt/libvirt.img
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system/libvirt
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system/libvirt: Directory not empty
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk1/system: Directory not empty
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk2/VMs/ISOs
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk2/VMs/Windows10
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk2/VMs
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk3/VMs/ISOs
Jun 25 16:41:54 cold-rewind root: move:  dir /mnt/disk3/VMs

 

Which is appropriate, but you will need to stop the VM service before mover can succeed.

Thanks so much guys! This instruction has been extremely helpful! I've got my cache drive emptied! I'll now halt my VMs to get them moved to SSD cache. Another question. if something is using prefer cache such as my VMs it sounds like it wouldn't be protected by the parity drive. Can you suggest a method to backup VM disks?