Mover not moving, inflated domain folder size

[Elmojo]

Hi all,

So I attempted to migrate my cache drive from a 128GB SSD to a 256GB SDD, because I was getting a little low on space.

Long story short, it seems to have backfired.

I tried following the best guide I could find, which seemed to be this video:

All went fairly well, except that Mover took forever, and refused to move everything off my existing cache drive. It left about 60MB of junk.

I finally gave up and decided to move it manually.

I installed the new drive, did the steps of adding it as cache, formatting, etc... then moved the stray data back to the new cache drive.

I then set the domain share to cache-prefer, and invoked Mover....nothing. I did it multiple times. It refuses to move anything.

I finally gave up, plus given how long it took to move things off the old cache drive, I decided just to move the appdata via Mover, then use MC to manually move the domains and system folders myself. This approach appeared to work well, in terms of speed anyway.

However, the domains folder has blown up from around 80GB before, to nearly 200GB now. I seem to recall reading something a while back that said due to compression or something related to the dynamic nature of the vdisk files, you can't really do what I just did.

So how am I supposed to get my domains folder on the new cache drive without breaking things?

I tried finding the official guide to replacing a cache drive in the unraid docs, but it appears to be MIA.

 

[Elmojo]

No one? Not even a guess? 

*crickets*

[trurl]

 

[Elmojo]

Thanks for that link, but I didn't see a solution in there... maybe I missed it?

[trurl]

On 7/12/2020 at 10:15 PM, JonathanM said:

You should be able to use

cp --sparse=always

to make the file sparse again.

  

 

[Elmojo]

So is that a direct command in the terminal? Does it act on all VMs automatically, or does it need to be pointed to a specific vdisk file?
Also, I noted in the thread you linked that that command didn't seem to work for the OP. I'm game to give it a shot, I just want to be sure I'm using it correctly.

My terminal-fu is really weak.

[trurl]

That is an idea for a command to run at the command line. It is the linux copy command 'cp' with an option to be used for sparse files. You have to supply source and destination for the command.

 

google linux cp command

[Elmojo]

Ah, I see. I'll look into it, thanks.

I'm afraid to monkey up these VMs, since they are in use, but I need to get these disk sizes under control.

They're taking up all the space on my new cache drive, defeating the purpose of the upgrade.

[Elmojo]

Okay, so I'm really sorry to bug you with this, but something just occurred to me.

If I use that command, and it works to "slim down" the vdisk. Won't it just blow right back up when I move it back to the cache drive?

There's not enough room on the drive now for me to do the copy there, so I'll have to move it to the array, do the conversion, then move it back.

I suspect I'll be shooting myself in the foot doing it that way. Is there a method of moving that won't blow it back up?

[JonathanM]

If you use the sparse copy from the array to the cache drive it should shrink.

 

However...

 

What you are doing, over allocating the images, will likely shoot you in the foot if you can't manage them. Keep in mind that the file size WILL eventually expand to the size it is right now through daily use of the VM, unless you take steps to keep that from happening.

 

This event should serve as a wake up call to either take the time to read the information on this forum and apply it to managing thin provisioned disks, or redo the VM's with smaller disk allocations so that even when they are full they don't cripple your server.

[trurl]

same cp command different source and destination

[Elmojo]

1 minute ago, JonathanM said:

If you use the sparse copy from the array to the cache drive it should shrink.

Ah, duh. Yeah, that'll work, thanks.

Not sure what you mean about the rest...

I'm not aware of any reason why the max disk size would be reached, unless I went all silly with activity inside the VMs.

These had been running for a couple months previously without issue. Their respective sizes had been fairly stable.

If there is information to the contrary, I'd love to know about it. Would you mind sharing the link(s) to what you were referring to that I need to read?

I seem to have trouble finding what I'm looking for here sometimes. The search just seems to pull random word pairings, and I haven't found a way to search within a specific forum. That's beside the point, though.

For now, I need to be schooled on proper VM management, if you're willing to help teach me.

[JonathanM]

Start here, you should be able to use the keywords to further your education.

https://forums.unraid.net/topic/51703-vm-faq/?tab=comments#comment-557606

 

[Squid]

5 minutes ago, Elmojo said:

If there is information to the contrary, I'd love to know about it. Would you mind sharing the link(s) to what you were referring to that I need to read?

 

 

Under default settings, the vDisk will always grow to it's maximum size and will never shrink to reflect deleting files etc from the vdisk

[Elmojo]

For the terminal cmd, does something like this look feasible?

cp --sparse=always /mnt/disk2/Win10\ Sandbox /mnt/cache/domains

If I understand the cp docs correctly, that should copy the folder "Win10 Sandbox" from disk2 (where I moved it temporarily), back to my cache drive, thinning it back down to size.

I'd like a more knowledgeable pair of eyes on it before I invoke the cmd and break something, like my destination domains file, for example.

[Elmojo]

Thanks for the links guys!

I had actually seen that FAQ before, but it was so old, I assumed that it must be out of date, like most of the other information I find from around that time period.

It's good to know that some of it is still valid, but just makes it harder for new-ish folks like myself to get up to speed.

Oh well, I'll get here, with some help.

[Elmojo]

Thanks SO MUCH for the help guys. Let's call this one SOLVED!

I was able to use a modified version of the cmd string above to get everything whipped back into shape.

 

For anyone finding this later, and being clueless like me, this is how it shook out...

1) Stop all VMs

2) Manually move (I used Krusader) the vdisk images for each of your VMs to some location that has space enough for them. If you have room for 2 copies of them on your cache drive, then you can omit this step. I also copied the entire folder (VM name) to help me keep track of which VM I was moving at the time.

3) Once the file(s) are moved, and you have some breathing room on your cache drive, open terminal, and use the following command (edited for your pathing of course):

cp -n -r --sparse=always /mnt/disk#/Sourcefoldername /mnt/cache/domains

Note: If you have any spaces in your VM names (like I did), then you have to use "\ " (backslash space) between the words [eg. "Windows 10 VM" would be "Windows\ 10\ VM"]

4) Wait for the copy to complete. Once the terminal is back to the ready prompt, you should be able to use your method of choice to confirm that the vdisk is back in your domains share where it belongs, and is now much smaller. I used QDirStat, but there are lots of other ways.

5) Before restarting the VM, edit its properties, flip over to XML mode (switch at top right corner), then add discard='unmap' to the location shown in this post. 

    If you're running a fairly recent version of qemu, you won't need to jump through all those other hoops, and can just add that one bit of code and be done.

6) Start the VM(s).

7) If it's Windows 10, these next tasks may help. For any other OS, I can't say much...

     A. Go to This PC, right click Local Disk, select {Properties}, [General] tab, then [Disk Cleanup] button (next to the usage graph).

         Click [Clean up System Files] button, let it reload. Choose what you'd like to clean. Let it run.

         I recommend the Windows Update files for sure. They're likely to be large.  When that's done (might take a bit)...        

     B. Go to This PC, right click Local Disk, select {Properties}, [Tools] tab, then [Optimize] button. In that dialog, verify that "Media Type" is shown as "Thin provisioned drive".

         Click [Optimize] button.  Wait some more...

Rejoice! You have your space back, and your VMs should be working fine again.

 

Thanks again to those who patiently lead me to the info I needed to get this sorted out. This really is a fantastic community!