Mover is killing (me) my server

[hopelessDude]

The script is not clearing my cache drive, but it is moving now to fill up the rest of my drives. I have no idea where it is getting all the data from to be honest. Disk 1 is now at 100% .. when will it stop. I have tried changing the share settings to force free space, but mover is ignoring it pretty much. I deleted a VM that had a 2tb diskimage.. (and likely was the cause to clog my cache drive which was only 500GB) but it is not really "clearing the space" now if that makes sense. Also mover has no verbose mode that I can look at (or understand where to look for it). 

 

Any advice would be cool. 

 

 

 

Edited January 19, 2021 by hopelessDude

[JorgeB]

Enable mover logging to see what it's doing. 

[itimpi]

25 minutes ago, JorgeB said:

Enable mover logging to see what it's doing. 

Worth mentioning that this is found under Settings >Scheduler - may not be obvious to new users  

[hopelessDude]

thank you @itimpi

 

I clicked for a bit to find it. 

 

It seems that some stuff has worked out now that I have changed all the shares to yes. appdata is lowkey complaining that it is not cache only.. but who cares right?

[itimpi]

11 minutes ago, hopelessDude said:

enabled it now - that being said it has cleared up now... no idea how that worked out.. orackserver-diagnostics-20210119-0953.zipnly took 24 hours of moving

You probably do not want to leave that setting on once you are happy that the problem is solved:

• It can generate a LOT of log output
• You may not want that information in posted diagnostics as by its very nature it cannot be anonymized.
•  

9 minutes ago, hopelessDude said:

appdata is lowkey complaining that it is not cache only.. but who cares right

It is normally recommended that appdata is kept on the cache for performance reasons and to stop array disks spinning up unnecessarily.  At the moment appdata is spread across disks 1, 2, 3 and has no files on the cache.  If you want appdata on the cache (and have the space on the cache) then you would need to change the Use Cache setting for appdata to "Prefer" to tell mover that the files  should be transferred from array to cache.

 

[hopelessDude]

thx for the feedback. It is awesome that attachements cannot be deleted.. so there is that 🤣

[itimpi]

11 minutes ago, hopelessDude said:

thx for the feedback. It is awesome that attachements cannot be deleted.. so there is that 🤣

I think you mean 'can'   

[hopelessDude]

I am pretty much back to my original problem now. The cache is full. The mover is running (log is at 61% now) and my VM is paused again. I am downloading data to the VM and it seems as though the download is faster than the mover..  pretty much unusable. 

[JonathanM]

4 minutes ago, hopelessDude said:

I am downloading data to the VM

Why not download to a user share set to cache : no?

[trurl]

It is impossible to move from faster cache to slower array as fast as you can write to cache. Mover is intended for idle time. 

[hopelessDude]

Hi @jonathanm I think I will do that next. Or possibly specify a disk where the vdisk image is stored during vm creation. 

In the log it sais that the mover is skipping the vdisk image of the vm

 

rackserver move: move: skip /mnt/cache/domains/Ubuntu/vdisk1.img

 

instead it has kilometers of jbreed-nessus entries: /appdata/jbreed-nessus/opt/nessus/lib/nessus/plugins/openSUSE-2019-1419.nasl

[hopelessDude]

Hi @trurl point taken. How does it usually work when the VM exceeds the cache size? Pause until cache becomes available again and so on?

 

[itimpi]

8 minutes ago, hopelessDude said:

Hi @trurl point taken. How does it usually work when the VM exceeds the cache size? Pause until cache becomes available again and so on?

 

You should never have a vdisk used by a VM that has a logical size larger than the physical drive on which it resides or you will always get problems at some point.

[trurl]

Lots of things to consider in order to use cache in the best ways. Please read carefully and ask if you have any questions.

 

appdata, domains, system should be cache-prefer and all their files should be on cache. If these have files on the array, docker and VM performance will be impacted by slower parity updates, and since files in these shares will always be open, array disks can't spin down.

 

appdata is the working storage of the individual docker applications, such as plex library where it stores data about your media. The media (or downloads, for example) should be on other user shares, but the application working storage should be in appdata on cache.

 

domains is the vdisks which contain the OS of each of your VMs. VMs can access Unraid user shares for additional storage but their OS should be on cache in the domains share.

 

system is where libvirt and docker img files are. libvirt.img contains the xml for each of your VMs, and docker.img contains the downloaded executable code for each of your docker containers. These files are always open by the Docker and VM Manager services. These should be in system share on cache.

 

mover can't move open files, so if any of those shares have files on the array, you will have to go to Settings and disable Docker and VM Manager so mover can move them to cache.

 

mover moves cache-yes shares from cache to array, and moves cache-prefer shares from array to cache. mover ignores cache-only and cache-no shares.

 

cache-yes and cache-prefer shares will overflow to the array if Unraid sees that cache doesn't have enough free space. It is impossible for Unraid to know how large a file will become when it chooses a disk for a new file. Each user share has a Minimum Free setting. If a disk has less than Minimum Free, Unraid will choose another disk for the file. You must set Minimum Free to larger than the largest file you expect to write to the user share. Cache also has a Minimum Free setting in Global Share Settings.

 

mover is intended for idle time. That is why its default schedule is once per day in the middle of the night.

1 hour ago, trurl said:

It is impossible to move from faster cache to slower array as fast as you can write to cache. 

 

Trying to move while still writing to cache makes things even worse. Might as well not cache at all if mover can't keep up.

 

I recommend not caching anything in the initial data load, and not caching any user share that you expect to write more in one day than cache will hold. Might as well go straight to the array with those files where they will already have parity protection since caching will not make things any faster in the end.

 

If you want additional or more specific advice post new diagnostics.

 

[trurl]

45 minutes ago, trurl said:

It is impossible for Unraid to know how large a file will become when it chooses a disk for a new file. Each user share has a Minimum Free setting. If a disk has less than Minimum Free, Unraid will choose another disk for the file.

For example, the share has Minimum Free set to 10GB, and a disk has 11GB free. You write a 5GB file. Since the disk has more than minimum, it can be chosen. If the file is written to that disk, it will then have 6GB free, and won't be chosen again since it has less than minimum.

 

Another example with a different result. The share has Minimum Free set to 10GB and a disk has 11GB free. You write a 15GB file. Since the disk has more than minimum, it can be chosen. If Unraid chooses the disk, you get an error when it runs out of space.