Mover optimization


Recommended Posts

Hi.

I've noticed that many times, when the Mover script needs to move a movie folder (which contains a large MKV file and a small SRT subtitles file) from my cache drive to the array, it moves the small subtitles file first, then fails to move the MKV file to the same drive due to not enough free space, so the MKV is getting stuck on the cache drive.

Is it possible to add a check for Mover to check the folder size & destination allocation restrictions (method + split level + free space) for the decision to which drive it should move the folder?

If it's too much to ask, maybe just making Mover move the files off the cache drive by their sizes? Starting with the biggest files I believe would resolve the problem I think.

 

Thanks

Link to comment

What is your Minimum Free for the share? You must set Minimum Free to larger than the largest file you expect to write to the share. Unraid has no way to know how large a file will become when it chooses a disk for it. If a disk has less than Minimum, it will choose another disk.

Link to comment

Please note that I'm referring to the Mover script - it's moving files already on Unraid from the cache pool/drive to the Array-protected disks, so it should already have the files sizes known.

If I'm setting a share Minimum Free restriction to 4GB, the movie file is 4.5GB and the subtitles file is 40kb, with a split level that restricts the subtitles & movie files to be on the same disk, and the mover moves the subtitles file to a disk with 4.1GB free, the movie file will be stuck on the cache drive.

If the mover script will move the files ordered from largest to smallest, the movie file will move first to a disk with enough free space, and the subtitles file will be moved to the same disk...

Link to comment
1 hour ago, gshlomi said:

Please note that I'm referring to the Mover script - it's moving files already on Unraid from the cache pool/drive to the Array-protected disks, so it should already have the files sizes known.

If I'm setting a share Minimum Free restriction to 4GB, the movie file is 4.5GB and the subtitles file is 40kb, with a split level that restricts the subtitles & movie files to be on the same disk, and the mover moves the subtitles file to a disk with 4.1GB free, the movie file will be stuck on the cache drive.

If the mover script will move the files ordered from largest to smallest, the movie file will move first to a disk with enough free space, and the subtitles file will be moved to the same disk...


The problem is that the logic for selecting a drive to use is at a much lower level in unRaid than that at which mover runs so the file size is not known at the level that selects the target drive.

 

I agree it would be nice if the size WAS taken into account before even attempting to move a file but I suspect that the changes required to achieve this may be non-trivial.   Still worth asking for in case someone at Limetech has a brainwave on an easy way to implement this.

Link to comment

I'm guessing unraid uses the same logic as "Hey we only have room for 3 people on the Elevator. So the 6 of you figure it out because 3 of you have to use the stairs."

 

Sadly I know its not the answer you wanted to hear, but if your files are that tight with space you might want to consider either upgrade your drives or do some creative file moving around. I know when I'm that low on space I start considering moving shares around or throw in larger/more drives. 

 

For the record I had the same problem happen a time or two as well..

 

 

Link to comment
On 4/10/2021 at 3:23 PM, itimpi said:


The problem is that the logic for selecting a drive to use is at a much lower level in unRaid than that at which mover runs so the file size is not known at the level that selects the target drive.

 

I agree it would be nice if the size WAS taken into account before even attempting to move a file but I suspect that the changes required to achieve this may be non-trivial.   Still worth asking for in case someone at Limetech has a brainwave on an easy way to implement this.

My assumption is that the Mover script moves the files from the cache drive/pool directly to the appropriate disk, not to the array share (which includes the cache itself), so the move operation can check all the target prerequisites before moving a file (including the target free space, file sizes & split levels), but I might be wrong...

Link to comment
35 minutes ago, gshlomi said:

My assumption is that the Mover script moves the files from the cache drive/pool directly to the appropriate disk, not to the array share (which includes the cache itself)

 

I think this is wrong and mover DOES move to the share and not directly to a drive.  I believe this is what the /mnt/user0 mount point is used for (it is the User Share including only the array disks ignoring anything on the cache).

Link to comment
4 hours ago, gshlomi said:

directly to the appropriate disk

How would you define "appropriate disk" if not based on the settings for the user share?

 

3 hours ago, itimpi said:

 

I think this is wrong and mover DOES move to the share and not directly to a drive.  I believe this is what the /mnt/user0 mount point is used for (it is the User Share including only the array disks ignoring anything on the cache).

Mover has gotten a bit more complicated over the years and with new multiple pools feature, but it is still basically moving between each /mnt/pool and /mnt/user0.

Link to comment
On 4/15/2021 at 5:56 PM, trurl said:

How would you define "appropriate disk" if not based on the settings for the user share?

 

Mover has gotten a bit more complicated over the years and with new multiple pools feature, but it is still basically moving between each /mnt/pool and /mnt/user0.

Thanks @itimpi and @trurl, forgot about /mnt/user0, so now it makes sense.

I guess it’s time to manually move some files around...

Link to comment
3 hours ago, gshlomi said:

Thanks @itimpi and @trurl, forgot about /mnt/user0, so now it makes sense.

I guess it’s time to manually move some files around...

 

I think it is never wrong to periodically bring this up as it would be a nice enhancement if mover could be made somehow to use the size of a file it is about to move as one of the selection criteria for the target disk.   It cannot be trivial to do or I am sure it would have already happened but it does seem a good idea to have as a potential roadmap item. 

 

Limetech did announce a while ago that /mnt/user0 was deprecated and could disappear in a future unRaid release so presumably they have thought of some other way of working that will not need it?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.