Jump to content
Squid

[Plugin] Mover Tuning

96 posts in this topic Last Reply

Recommended Posts

40 minutes ago, MvL said:

Ah, so I have to set "Force move of all files on a schedule" to yes and set a time?

That will run mover at whatever time you set regardless of whether you have exceeded the threshold or not.

 

An example, I have mine configured to check hourly. If my threshold is exceeded (60%) then mover will run. I also set a forced move to run every 24 hours just so I don’t have files sitting in the cache for long periods if I’m not transferring many files to my shares.

Share this post


Link to post

I'm confused.., so if the threshold is reached then the mover will do his work without setting a schedule? If I want every 60 minutes a move job then I have to set the schedular for 60 minutes?

Share this post


Link to post
6 minutes ago, MvL said:

I'm confused.., so if the threshold is reached then the mover will do his work without setting a schedule? If I want every 60 minutes a move job then I have to set the schedular for 60 minutes?

 

Here’s mine-

 

SafariScreenSnapz093.jpg.8ebd76ee2b843649f3fe47196fa9b751.jpg

 

Mover Schedule is set to “hourly” so every hour the plugin will check to see jf the threshold (60%) has been reached. If it has then mover will run. If not, then it will check again in another hour.

 

Force move of all files is set to run every morning at 3:30 am, so at 3:30 mover will run regardless of whether the threshold has been exceeded or not.

 

Turning on “help” will give you a brief explanation of each setting.

Share this post


Link to post

Ah yes, now it's clear. I did see the plugin mover and unRAID mover as 2 separate things but the work together. It's just a enhancement. Like the plugin name says. 

 

Maybe it's a idea to make this more clear that these functions work together thus unRAID OS schedule checker and setting the threshold.

 

Going to play a bit with setting and check if it's working now.

 

Thanks wgstarks!

Share this post


Link to post

If you really just want to run mover hourly you may not need this plugin. It just adds some advanced scheduling features.

Share this post


Link to post

I want when the disk is for example 75 percent full that everything is moved to the array.

Share this post


Link to post

In that case you can set the mover schedule to hourly and the threshold to 75%. The plugin will check every hour to see if it’s 75% full yet and when it is it’ll run mover. Just remember that depending on the size of your cache drive and the files you’re transferring it might take a while to reach 75%.

Share this post


Link to post

Is there a way to check manually via a script more frequently if the threshold has been passed e.g. every 30 or 15 minutes, and if so run mover? Every hour is too late for me as I have a fast connection .  If I install the old script will they conflict?

 

Thanks

Share this post


Link to post
4 hours ago, DZMM said:

Is there a way to check manually via a script more frequently if the threshold has been passed e.g. every 30 or 15 minutes, and if so run mover? Every hour is too late for me as I have a fast connection .  If I install the old script will they conflict?

 

Thanks

I strongly recommend rethinking your arrangement if you are running mover that often. It would probably be much more efficient and faster overall to download or unpack directly to the array instead of the cache.

 

Normally the cache drive is used for 2 primary functions. Keep the array spun down as much as possible, and speed up user initiated new data writes. If you need to run the mover that often, your destination drive will stay spun up anyway, and the new data is coming from a source that you aren't directly waiting on, like if you were copying a file from your desktop to the server.

Share this post


Link to post
6 minutes ago, jonathanm said:

I strongly recommend rethinking your arrangement if you are running mover that often. It would probably be much more efficient and faster overall to download or unpack directly to the array instead of the cache.

 

Normally the cache drive is used for 2 primary functions. Keep the array spun down as much as possible, and speed up user initiated new data writes. If you need to run the mover that often, your destination drive will stay spun up anyway, and the new data is coming from a source that you aren't directly waiting on, like if you were copying a file from your desktop to the server.

I understand what you are saying.  That's exactly how I use my cache 90% of the time.

 

The problem I have is if I suddenly have a lot of writes to my cache, because my download speed is fast it can fill up quickly.  Normal usage is the mover won't run more than once a day, but if it does fill fast I have a problem at the moment.  I just want a bit of insurance by being able to poll more often.

 

It's also hard for me to seperate 'background writes' from 'foreground writes' i.e. yes I'm ok with movies getting downloaded and written to /movies_adults slowly, but when I'm manually ripping or moving a movie to the same directory I want it to happen quickly.

 

 

Share this post


Link to post
2 hours ago, DZMM said:

It's also hard for me to seperate 'background writes' from 'foreground writes' i.e. yes I'm ok with movies getting downloaded and written to /movies_adults slowly, but when I'm manually ripping or moving a movie to the same directory I want it to happen quickly.

Not hard at all. Set the download destination in your docker to /mnt/user0/movies_adults.

That way downloads go directly to the array drives, but writes to the user share movies_adults go to the cache drive.

Share this post


Link to post
9 minutes ago, jonathanm said:

Not hard at all. Set the download destination in your docker to /mnt/user0/movies_adults.

That way downloads go directly to the array drives, but writes to the user share movies_adults go to the cache drive.

That would still mess up my flow or lead to inefficiencies as I can't fully split my flows e.g. I download tv show via   /mnt/user0/downloads --->  /mnt/user0/tv_shows, but then I want to manually import a file in sonarr from  /mnt/user0/downloads - I want it to happen fast so I want to use the cache drive not /mnt/user0/tv_shows.

 

Also, by pooling my writes (normally max two big writes per day max) means my array drives spin up fewer times so I get less noise, and the turbo write plugin kicks in nicely to make the array write go quickly in one big chunk, rather than drives spinning up and down all day.

 

I'm happy with my flow, I've just been caught out a few times when doing a big download session

Share this post


Link to post

Two choices.

You can manually modify shares.cfg and change the cron entry there or you can create a user script to run mover (would probably need to specify the complete path in the script)

AFK though and I can't remember the logic I'm using to determine if it's the forced mover or the scheduled mover running. Will have to check tomorrow

Sent from my SM-T560NU using Tapatalk

Share this post


Link to post
1 hour ago, Squid said:

Two choices.

You can manually modify shares.cfg and change the cron entry there or you can create a user script to run mover (would probably need to specify the complete path in the script)

AFK though and I can't remember the logic I'm using to determine if it's the forced mover or the scheduled mover running. Will have to check tomorrow

Sent from my SM-T560NU using Tapatalk
 

thanks

Share this post


Link to post

Hi,

 

I do not know if it's due to this plugin, but after pressing "Move now", more than 100Go moved from array to cache. This concerns two shares that have the option "Use cache disk" to "prefer".
These two shares are no longer on any of the three disks in the array, but only on the cache.
I'm surprised because mover should move data from cache to array but not vice versa. I am wrong  ?

 

How to force mover to move the data from the array to the cache ?

 

mover every hour:

Aug 25 23:00:01 Tower root: Cache used space threshhold (60) not exceeded.  Used Space: 22.  Not moving files

 

manual mover:

Aug 25 23:12:27 Tower emhttpd: req (18): shareMoverSchedule=0+*%2F1+*+*+*&shareMoverLogging=yes&cmdStartMover=Move+now&csrf_token=****************
Aug 25 23:12:27 Tower emhttpd: shcmd (21540): /usr/local/sbin/mover |& logger &
Aug 25 23:12:27 Tower root: Manually starting mover
Aug 25 23:12:27 Tower move: move: file /mnt/disk1/Logiciels/Mac/Apple/xxx.dmg
Aug 25 23:12:48 Tower move: move: file /mnt/disk1/Logiciels/Mac/Apple/xxxxx].d

 

Share this post


Link to post

Mover can move files in either direction!  What you see is expected behaviour if you have set a user share to Use Cache=Prefer.   You want Use Cache=Yes for mover to move files from cache to array.    Turn on the Help in the GUI to get more details on how the setting affects mover.

Edited by itimpi

Share this post


Link to post
7 hours ago, itimpi said:

Mover can move files in either direction!  What you see is expected behaviour if you have set a user share to Use Cache=Prefer.   You want Use Cache=Yes for mover to move files from cache to array.    Turn on the Help in the GUI to get more details on how the setting affects mover.

 

I just tried it and it works: the data comes back from cache to array.

I have only one SSD for the cache. I have to buy another one to secure my data.

 

Thank you very much.

Share this post


Link to post

I am having trouble getting this to work. It will let my cache fill up and never run mover, what are some troubleshooting steps I should do?

Share this post


Link to post
9 hours ago, userdefault said:

I am having trouble getting this to work. It will let my cache fill up and never run mover, what are some troubleshooting steps I should do?

Enable log when not moving due to rules, a screenshot of the settings, and the diagnostics

Share this post


Link to post

Hi, may a suggest for another option ;)

 

free to a specified amount and move by age.

 

sample

 

mover runs when cache >85 %, then free up to <50 % by moving the "oldest" files.

 

Like this more "actual" files keep in cache, example like media elements, actual records and stuff, i assume drives would have less spinups ...

 

just as suggestion ;)

 

actually this script is an awesome improvement already, thanks alot herefore ;)

Share this post


Link to post

Nothing I can do about that.  The plugin just executes the existing mover script at criteria you specify.  Since the script doesn't support what you'd like, I can't

Share this post


Link to post

what do i have to do if i want the mover to run on a schedule and when the cache size is filled to a certain percentage (not bound to a schedule, but triggered at x%)? I understand that with this plugin it is possible to do this every 1 hour, but this is not refined enough for me since it is bound to a schedule and the cache drive can easily be filled within 1 hour.

 

i am pretty astonished that this is is still not included in unraid, seems like such a basic feature to prevent unnecessary startups of the array while also assuring that the cache drive is used most efficiently.

Share this post


Link to post

If mover is running more frequently it's going to impact performance since it has to write the parity array. A better approach would be to reevaluate how and why you cache. Sometimes it's better to just write directly to the array, possibly with Turbo Write.

  • Upvote 1

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now