[Plugin] Mover Tuning

[wgstarks]

40 minutes ago, MvL said:

Ah, so I have to set "Force move of all files on a schedule" to yes and set a time?

That will run mover at whatever time you set regardless of whether you have exceeded the threshold or not.

 

An example, I have mine configured to check hourly. If my threshold is exceeded (60%) then mover will run. I also set a forced move to run every 24 hours just so I don’t have files sitting in the cache for long periods if I’m not transferring many files to my shares.

[MvL]

I'm confused.., so if the threshold is reached then the mover will do his work without setting a schedule? If I want every 60 minutes a move job then I have to set the schedular for 60 minutes?

[wgstarks]

6 minutes ago, MvL said:

I'm confused.., so if the threshold is reached then the mover will do his work without setting a schedule? If I want every 60 minutes a move job then I have to set the schedular for 60 minutes?

 

Here’s mine-

 

 

Mover Schedule is set to “hourly” so every hour the plugin will check to see jf the threshold (60%) has been reached. If it has then mover will run. If not, then it will check again in another hour.

 

Force move of all files is set to run every morning at 3:30 am, so at 3:30 mover will run regardless of whether the threshold has been exceeded or not.

 

Turning on “help” will give you a brief explanation of each setting.

[MvL]

Ah yes, now it's clear. I did see the plugin mover and unRAID mover as 2 separate things but the work together. It's just a enhancement. Like the plugin name says. 

 

Maybe it's a idea to make this more clear that these functions work together thus unRAID OS schedule checker and setting the threshold.

 

Going to play a bit with setting and check if it's working now.

 

Thanks wgstarks!

[wgstarks]

If you really just want to run mover hourly you may not need this plugin. It just adds some advanced scheduling features.

[MvL]

I want when the disk is for example 75 percent full that everything is moved to the array.

[wgstarks]

In that case you can set the mover schedule to hourly and the threshold to 75%. The plugin will check every hour to see if it’s 75% full yet and when it is it’ll run mover. Just remember that depending on the size of your cache drive and the files you’re transferring it might take a while to reach 75%.

[MvL]

Thanks for clarifying! 

[DZMM]

Is there a way to check manually via a script more frequently if the threshold has been passed e.g. every 30 or 15 minutes, and if so run mover? Every hour is too late for me as I have a fast connection .  If I install the old script will they conflict?

 

Thanks

[JonathanM]

4 hours ago, DZMM said:

Is there a way to check manually via a script more frequently if the threshold has been passed e.g. every 30 or 15 minutes, and if so run mover? Every hour is too late for me as I have a fast connection .  If I install the old script will they conflict?

 

Thanks

I strongly recommend rethinking your arrangement if you are running mover that often. It would probably be much more efficient and faster overall to download or unpack directly to the array instead of the cache.

 

Normally the cache drive is used for 2 primary functions. Keep the array spun down as much as possible, and speed up user initiated new data writes. If you need to run the mover that often, your destination drive will stay spun up anyway, and the new data is coming from a source that you aren't directly waiting on, like if you were copying a file from your desktop to the server.

[DZMM]

6 minutes ago, jonathanm said:

I strongly recommend rethinking your arrangement if you are running mover that often. It would probably be much more efficient and faster overall to download or unpack directly to the array instead of the cache.

 

Normally the cache drive is used for 2 primary functions. Keep the array spun down as much as possible, and speed up user initiated new data writes. If you need to run the mover that often, your destination drive will stay spun up anyway, and the new data is coming from a source that you aren't directly waiting on, like if you were copying a file from your desktop to the server.

I understand what you are saying.  That's exactly how I use my cache 90% of the time.

 

The problem I have is if I suddenly have a lot of writes to my cache, because my download speed is fast it can fill up quickly.  Normal usage is the mover won't run more than once a day, but if it does fill fast I have a problem at the moment.  I just want a bit of insurance by being able to poll more often.

 

It's also hard for me to seperate 'background writes' from 'foreground writes' i.e. yes I'm ok with movies getting downloaded and written to /movies_adults slowly, but when I'm manually ripping or moving a movie to the same directory I want it to happen quickly.

 

 

[JonathanM]

2 hours ago, DZMM said:

It's also hard for me to seperate 'background writes' from 'foreground writes' i.e. yes I'm ok with movies getting downloaded and written to /movies_adults slowly, but when I'm manually ripping or moving a movie to the same directory I want it to happen quickly.

Not hard at all. Set the download destination in your docker to /mnt/user0/movies_adults.

That way downloads go directly to the array drives, but writes to the user share movies_adults go to the cache drive.

[DZMM]

9 minutes ago, jonathanm said:

Not hard at all. Set the download destination in your docker to /mnt/user0/movies_adults.

That way downloads go directly to the array drives, but writes to the user share movies_adults go to the cache drive.

That would still mess up my flow or lead to inefficiencies as I can't fully split my flows e.g. I download tv show via   /mnt/user0/downloads --->  /mnt/user0/tv_shows, but then I want to manually import a file in sonarr from  /mnt/user0/downloads - I want it to happen fast so I want to use the cache drive not /mnt/user0/tv_shows.

 

Also, by pooling my writes (normally max two big writes per day max) means my array drives spin up fewer times so I get less noise, and the turbo write plugin kicks in nicely to make the array write go quickly in one big chunk, rather than drives spinning up and down all day.

 

I'm happy with my flow, I've just been caught out a few times when doing a big download session

[Squid]

Two choices.

You can manually modify shares.cfg and change the cron entry there or you can create a user script to run mover (would probably need to specify the complete path in the script)

AFK though and I can't remember the logic I'm using to determine if it's the forced mover or the scheduled mover running. Will have to check tomorrow

Sent from my SM-T560NU using Tapatalk

[DZMM]

1 hour ago, Squid said:

Two choices.

You can manually modify shares.cfg and change the cron entry there or you can create a user script to run mover (would probably need to specify the complete path in the script)

AFK though and I can't remember the logic I'm using to determine if it's the forced mover or the scheduled mover running. Will have to check tomorrow

Sent from my SM-T560NU using Tapatalk
 

thanks

[Shyrka973]

Hi,

 

I do not know if it's due to this plugin, but after pressing "Move now", more than 100Go moved from array to cache. This concerns two shares that have the option "Use cache disk" to "prefer".
These two shares are no longer on any of the three disks in the array, but only on the cache.
I'm surprised because mover should move data from cache to array but not vice versa. I am wrong  ?

 

How to force mover to move the data from the array to the cache ?

 

mover every hour:

Aug 25 23:00:01 Tower root: Cache used space threshhold (60) not exceeded.  Used Space: 22.  Not moving files

 

manual mover:

Aug 25 23:12:27 Tower emhttpd: req (18): shareMoverSchedule=0+*%2F1+*+*+*&shareMoverLogging=yes&cmdStartMover=Move+now&csrf_token=****************
Aug 25 23:12:27 Tower emhttpd: shcmd (21540): /usr/local/sbin/mover |& logger &
Aug 25 23:12:27 Tower root: Manually starting mover
Aug 25 23:12:27 Tower move: move: file /mnt/disk1/Logiciels/Mac/Apple/xxx.dmg
Aug 25 23:12:48 Tower move: move: file /mnt/disk1/Logiciels/Mac/Apple/xxxxx].d

 

[itimpi]

Mover can move files in either direction!  What you see is expected behaviour if you have set a user share to Use Cache=Prefer.   You want Use Cache=Yes for mover to move files from cache to array.    Turn on the Help in the GUI to get more details on how the setting affects mover.

Edited August 26, 2018 by itimpi

[Shyrka973]

7 hours ago, itimpi said:

Mover can move files in either direction!  What you see is expected behaviour if you have set a user share to Use Cache=Prefer.   You want Use Cache=Yes for mover to move files from cache to array.    Turn on the Help in the GUI to get more details on how the setting affects mover.

 

I just tried it and it works: the data comes back from cache to array.

I have only one SSD for the cache. I have to buy another one to secure my data.

 

Thank you very much.

[userdefault]

I am having trouble getting this to work. It will let my cache fill up and never run mover, what are some troubleshooting steps I should do?

[Squid]

9 hours ago, userdefault said:

I am having trouble getting this to work. It will let my cache fill up and never run mover, what are some troubleshooting steps I should do?

Enable log when not moving due to rules, a screenshot of the settings, and the diagnostics

[alturismo]

Hi, may a suggest for another option

 

free to a specified amount and move by age.

 

sample

 

mover runs when cache >85 %, then free up to <50 % by moving the "oldest" files.

 

Like this more "actual" files keep in cache, example like media elements, actual records and stuff, i assume drives would have less spinups ...

 

just as suggestion

 

actually this script is an awesome improvement already, thanks alot herefore

[Squid]

Nothing I can do about that.  The plugin just executes the existing mover script at criteria you specify.  Since the script doesn't support what you'd like, I can't

[alturismo]

thought so already  nvm, its great already anyway

[pingu3000]

what do i have to do if i want the mover to run on a schedule and when the cache size is filled to a certain percentage (not bound to a schedule, but triggered at x%)? I understand that with this plugin it is possible to do this every 1 hour, but this is not refined enough for me since it is bound to a schedule and the cache drive can easily be filled within 1 hour.

 

i am pretty astonished that this is is still not included in unraid, seems like such a basic feature to prevent unnecessary startups of the array while also assuring that the cache drive is used most efficiently.

[trurl]

If mover is running more frequently it's going to impact performance since it has to write the parity array. A better approach would be to reevaluate how and why you cache. Sometimes it's better to just write directly to the array, possibly with Turbo Write.