[Plugin] Mover Tuning

[hugenbdd]

1 hour ago, dopeytree said:

If I use the move files at 70% of cache does that mean it will keep 30% of the cache files on drive???

 

Trying to think of a simple solution to prefer files on cache but move some when necessary BUT the priority is to have files on the cache.

 

So I would like to maintain 70% utilisation of my cache drives rather than mover dumping everything when the cache reaches 70% fullness as then it would be 0% which takes a while to fill up so loose the benefit of the SSD.

 

Any ideas?

It's the reason I took over the plug-in.  But I can't do it exactly that way.

 

Mover only kicks in (moves files to the array) if the disk utilization target it met.  Once it is met, then it starts working on the other filters.  To accomplish this, I put a filter on, based on how old a file is.

 

Below, Scheduler checks every morning (1:45am), then Mover only kicks in if disk is at 50%, and then it only moves files older than 15 days.

 

You will have to find a balance to your system.  It's not perfect, but it works pretty well.

 

[dopeytree]

Thanks great job with the plugin.

Do you personally use the setting c-time?

[hugenbdd]

I do not. As my cache is large enough to hold another day (Or whatever the difference may be) by not adding that flag.

[chris smashe]

My move when the drive gets to a certain percentage is no longer working. I am pretty sure it used to work but now i am unable to get it to fire. 

 

My settings are below

 

 

 

I have it set so when it gets to 70% run mover but I am at 70% (and i have tried higher) and mover does not start

 

 

Any ideas?

 

Thanks

 

[hugenbdd]

2 minutes ago, chris smashe said:

My move when the drive gets to a certain percentage is no longer working. I am pretty sure it used to work but now i am unable to get it to fire. 

 

My settings are below

 

 

I have it set so when it gets to 70% run mover but I am at 70% (and i have tried higher) and mover does not start

 

Any ideas?

 

Thanks

 

Set the first % to a small number instead of 0. Like 5 or 10%.

 

Also, turn on logging, so that I can see what it's trying to do.  You should be able to see the entries in the syslog or by clicking the log icon (Upper right).

[chris smashe]

Thanks @hugenbdd. I clicked the mover button by accident. Once this is done running, I will try your suggestions and let you know. 

[chris smashe]

@hugenbdd

I updated the only move threshold to 5% and the move all from cache percentage to 20% so it will run faster. I also turned on the logs. 

 

It still does not run and nothing about the mover is shown in the logs. 

 

 

 

 

[hugenbdd]

4 hours ago, chris smashe said:

@hugenbdd

I updated the only move threshold to 5% and the move all from cache percentage to 20% so it will run faster. I also turned on the logs. 

 

It still does not run and nothing about the mover is shown in the logs. 

 

 

 

 

It will only kick off on your daily schedule.  Which looks to be at 3am.

[chris smashe]

3 minutes ago, hugenbdd said:

It will only kick off on your daily schedule.  Which looks to be at 3am.

@hugenbddwhat is the Move All from Cache-yes shares pool percentage for then? Does it not start the mover once the drive gets to a certain percentage full?  

[hugenbdd]

10 minutes ago, chris smashe said:

@hugenbddwhat is the Move All from Cache-yes shares pool percentage for then? Does it not start the mover once the drive gets to a certain percentage full?  

Everything is kicked off by the schedule.  there is no "watcher" for the cache.

 

"Set to "Yes" if you want to move all files from a Cache-Yes share to the array if the percentage below is exceeded. This is similar to the original mover and does not apply any filters from this plug-in to the find command sent to the mover binary."

[chris smashe]

3 minutes ago, hugenbdd said:

Everything is kicked off by the schedule.  there is no "watcher" for the cache.

 

"Set to "Yes" if you want to move all files from a Cache-Yes share to the array if the percentage below is exceeded. This is similar to the original mover and does not apply any filters from this plug-in to the find command sent to the mover binary."

Ok. I must have misunderstood what it did. I was trying to have something that would start the mover if i went crazy with the downloads and the cache drive filled up before the scheduled task fired. 

that would be very helpful if it could be added.

 

Thanks for your help

 

[Kilrah]

21 minutes ago, chris smashe said:

I was trying to have something that would start the mover if i went crazy with the downloads and the cache drive filled up before the scheduled task fired. 

Since moving to array would likely be slower than the downloads it typically wouldn't help. The right thing to do is set up min free space properly so things overflow to the array in this case.

[chris smashe]

10 minutes ago, Kilrah said:

Since moving to array would likely be slower than the downloads it typically wouldn't help. The right thing to do is set up min free space properly so things overflow to the array in this case.

I had it set to 70% so it would start moving before it was full to account for that. 

I have since changed out my cache drive for a bigger one today so it won't be such an issue, but it would be a nice feature and one that I thought the plugin already did.

 

[hugenbdd]

30 minutes ago, chris smashe said:

I had it set to 70% so it would start moving before it was full to account for that. 

I have since changed out my cache drive for a bigger one today so it won't be such an issue, but it would be a nice feature and one that I thought the plugin already did.

 

The closest way to that, is by setting the mover to hourly and having a large enough cache (1TB or more).  Or if you want to you can dig into a custom script and put it in a cron job for every minute.  I just don't think it's worth coding.

[Arch]

Hello everyone! Does this plugin compatible with unRaid 6.9.2? for me invalid cron generated each time.

 

# Generated schedule for forced move
 /usr/local/sbin/mover.old start 2>/dev/null

 

[Squid]

27 minutes ago, Arch said:

Hello everyone! Does this plugin compatible with unRaid 6.9.2? for me invalid cron generated each time.

 

# Generated schedule for forced move
 /usr/local/sbin/mover.old start 2>/dev/null

 

You really should upgrade to 6.11.x

 

But, you'd need to show a screenshot of your settings

[Arch]

28 minutes ago, Squid said:

You really should upgrade to 6.11.x

 

But, you'd need to show a screenshot of your settings

 

found it Cron Schedule to force move all of files was empty

thanks

Edited November 26, 2022 by Arch

[eshchar]

Is there a chance to add an option to run the mover every day at night but stop the mover at a specific time? no mater if it finished or not?

[hugenbdd]

3 hours ago, eshchar said:

Is there a chance to add an option to run the mover every day at night but stop the mover at a specific time? no mater if it finished or not?

You could setup a cron job to stop it with the command below.  This should provide a "soft" stop of mover.

 

mover stop

 

 

[eshchar]

On 12/1/2022 at 4:21 PM, hugenbdd said:

You could setup a cron job to stop it with the command below.  This should provide a "soft" stop of mover.

 

mover stop

 

 

I know the command what I am asking for is a auto scheduler that will do that automatically.

[trurl]

1 hour ago, eshchar said:

I know the command what I am asking for is a auto scheduler that will do that automatically.

User Scripts plugin

[Brandon_K]

On 11/18/2022 at 3:55 PM, hugenbdd said:

It will only kick off on your daily schedule.  Which looks to be at 3am.

I am (was?) having issues with the plugin not moving.  My goal is to only spin up the array disks and move from cache to the array if the cache was 70% full.

I didn't realize that the mover schedule was effectively a gate keeper on if the mover plugin will run.  

If I'm understanding correctly; if I have the mover schedule set to run every 8 hours and the tuning plugin set to 70% threshold used, if the cache is at 50%, it will NOT move anything nor spin up disks unless the cache is < 70%? It may wait xx days to spin up disks and move (which is what I want)?

 

[hugenbdd]

7 hours ago, Brandon_K said:

I am (was?) having issues with the plugin not moving.  My goal is to only spin up the array disks and move from cache to the array if the cache was 70% full.

I didn't realize that the mover schedule was effectively a gate keeper on if the mover plugin will run.  

If I'm understanding correctly; if I have the mover schedule set to run every 8 hours and the tuning plugin set to 70% threshold used, if the cache is at 50%, it will NOT move anything nor spin up disks unless the cache is < 70%? It may wait xx days to spin up disks and move (which is what I want)?

 

Correct.  Mover doesn't run until it's scheduled time.

Once it is kicked off based on the schedule, the first thing it does it check the % used, if it reaches the threshold it will continue to move files, if the % used is less than the threshold, it will not move files.

[-Daedalus]

I think this has been asked before, but it was a while ago, so forgive me if you end up repeating yourself:

 

Is there an option to have certain shares ignore tuning rules, but still get moved on schedule?

There's an option to have files on a file list excluded, but this seems to exclude them from the mover entirely.

 

My use-case is that I want my backups going to my cache for speed, but I don't want them sitting there for three days taking up space that could be used for actually frequently accessed files.

Edited December 12, 2022 by -Daedalus

[Brandon_K]

On 12/8/2022 at 9:38 AM, hugenbdd said:

Correct.  Mover doesn't run until it's scheduled time.

Once it is kicked off based on the schedule, the first thing it does it check the % used, if it reaches the threshold it will continue to move files, if the % used is less than the threshold, it will not move files.

I'm still having some issues that I've yet to be able to solve.

What I want is pretty simple; when the cache is >70% full, move everything (all "cache: yes" shares) to the array.  I've been playing with options and I can get it to move, but it only moves data that is above 70% and leaves everything else on cache. I'm mostly just trying to avoid spinning up disks every day if I don't have to.  It usually takes 3-5 days to get my cache disk to ~70% utilization.

My settings are below;

 

No
 *Note that setting this to yes effectively disables this plugin

Priority for mover process: Normal
Priority for disk I/O: Normal
Only move at this threshold of used cache space: 70 %
Move files off cache based on age? No
Move files that are greater than this many days old: 1
Use CTIME: No
Move files based on minimum size? No
Move files that are larger than this size (In M). 1
Move files off cache based on sparsenes? No
Move files that are greather than this sparseness: .1
Ignore files listed inside of a text file: No
File list path:
Ignore file types: No
comma seperated list of file types:
Let scheduled mover run during a parity check / rebuild: No
Force turbo write on during mover: No
Log when not moving due to rules: Yes
Force move of all files on a schedule: No
Cron Schedule to force move all of files:
Allow force mover schedule to run during a parity check/rebuild: No
Ignore All hiden files and directories: No
Script to run before mover (No checks, always runs):
Script to run after mover (No checks, always runs):
Move All from Cache-Yes shares when disk is above a certain percentage: Yes
Move All from Cache-yes shares pool percentage: 75 %
Move Now button follows plug-in filters: Yes
Test Mode: No

 

What am I missing or doing wrong?