[Plugin] Mover Tuning

[Wolfy]

Apologies if this has already been asked, but how do I set mover settings per share? I have one share that I want files moved to the array much more often (basically just using my cache as a write cache, and want the files to be protected on the array), and a second share that I want files to stay in the cache as long as possible (for use while keeping my drives spun down, but eventually move to the array for long term storage). Is this possible?

[kizer]

On 7/28/2022 at 3:11 PM, Wolfy said:

Apologies if this has already been asked, but how do I set mover settings per share? I have one share that I want files moved to the array much more often (basically just using my cache as a write cache, and want the files to be protected on the array), and a second share that I want files to stay in the cache as long as possible (for use while keeping my drives spun down, but eventually move to the array for long term storage). Is this possible?

 

There really isn't a setting per share, however you can go into your particular share settings and click the Mover Tuning - Move All Files From Share at the top and then click on the Move now button. It'll move only that share. 

 

There is a command line option you can use with cron that would move shares individually, but I don't recall the code. It is buried somewhere in all these comments thou. 

 

 

[zoggy]

minor cosmetic bug, it shows the same log entry but one is skipfile list one is skipfiles type :

 

        mvlogger "No Skipfiles Argument Supplied"

        mvlogger "No Skipfiles Argument Supplied"

 

https://github.com/hugenbd/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L149

 

kinda shocked there isnt an option to skip a folder, as lots of people probably want to skip their seeding/incomplete folder so mover doesnt move files out from under a downloader (let alone waste disk io and slow everything down). guessing people just get creative and try to just set folder as hidden? or use move by age and set the age high engouh?

 

 

 

[kizer]

1 hour ago, zoggy said:

minor cosmetic bug, it shows the same log entry but one is skipfile list one is skipfiles type :

 

        mvlogger "No Skipfiles Argument Supplied"

        mvlogger "No Skipfiles Argument Supplied"

 

https://github.com/hugenbd/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L149

 

kinda shocked there isnt an option to skip a folder, as lots of people probably want to skip their seeding/incomplete folder so mover doesnt move files out from under a downloader (let alone waste disk io and slow everything down). guessing people just get creative and try to just set folder as hidden? or use move by age and set the age high engouh?

 

 

 

 

The old Mover, which I've not tried with this you would simply put a dot in front of the folder and it wouldn't move it. 

Example .Downloads

[eatoff]

First up, great plugin, has definitely reduced the power usage on my server using this plugin and having the drives spun down.

 

Feature request - settings per cache pool.

 

Use case - I currently have 2 cache pools, 1 for general data ingestion, VMs, docker etc, and a second for surveillance footage. I would like to have the criteria for the mover running different between the two caches. the surveillance i just want it to run when its at say 80% full regardless of file age, but on my data cache i want to keep files on there for up to 20 days before moving them off.

 

I get to this odd scenario where the drives will spin up every hour where there is footage there that is more than 20 days old, but the cache is under 80% full. then the drive will slowly fill up, and all data comes off the surveillance cache and its fine again.

[Nanobug]

Hello Unraid forums,

 

I've a problem with my mover.

This is my settings for the regular mover atm:

For some reason, it doens't work.

 

For the Mover Tuning pluging (I know it's disabled) this is what I have:

 

I can't figure out why it's not working.

 

What I'd like to end up with, is that it moves it daily at 3 AM or when the cache it's 70 % used.

 

I've tried changing a few things at a time, but for some reason, it doesn't move anything automatically.

Does any of you know why, or can point me in the right direction?

[Kilrah]

 

Just like the text says you disabled it completely. 

[Nanobug]

2 minutes ago, Kilrah said:

 

Just like the text says you disabled it completely. 

It's not working if I enable it either.

That what I wrote as well.

[Kilrah]

You told it to only move files that are >10 days old, is that intended?

You also told it "Force move all files on a schedule" which would bypass most of your settings

Edited August 16, 2022 by Kilrah

[Nanobug]

3 minutes ago, Kilrah said:

You told it to only move files that are >10 days old, is that intended?

You also told it "Force move all files on a schedule" which would bypass most of your settings

Greater than 10 days old, yes.

The other one was just tro try if it worked.

 

As I wrote, I've tried multiple settings, and the pictures was just where I'm at now.

[kizer]

Ok the force all to move you don't want that on. You want to set your Mover Schedule which you have to every hour. That's where you want it to be 3AM. 

 

Since you have your threshold set to 70% unless your drive is at 70% its not going to move anything. I set mine low so it signals the script to run to 5% I believe then I set my days to 15.  The reason I set my percentage so low it makes the script check every day at 5AM and since I know its going to have more than 5% it will only move files that are 15 days. 

 

Basically the script operates on an either/or or both. Write now your telling the system  Don't move my files unless the drive is at 70% and the files must be at least 10 days old. 

 

[Nanobug]

17 hours ago, kizer said:

Ok the force all to move you don't want that on. You want to set your Mover Schedule which you have to every hour. That's where you want it to be 3AM. 

 

Since you have your threshold set to 70% unless your drive is at 70% its not going to move anything. I set mine low so it signals the script to run to 5% I believe then I set my days to 15.  The reason I set my percentage so low it makes the script check every day at 5AM and since I know its going to have more than 5% it will only move files that are 15 days. 

 

Basically the script operates on an either/or or both. Write now your telling the system  Don't move my files unless the drive is at 70% and the files must be at least 10 days old. 

 

This makes a lot more sense then.

Then I think it should be a bit different.

 

With the settings I've tried to set up now, from what I understood from your explanation, it will check at 3 AM, and move files based on one (or more) of the following conditions:

• If the disk is more than 5 % full
• If the files are more than 10 days old
• If the total disk usage is at 70 %

Is that correct?

Pictures below to see if I've done it right.

 

 

[kizer]

Ok bases on your settings my understand is the following. 

 

Since your set to hourly. Its to run the script every hour and look for the following:

At least 5% of the drive must be full.

Only move Content that is older than 10 days

At 3AM force move all content...

 

If its your goal to move everything off your drive at 3AM this will do that. If its your goal to keep items that are 10 days or less on your drive this will do that, but it would be moved at 3AM no matter what its age is. 

 

I guess it just depends on what your trying to achieve. The script is pretty straight forward once you figure it out, but with all the options it does get a little confusing. 

[Nanobug]

26 minutes ago, kizer said:

Ok bases on your settings my understand is the following. 

 

Since your set to hourly. Its to run the script every hour and look for the following:

At least 5% of the drive must be full.

Only move Content that is older than 10 days

At 3AM force move all content...

 

If its your goal to move everything off your drive at 3AM this will do that. If its your goal to keep items that are 10 days or less on your drive this will do that, but it would be moved at 3AM no matter what its age is. 

 

I guess it just depends on what your trying to achieve. The script is pretty straight forward once you figure it out, but with all the options it does get a little confusing. 

The it's how I wanted it to be.

Missed the force move part though.

Changed it to this:

 

So it will move all the files the 28th of the months no matter.

I just want the data to be there for the parity check the 1st of each month when it's running.

And I couldn't find a cron expression that would go for the last day, so I went with the 28th (february).

[kizer]

Ok there you go. So now it should.........

Run every hour, look to make sure you have at least 5% and be older than 10 days. 

Even if files are not older than 10 days old will be moved on the 28th of every month. 

 

At least from what I'm looking at it should work the way you want. 😃

[Nanobug]

1 hour ago, kizer said:

Ok there you go. So now it should.........

Run every hour, look to make sure you have at least 5% and be older than 10 days. 

Even if files are not older than 10 days old will be moved on the 28th of every month. 

 

At least from what I'm looking at it should work the way you want. 😃

Wait...
It's BOTH older than 10 days AND 5 % used?
Or is it one or the other?

[kizer]

Normally if I understand correctly. 

It'll check the drive to see how much space you have. If it has that much spaced used it moves on to the next item of how old. 

That's why I use 5% because its as low as the settings allow and I always have at least 5% used. Then it checks files and says these are 10 days old and moves. Only files that are 10days old. 

[zoggy]

if you are confused, just enable mover logging, enable test mode on the mover tuning plugin, and set the "Move Now button follows plug-in filters:" to yes. so you can just fire off the script and see what it would move. tweak values and repeat. once all good, turn off test mode...

 

testing myself as i have it set to move stuff older than 10 days and "Move All from Cache-yes shares pool percentage:" set to 50%.  with mover running weekly.

 

with only 18% cache used, and a few files older than 10 days. firing it off.. it moved a few files older than 10 days. so the logic to move by age does not limit by the cache usage.

Edited August 17, 2022 by zoggy

[Serverus]

How could I add a notification for when the Mover is invoked?  I only have files move when my cache hits 30% capacity, so not that often.  Would be a nice optional feature to brew into the app.

[trurl]

5 minutes ago, Serverus said:

How could I add a notification for when the Mover is invoked?  I only have files move when my cache hits 30% capacity, so not that often.  Would be a nice optional feature to brew into the app.

The plugin already allows you to run a script before and/or after moving.

 

 

[diffz]

Hi,

 

I have a share data where all my media and torrents are stored. I want my torrents folder to stay on the cache so i created a text file to ignore /mnt/user/data/torrents/ when I read the logs i see it's excluding /mnt/user/data/torrents/ and all the files inside but it's also excluding the whole share data and subfolders not listed in the text file.

 

is what i want even possible?

Creating a separate share for torrents is not an option because i want radarr/sonarr to hardlink/instant move torrents to my media folder.

[kim_sv]

On 9/2/2022 at 1:52 PM, diffz said:

Hi,

 

I have a share data where all my media and torrents are stored. I want my torrents folder to stay on the cache so i created a text file to ignore /mnt/user/data/torrents/ when I read the logs i see it's excluding /mnt/user/data/torrents/ and all the files inside but it's also excluding the whole share data and subfolders not listed in the text file.

 

is what i want even possible?

Creating a separate share for torrents is not an option because i want radarr/sonarr to hardlink/instant move torrents to my media folder.

I've done the same thing, and it works for me. I think your problem is the trailing / in your path. Try it like this instead:

Quote

/mnt/user/data/torrents

Never mind. I thought it worked but no...

 

Looking at the log (enabled logging, and looking at unraids logs) I see that it skipped one file then transferred everything else in the directory.

Edited September 6, 2022 by kim_sv
My first advice was shite...

[kim_sv]

@hugenbdd any possibility that you could show a setup of how to ignore a folder, using the text file method? 

 

I've tried with this setup (Disable Mover running on a schedule: Yes in this picture is only because I want it turned off when it's not working properly)

 

The path it points to: /mnt/cache/appdata/plugins/CA_Mover_Tuning/ignore_file_list.txt

 

This is how the txt-file is written:

 

I tried both with and without the second line with the *.

 

Please, any suggestion on how to solve this?

 

 

[hugenbdd]

5 hours ago, kim_sv said:

@hugenbdd any possibility that you could show a setup of how to ignore a folder, using the text file method? 

 

I've tried with this setup (Disable Mover running on a schedule: Yes in this picture is only because I want it turned off when it's not working properly)

 

The path it points to: /mnt/cache/appdata/plugins/CA_Mover_Tuning/ignore_file_list.txt

 

This is how the txt-file is written:

 

I tried both with and without the second line with the *.

 

Please, any suggestion on how to solve this?

 

 

Change Disabled to "No", the way you have it now, it will never run mover unless hit the button.

Also, turn logging on, then we can look at the syslog and see where it might be having issues.  Might also want to turn on the test mode until we get it figured out.  That way no actual files will be moved, but logs will be created.

 

Sometimes text files have a weird end of line character (a ^) if it was created on a windows machine.

[kim_sv]

3 hours ago, hugenbdd said:

Change Disabled to "No", the way you have it now, it will never run mover unless hit the button.

Like I wrote in the post, it is only set to Yes in the screenshot due to it not working properly when I tested so I turned it off.

3 hours ago, hugenbdd said:

Also, turn logging on, then we can look at the syslog and see where it might be having issues.  Might also want to turn on the test mode until we get it figured out.  That way no actual files will be moved, but logs will be created.

I will definitely try this and get back to you when I’m back home again. 👍

4 hours ago, hugenbdd said:

Sometimes text files have a weird end of line character (a ^) if it was created on a windows machine.

I created the file on macOS and added the path lines within the unRAID UI to get proper formatting (I believe).