[Plugin] Mover Tuning

[evan326]

Thank you. That command only shows 1.

Quote

root@Garfield:~# df --output=pcent /mnt/ssd_unraid_cache/
Use%
  1%

Going within the cache shows it properly though. 

Quote

root@Garfield:~# df --output=pcent /mnt/ssd_unraid_cache/data/
Use%
 93%

 

Edited April 17, 2023 by evan326

[kysdaddy]

Hello All, 

I hope that I am in the right spot to ask this question.

I am not a noob but this is most likely a noob question.

I have 2 cache pools (249gb and 1 tbp), both have 2 disks setup raid1.

When I first set up my VM I set my domains share to prefer cache_240gb. Since I created my VM for Home assistant, it has grown to 32 GB I am trying to move that domains folder to prefer cache 1tbp.

 

A couple of weeks ago, I set that share to "NO" and ran mover, I did not beleive that it did anything. Today I set that share to Prefer cache 1tbp hoping to move the domain folder. I shut down the VM service and the docker service and ran mover, I saw nothing moving . A couple of hrs later I restarted docker and vms.

 

Bottom line, I'm doing something wrong. I am attaching a diagnostic, if anyone can figure out what I have screwed up and point me in the right direction I'd really appreciate it.

 

Chas

[kysdaddy]

Opps

tower-diagnostics-20230420-1339.zip

[hugenbdd]

3 hours ago, kysdaddy said:

Hello All, 

I hope that I am in the right spot to ask this question.

I am not a noob but this is most likely a noob question.

I have 2 cache pools (249gb and 1 tbp), both have 2 disks setup raid1.

When I first set up my VM I set my domains share to prefer cache_240gb. Since I created my VM for Home assistant, it has grown to 32 GB I am trying to move that domains folder to prefer cache 1tbp.

 

A couple of weeks ago, I set that share to "NO" and ran mover, I did not beleive that it did anything. Today I set that share to Prefer cache 1tbp hoping to move the domain folder. I shut down the VM service and the docker service and ran mover, I saw nothing moving . A couple of hrs later I restarted docker and vms.

 

Bottom line, I'm doing something wrong. I am attaching a diagnostic, if anyone can figure out what I have screwed up and point me in the right direction I'd really appreciate it.

 

Chas

Should probably start a new thread as this is for the Mover Tuning Plug-in.

 

With that said...  setting cache to "NO" will mean mover will never look at the files that are already on the cache and orphan them.

 

"Prefer" will try to move files to the cache from the array.

 

If you want to move them off, I would suggest not using the plug-in, set cache to "yes" and make sure you have the same cache paths that were set before, and hit the "move" button.   If you have logs enabled you should be able to see the files move off.  Once moved, set cache back to no.

[kysdaddy]

Moved to new topic 

Unraid OS 6 Support/ General Support/ Moving Share from one cache to another

[silverkin]

How Mover tuning plugin behaves with moving files based off on age? Let's say I have configured the threshold of used space at 70%, and want to move files only that are older than 21 days. Will it move files before filling 70%, or will it wait for it, and then only move files that are older than 21 days?

[alturismo]

4 hours ago, silverkin said:

Will it move files before filling 70%, or will it wait for it, and then only move files that are older than 21 days?

it will wait, i have 90 % and 30 days here and mover is waiting ...

[psyc3]

Hi, I am having trouble applying settings to the CA Mover plugin.  It throws up an update.php page but is blank and does nothing, setting dont apply.

I looked in the syslog and I am getting error: /update.php: missing csrf_token whenever I try to apply the settings.  I have reinstalled plugin and checked for updates to it.  Same result.  

Any ideas as to why this might be happening?

[hugenbdd]

7 hours ago, psyc3 said:

Hi, I am having trouble applying settings to the CA Mover plugin.  It throws up an update.php page but is blank and does nothing, setting dont apply.

I looked in the syslog and I am getting error: /update.php: missing csrf_token whenever I try to apply the settings.  I have reinstalled plugin and checked for updates to it.  Same result.  

Any ideas as to why this might be happening?

What release are you on?

[psyc3]

1 hour ago, hugenbdd said:

What release are you on?

2022.04.13

[hugenbdd]

Just now, psyc3 said:

2022.04.13

not the plug-in, unRAID

[psyc3]

6.11.5

[hugenbdd]

7 minutes ago, psyc3 said:

6.11.5

Shouldn't be an issue.

 

I would suggest uninstalling, re-installing.

 

Make one small change to the config, save.

 

Then leave the schedule page and come back to make the rest of your changes.

[psyc3]

6 minutes ago, hugenbdd said:

Shouldn't be an issue.

 

I would suggest uninstalling, re-installing.

 

Make one small change to the config, save.

 

Then leave the schedule page and come back to make the rest of your changes.

I will try what you are suggesting but i have uninstalled a few times to try to fix it.  I will let you know.

[psyc3]

Same.

But.... I uninstalled the plugin and tried to change scheduler without it installed; that's giving the same issue.  Odd because I have used both the scheduler and the plugin previously with no issue.

 

----

Tried in safe mode, the scheduler works.  I will see if there are any other plugins i installed for scheduler and refresh them. then get back 2 u.

 

----

It seems the culprit is Dynamix Share Floor settings plugin.  I think they broke something with the update; its related to php.  I only noticed the problem when trying to change my mover tuning, so presumed it was that.  Apologies.  Thanks for help, you still prompted me to look into the installation within scheduler.

Edited April 24, 2023 by psyc3

[itimpi]

I looked at a share (in Unraid 6.12-rc4) that was of the type previously known as 'Prefer' and the wording in the Mover Tuning section was:

Move all files from cache pool selected above

i.e. from' rather than 'to'    

 

Also would it now be better to simply say 'pool' rather than 'cache pool' ?

[mrpops2ko]

i'm assuming this isn't possible, since the options don't seem to exist but is it possible to configure mover to work based upon atime and set it to move around 10%?

so anytime I hit (my cache is an 8tb nvme cache) 7tb used, it'll clear out 1tb of the least accessed content, onto spinning rust

 

can i accomplish this and if so, how?

[hugenbdd]

2 hours ago, mrpops2ko said:

i'm assuming this isn't possible, since the options don't seem to exist but is it possible to configure mover to work based upon atime and set it to move around 10%?

so anytime I hit (my cache is an 8tb nvme cache) 7tb used, it'll clear out 1tb of the least accessed content, onto spinning rust

 

can i accomplish this and if so, how?

No it's not an option.  Best you can do is try and hit that number with "days old" and %.  Can usually get close enough.

 

Before that "move 10% of files" comes around I have to change to a filelist for moving files.  I have this completed just not released.  Then the data of filepath/name and size is tracked, and it would be possible to add this option.  However, I suspect that this is probably months away at the earliest.

[mrpops2ko]

12 hours ago, hugenbdd said:

No it's not an option.  Best you can do is try and hit that number with "days old" and %.  Can usually get close enough.

 

Before that "move 10% of files" comes around I have to change to a filelist for moving files.  I have this completed just not released.  Then the data of filepath/name and size is tracked, and it would be possible to add this option.  However, I suspect that this is probably months away at the earliest.

 

Thank you, i don't quite understand how to perfect the settings to accomplish this. 

These are mine currently but forgive my lack of process understanding, wouldn't this mean that once my cache hits 50% utilisation it would immediately transfer ALL files that exceed 15 days and not cease transferring them until it satisfies the <14 days requirement? because that'll be a show stopper if so. 

Or does mover just instantly stop the moment I reach below 50%? because I tested this and had some strange observations, I hit the 50% and then now my cache is at 44%, so some files got transferred. I'm assuming it moves files down to the nearest neighbour % interval? 

[hugenbdd]

5 hours ago, mrpops2ko said:

 

Thank you, i don't quite understand how to perfect the settings to accomplish this. 

These are mine currently but forgive my lack of process understanding, wouldn't this mean that once my cache hits 50% utilisation it would immediately transfer ALL files that exceed 15 days and not cease transferring them until it satisfies the <14 days requirement? because that'll be a show stopper if so. 

Or does mover just instantly stop the moment I reach below 50%? because I tested this and had some strange observations, I hit the 50% and then now my cache is at 44%, so some files got transferred. I'm assuming it moves files down to the nearest neighbour % interval? 

 

It won't stop once it hits a certain percentage moved.  The Percent used is what will allow it to "kick off" mover.  i.e. mover doesn't move anything unless the drive is used equal to or greater than the percent you choose.  Then it applies the rest of the config items to "reduce" the amount of files returned in the find command.

 

Each persons setting will be a bit different based on how much they download.  If you are in a stage where you are mass filling a server with media, it's recommended not to use cache.  If you are onto your "daily" gathering of media and it's always about the same each day, then you can really hone in on the settings.

 

If you're onto your daily gathering, I would suggest setting to 75%, and 40 or more days.   This will take some testing to get close to your desired results.  Some days you will move more files than others.  But it allows you to utilize the cache a bit better.

[mrpops2ko]

43 minutes ago, hugenbdd said:

 

It won't stop once it hits a certain percentage moved.  The Percent used is what will allow it to "kick off" mover.  i.e. mover doesn't move anything unless the drive is used equal to or greater than the percent you choose.  Then it applies the rest of the config items to "reduce" the amount of files returned in the find command.

 

Each persons setting will be a bit different based on how much they download.  If you are in a stage where you are mass filling a server with media, it's recommended not to use cache.  If you are onto your "daily" gathering of media and it's always about the same each day, then you can really hone in on the settings.

 

If you're onto your daily gathering, I would suggest setting to 75%, and 40 or more days.   This will take some testing to get close to your desired results.  Some days you will move more files than others.  But it allows you to utilize the cache a bit better.

 

hahahah ok well thats a show stopper. i'm thinking though if its possible to execute scripts (does that work?) then we can fire off something something which starts a cron to invoke 'mover stop' after x period of time, and we use the maths to figure out how long it'd take to shift y amount of data (so in my case 1tb) my array performs on average 120 MB/s per drive. (assuming large sequential file transfer) 

 

(1 tb) / (120 (MB / s)) = 2.31 hours

 

so i just need to do a daily mover, for 2 hours to shift like 800gb of data whenever its filled 85% and the script to stop it. that should in theory allow us to do it that way right?

[a632079]

Hello, I am using Unraid v6.12.0-rc5 with ZFS based cache pool. The `Pool Pct Used` value always be `1%` because this

command just fetch `/mnt/cache`'s percentage usage.

The `df -h` messae:

```

cache           142G  128K  142G   1% /mnt/cache
cache/appdata   142G  165M  142G   1% /mnt/cache/appdata
cache/system    147G  5.9G  142G   4% /mnt/cache/system
cache/domains   165G   24G  142G  15% /mnt/cache/domains

```

So, we can't get zpool usage just by df.

 

Maybe we can try zpool command to get usage:

```

zpool get -o value capacity cache | tail -n1

```

it returns

```

86%

```

 

 

I have temporary changed this line in mover_age:

```

POOLPCTUSED=`df --output=pcent /mnt/$CACHEPOOLNAME | tail -n 1 | tr -d '%'`

```

to

```

POOLPCTUSED=`zpool get -o value capacity cache | tail -n 1 | tr -d '%'`

```

 

Maybe we should determine which command should be used by judging which fs is used.

Edited May 11, 2023 by a632079

[hugenbdd]

Mover Tuning - New Release - 5-16-2023

 

Major update to 1 file.  /usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

Known Issues: On the first run an error will be posted in logs that files in /tmp/Mover are not accessable.  Can be ignored, as /tmp/Mover is created by the second run. (See above post, may not work with ZFS caches on RC5)

 

Change Details:  This release changes the age_mover bash script which does most of the work for the plug-in.  In the past it would move files similar to the original mover script by sending the output of the find command to the binary mover.  This change, creates a file list of all files that to be moved based on the filters.  Then loops through the file list sending each file to the mover binary.  This will allows us to know a few things.  total number of files to move, size of each file, how much is being moved and how much is left.  I also suggest that you run in test mode and with logging enabled.  In Test mode, there is a 2 second wait between files so that you can review the GUI on the tower/Main page near the "MOVE" button.

 

If you are brave enough to update unRAID files on your own, you can download the following files from my github.

https://github.com/hugenbd/ca.mover.tuning

/usr/local/emhttp/plugins/ArrayOperation.page
/usr/local/emhttp/plugins/dynamix/nchan/parity_list

 

#Create backup of ArrayOperation.page
cp /usr/local/emhttp/plugins/ArrayOperation.page /usr/local/emhttp/plugins/ca.mover.tuning/ArrayOperation.page-premovertuning

 

#Place the new files in their respective directories

#Set Permissions
#Read Only ArrayOperation.page
chmod 644 /usr/local/emhttp/plugins/ArrayOperation.page
chown root:root /usr/local/emhttp/plugins/ArrayOperation.page

 

#All, read/execute, read/execute for parity_list
chmod 755 /usr/local/emhttp/plugins/dynamix/nchan/parity_list
chown root:root /usr/local/emhttp/plugins/nchan/parity_list
 

[veri745]

On 9/15/2022 at 6:44 AM, kim_sv said:

@hugenbdd any possibility that you could show a setup of how to ignore a folder, using the text file method? 

...

 

The path it points to: /mnt/cache/appdata/plugins/CA_Mover_Tuning/ignore_file_list.txt

 

This is how the txt-file is written:

 

I tried both with and without the second line with the *.

 

Please, any suggestion on how to solve this?

 

 

So what is the proper way to exclude a folder?  I've seen the question asked several times but no definitive recommendations or answer.

Do I need a trailing `/`?

Does the script accept wildcards in the filename?

Should I point it to /mnt/cache/<Share> or /mnt/user/<Share>?

 

It might be good to put something in the actual help text of the "File list path" field in the UI, or add something to the pinned messages so people don't have to dig through 40 pages of posts

 

So is it

A) /mnt/user/<Share>/folder
  
B) /mnt/user/<Share>/folder/
  
C) /mnt/user/<Share>/folder/*
  
D) /mnt/cache/<Share>... and one of A, B, or C
  
E) Some combination

 

Edited May 16, 2023 by veri745
which option

[hugenbdd]

54 minutes ago, veri745 said:

So what is the proper way to exclude a folder?  I've seen the question asked several times but no definitive recommendations or answer.

Do I need a trailing `/`?

Does the script accept wildcards in the filename?

Should I point it to /mnt/cache/<Share> or /mnt/user/<Share>?

 

It might be good to put something in the actual help text of the "File list path" field in the UI, or add something to the pinned messages so people don't have to dig through 40 pages of posts

 

So is it

A) /mnt/user/<Share>/folder
  
B) /mnt/user/<Share>/folder/
  
C) /mnt/user/<Share>/folder/*
  
D) /mnt/cache/<Share>... and one of A, B, or C
  
E) Some combination

 

Few Things.

1.) You should store the ignore file somewhere on the array and not a cache file.  In case it gets moved to the array.  I use /mnt/user/Backups/ignore-file.txt

2.)  D   - Mostly.  An example folder to ignore would be /mnt/cach/TV  You can make this even more specific, a / at the end is okay, however wildcards are not.  Make sure that this file is not created in a Windows OS as it will leave ^M (Control M) characters at the end of the lines and cause it to fail.

 

I would suggest, enable logging, set Test Mode=Yes and Move Now follows plug-in filters.  Then review what gets moved.  If your on the latest version.  The logs will be under /tmp/Mover.  If you on an old version, then review the syslog.  The test mode really helps you get the settings correct quickly with the "move now"