[Plugin] Mover Tuning

[alturismo]

24 minutes ago, akazu said:

cat /mnt/cachessd/appdata/moverignore/moverignore.txt 
/mnt/cachessd/media/Pictures/

 

like its described ... please read again

 

[akazu]

13 minutes ago, alturismo said:

like its described ... please read again

 

 

My English sir :-) Thanks as always for your help. It looks like it works

[akazu]

Thanks also to the developers who provide these useful functions. 

[akimboflix]

I'm sure this has been requested before but is it possible to move the oldest/least-accessed files first?

[methanoid]

20 hours ago, akazu said:

 

My English sir :-) Thanks as always for your help. It looks like it works

 

Done exact same thing and doesnt work for me.. or doesnt seem to

[thecyborg]

I found some unexpected behavior when updating my cache drive from 2x1TB btrfs RAID1 to a single 2TB zfs volume.  The "Only move at this threshold of used cache space" setting is compared in the mvallfrompool() function with (zpool get -o value capacity).  The output from zpool is at a granularity of 1 percent.  This means that unless the cache utilization is more than 1% (which can be pretty large on bigger cache volumes), the mover won't do anything.  Is it possible to change this to a value with higher granularity (perhaps df in bytes or similar)?

[Swarles]

On 11/1/2023 at 8:10 AM, dopeytree said:

Sure. 

it's probably something I've mis-set etc.

 

Noticed this: 

No /tmp/Mover/Custom_Mover_Directory_2023-10-30T180820.list created as no empty directories

 

Files attached.

Mover_Tuning_2023-10-30T180820.log 3.53 kB · 0 downloads Mover_Tuning_2023-10-30T180825.log 3.53 kB · 0 downloads Mover_Tuning_2023-10-30T181114.log 610 B · 0 downloads Mover_Tuning_2023-10-30T181129.log 2.51 kB · 0 downloads Mover_Tuning_2023-10-30T181154.log 2.51 kB · 0 downloads Mover_Tuning_2023-10-30T181235.log 2.51 kB · 0 downloads

I took a quick look at your latest log, my first guess is the threshold.

1) Threshold: currently set to 50%. Files won’t be moved at all until more than 50% of your cache is exceeded. From the logs it appears your highest cache is only 49% full? So you could try reducing the threshold to see if it moves the files, or loading on more content.

 

2) Also your move-all threshold is set to 95%, this won’t be causing issues but is important to know that if the cache storage ever exceeds 95% it will attempt to move everything off, ignoring all mover tuning settings.

Edited November 3, 2023 by Swarles

[Swarles]

2 hours ago, methanoid said:

 

Done exact same thing and doesnt work for me.. or doesnt seem to

What path are you using in your skip list? You will want to always avoid using “user” and instead use the name of your cache pool.

Eg.

/mnt/user/data/media/linux-isos   <--Bad
/mnt/cache/data/media/linux-isos  <--Good

 

[methanoid]

On 11/3/2023 at 2:54 PM, Swarles said:

What path are you using in your skip list? You will want to always avoid using “user” and instead use the name of your cache pool.

Eg.

/mnt/user/data/media/linux-isos   <--Bad
/mnt/cache/data/media/linux-isos  <--Good

 

 

My file is stored as /mnt/cache/appdata/ignore.files and contains

 

/mnt/cache/data/torrents/INCOMPLETE
/mnt/cache/data/usenet/incomplete
/mnt/cache/data/nzb
/mnt/cache/data/post
 

And it has just moved 140Gb of incomplete torrents from cache to array disk.  Just doesnt follow the rules as far as I can see

[Swarles]

6 hours ago, methanoid said:

 

My file is stored as /mnt/cache/appdata/ignore.files and contains

 

/mnt/cache/data/torrents/INCOMPLETE
/mnt/cache/data/usenet/incomplete
/mnt/cache/data/nzb
/mnt/cache/data/post
 

And it has just moved 140Gb of incomplete torrents from cache to array disk.  Just doesnt follow the rules as far as I can see

Hmm strange, might need to provide some logs to really get to the bottom of this. If you enable logging in the mover settings under scheduler, you can also put it into test mode, so that it doesn't move anything but will still go through the whole process. Then you can find logs in the syslog and at "/tmp/Mover".

 

My initial thoughts are:

• If you're using the mover tuning settings override for the "data" share, make sure that you enable the skip list and provide the list in those settings too.
• Double check that "Skip files listed in text file:" is set to "Yes" and that the correct path is supplied under "File list path:"
• Perhaps the list is written in CTLF instead of LF. Recreate the skip file using nano in terminal:

  nano /mnt/cache/appdata/ignore.files
  *paste in your list*
  *ctrl+o to save*
  *ctrl+x to exit*

  Run in test mode again to see if it ignores the list.

[methanoid]

7 hours ago, Swarles said:

Hmm strange, might need to provide some logs to really get to the bottom of this. If you enable logging in the mover settings under scheduler, you can also put it into test mode, so that it doesn't move anything but will still go through the whole process. Then you can find logs in the syslog and at "/tmp/Mover".

 

My initial thoughts are:

• If you're using the mover tuning settings override for the "data" share, make sure that you enable the skip list and provide the list in those settings too.
• Double check that "Skip files listed in text file:" is set to "Yes" and that the correct path is supplied under "File list path:"
• Perhaps the list is written in CTLF instead of LF. Recreate the skip file using nano in terminal:

  nano /mnt/cache/appdata/ignore.files
  *paste in your list*
  *ctrl+o to save*
  *ctrl+x to exit*

  Run in test mode again to see if it ignores the list.

 

Ah, I hadnt even noticed there were TWO sets of settings and didnt change the settings for the Data share explicitly. Suspect that will fix the issue!!

[Swarles]

1 hour ago, methanoid said:

 

Ah, I hadnt even noticed there were TWO sets of settings and didnt change the settings for the Data share explicitly. Suspect that will fix the issue!!

Yeah so it should only be the source of the issue if you had "Override Mover Tuning settings for this share:" set to "Yes", but didn't have the skip list supplied. It's a sort of feature flaw that's a result of how the per share settings is implemented. If you had it set to "No" then it should be a different problem.

[AngelAbaddon]

Not sure if I'm missing something, but is there a way to override the cache threshold setting at a share level? I've got my main mover set to move when the cache is over 70% and files are 40 days old, but I have a backup share that I want moved daily - I've set the share to override the Mover Tuning age settings, but threshold isn't one of the options, and on testing it looks like thats global - if I leave it at 70% then no move occurs, but set it to 0% and backup moves as intended while my "data" share is untouched.

[Swarles]

12 hours ago, AngelAbaddon said:

Not sure if I'm missing something, but is there a way to override the cache threshold setting at a share level? I've got my main mover set to move when the cache is over 70% and files are 40 days old, but I have a backup share that I want moved daily - I've set the share to override the Mover Tuning age settings, but threshold isn't one of the options, and on testing it looks like thats global - if I leave it at 70% then no move occurs, but set it to 0% and backup moves as intended while my "data" share is untouched.

Yeah it's not possible at the moment because the threshold is for the entire pool and in it's current state, the logic is not there to be able to easily implement it for shares. I will be looking to implement this eventually, but it will likely come with a big rewrite of the mover tuning logic so I wouldn't expect it anytime soon.

Some considerations need to be made like which thresholds take precedence.

 

For the share that you want to move daily, are you using any other mover tuning settings? There would be a way to setup a Cron job to always move that share but it would ignore the mover tuning settings, similarly to pressing the "Mover share now" button under the shares settings.

[AngelAbaddon]

7 hours ago, Swarles said:

Yeah it's not possible at the moment because the threshold is for the entire pool and in it's current state, the logic is not there to be able to easily implement it for shares. I will be looking to implement this eventually, but it will likely come with a big rewrite of the mover tuning logic so I wouldn't expect it anytime soon.

Some considerations need to be made like which thresholds take precedence.

 

For the share that you want to move daily, are you using any other mover tuning settings? There would be a way to setup a Cron job to always move that share but it would ignore the mover tuning settings, similarly to pressing the "Mover share now" button under the shares settings.

That makes sense, thought it mioght be something like that but worth just double checking.

 

Regarding the share, no additional options, I essentially wanted it to bypass all the tuning settings and follow the default mover - as you put it, no different to pressing the "Mover share now" button.

[wibble]

Did anyone manage to resolve "MOVE" and "MOVE NOW" buttons in the webUI not running mover with the plugin installed?

[Meles Meles]

I'm getting an error when mover is running (caused by dodgy data, but whatever...)

 

I have a folder in my cache which ends with a trailing space

 

"/mnt/cache/transcode/thumbs/Thumbs/2012-08-02 - "

 

the value is being written correctly into the list (/tmp/Mover/Mover_Tuning_2023-11-10T105025.list), but when it gets read back in (for the purpose of checking for hardlinks) the script is being helpful and trimming that space off the end for me..

 

mvlogger: Hard File Path: .."/mnt/cache/transcode/thumbs/Thumbs/2012-08-02 -"..

 

stat: cannot statx '/mnt/cache/transcode/thumbs/Thumbs/2012-08-02 -': No such file or directory

 

        #Test for hardlinks of the file
        LINK_COUNT=$(stat -c "%h" "$HARDFILEPATH")

 

 

downstream it's also causing failures in here...

 

totalsizeFilelist() {
    #Loop throug the custom mover filelist and add up all the sizes for each entry.

    #echo "Grabbing total filesize"

    #start looping
    while read CUSTOMLINE; do
        #PULL out file size
        mvlogger "..$CUSTOMLINE.."
        CUSTOMLINESIZE=$(echo "$CUSTOMLINE" | sed -n 's/.*;\(.*\)/\1/p')
        mvlogger "Custom Size:  ..${CUSTOMLINESIZE}.."

        ((TOTALCACHESIZE+=$CUSTOMLINESIZE))

    done <$CUSTOM_MOVER_FILELIST

}

 

 

 

Changing all (9) occurances of

 

while read 

in /usr/local/emhttp/plugins/ca.mover.tuning/age_mover to

while IFS=""; read 

appears to fix this issue, but is this going to break anything else?

Edited November 12, 2023 by Meles Meles
Update...

[danringer]

On 11/9/2023 at 3:10 AM, wibble said:

Did anyone manage to resolve "MOVE" and "MOVE NOW" buttons in the webUI not running mover with the plugin installed?

this just worked for me.  
Plugins>CA Mover Tuning>Move Now button follows plug-in filters: Yes

[Freender]

On 11/7/2023 at 6:54 AM, AngelAbaddon said:

Not sure if I'm missing something, but is there a way to override the cache threshold setting at a share level? I've got my main mover set to move when the cache is over 70% and files are 40 days old, but I have a backup share that I want moved daily - I've set the share to override the Mover Tuning age settings, but threshold isn't one of the options, and on testing it looks like thats global - if I leave it at 70% then no move occurs, but set it to 0% and backup moves as intended while my "data" share is untouched.

 

I decided to set global

"Move files that are greater than this many days old" = 1 

"Only move at this threshold of used cache space" = 0%

 

and override it for media share as

"Move files that are greater than this many days old" = 60

 

20 minutes before mover runs, I dynamically calculate daysold and update corresponding daysold parameter in  /boot/config/plugins/ca.mover.tuning/shareOverrideConfig/media.cfg

 

Script starts with 60 days and decreases by 5 (55, 50, 45 ... ) if target cache free space criteria is not met ( In my case it's 370GB).

 

I will keep it running for couple of days, if stable might share if somebody is interested

[Freender]

Script:

https://github.com/freender/mover_tunning_daysold/blob/main/calculate_daysold.sh

 

[xra]

On 11/2/2023 at 2:15 PM, akimboflix said:

I'm sure this has been requested before but is it possible to move the oldest/least-accessed files first?

That's what got me here.  I believe you can do this by setting "Move files off cache based on age:" What I don't know is if that age is based on last access time or modified time.  Where there is a setting to use CTIME I presume the default is modification time.  Not exactly what I am looking for.

 

So, I am thinking of simply adding a wild card in either the ignore file or ignore file type list to make it move nothing.  Then I have a bash script that will do the moving the way I want:

 

Move everything that has a last access age of X days.

Keep moving stuff until the volume of the source is below Y% capacity.

 

Backing up a bit, I basically want tiered storage.  Requirements:

 

1. Create a single share that consists of an SSD and a hard drive (that's easy, that is what unraid does)
2. Disable the default mover (only way I can see to do this is using mover tuning to ignore all files somehow)
3. Move everything that has a last access age (not last modified) of X days from the SSD to the hard drive
4. Keep moving stuff until the volume of the source is below Y% capacity.

 

Only  a month or so into using unraid so there may be a way to do this I just haven't found yet.

 

 

UPDATE (later that same day)

 

The age is based on modified time aka mtime.  So not last accessed time.  Seems easy enough to change or add additionally last accessed time:

https://github.com/hugenbd/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

                    if [ "$CTIMEA" == "yes" ]; then
                        FINDSTR+=" -ctime +$RAGE"

                    else
                        FINDSTR+=" -mtime +$RAGE"
                    fi
                fi

 

 

Edited November 15, 2023 by xra
more info

[hugenbdd]

2 hours ago, xra said:

That's what got me here.  I believe you can do this by setting "Move files off cache based on age:" What I don't know is if that age is based on last access time or modified time.  Where there is a setting to use CTIME I presume the default is modification time.  Not exactly what I am looking for.

 

So, I am thinking of simply adding a wild card in either the ignore file or ignore file type list to make it move nothing.  Then I have a bash script that will do the moving the way I want:

 

Move everything that has a last access age of X days.

Keep moving stuff until the volume of the source is below Y% capacity.

 

Backing up a bit, I basically want tiered storage.  Requirements:

 

1. Create a single share that consists of an SSD and a hard drive (that's easy, that is what unraid does)
2. Disable the default mover (only way I can see to do this is using mover tuning to ignore all files somehow)
3. Move everything that has a last access age (not last modified) of X days from the SSD to the hard drive
4. Keep moving stuff until the volume of the source is below Y% capacity.

 

Only  a month or so into using unraid so there may be a way to do this I just haven't found yet.

 

 

UPDATE (later that same day)

 

The age is based on modified time aka mtime.  So not last accessed time.  Seems easy enough to change or add additionally last accessed time:

https://github.com/hugenbd/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

                    if [ "$CTIMEA" == "yes" ]; then
                        FINDSTR+=" -ctime +$RAGE"

                    else
                        FINDSTR+=" -mtime +$RAGE"
                    fi
                fi

 

 

Not sure if this has changed, but atime was not supported by the shares. (Back in 2020). If you can provide me some evidence that it does support it I can add -atime to the find string.
 

 

Also, the way mover works now.  it creates a file list.  If you want to create a function that I could add to re-order or relist the files I'm willing to add it as an option.

[xra]

1 hour ago, hugenbdd said:

Not sure if this has changed, but atime was not supported by the shares. (Back in 2020). If you can provide me some evidence that it does support it I can add -atime to the find string.
 

 

Also, the way mover works now.  it creates a file list.  If you want to create a function that I could add to re-order or relist the files I'm willing to add it as an option.

 

Ah, here is where my ignorance of unraid (and likely linux file handling) comes in.  It sounds like the trouble will be that atime is never updated making being able to read it useless.

[bukunraid]

On 11/2/2023 at 6:15 PM, akimboflix said:

I'm sure this has been requested before but is it possible to move the oldest/least-accessed files first?

Not sure if we mean the same thing but I'd love it if mover could be configured to move 'x' % of oldest files when share reaches 'x' %. I wish to keep most media on nvme cache for as long as possible and only move some of it to the array when the cache is nearly full.

[alturismo]

9 hours ago, bukunraid said:

I wish to keep most media on nvme cache for as long as possible and only move some of it to the array when the cache is nearly full.

not really, but you can say keep "newer" files on cache (better speaking, move only older file >>)

 

 

like here, when cache reaches 90+ fill rate, i ll move, but only files older 90 days ... Rest is left on cache ...

 

this can be 1 byte to complete cache, depending now on the usage

 

what doesnt work (would be sweet), fillrate 90+ reached, now move oldest files UNTIL (sample) 70 % is reached ...

better speaking, always keeping xy % alive on cache ... in the end, im really fine as is with the age setup only, but you probably looking for another option to specify the "keep on cache rate" which doesnt exist (yet).