[Plugin] Mover Tuning


Recommended Posts

16 minutes ago, diarnu said:

Agreed, great feature.

 

Just happened to be adding a new share today, and noticed that the new feature causes some warning messages on the add share page b/c the new share being created does not exist yet (lol):

 

per_share_mover_oops.thumb.png.4796472da3f3501d539a9f7eb142fc20.png

Good catch! I did not test this scenario.

Will work on this the next day or two and put a release out to fix.

Link to comment
On 11/26/2021 at 8:28 PM, diarnu said:

Agreed, great feature.

 

Just happened to be adding a new share today, and noticed that the new feature causes some warning messages on the add share page b/c the new share being created does not exist yet (lol):

 

per_share_mover_oops.thumb.png.4796472da3f3501d539a9f7eb142fc20.png

11-28-2021

- Fixed Share Mover GUI error when creating new share.

 

This should now work.

Link to comment
  • 3 weeks later...
On 7/8/2021 at 9:56 PM, IMI4tth3w said:

 

Figured it out.

 

My mover-ignore.txt had the wrong directories in it..

 

old

/mnt/user/data/downloads/

 

fixed

/mnt/cache/data/downloads/

 

replaced user with cache...

 

dumb mistake. But i did find out that adding the * to the end of the directory is not needed. grep is effectively looking at the strings of the file name looking for matches. obviously "/mnt/user" wasn't going to match to anything that is "/mnt/cache"

 

thank you again for the help!

 

I attempted the above but I can't seem to get the skip file list to skip files inside a directory. Maybe it's not looking recursively at sub-directories? Any suggestions?

 

Log snippet

Dec 16 13:10:57 Tower root: mvlogger: Adding Skip File List
Dec 16 13:10:57 Tower root: mvlogger: Skip File List Path: /mnt/user/appdata/mover_ignore.txt
Dec 16 13:10:57 Tower root: mvlogger: Skipfiletypes string: find "/mnt/downloads/Media" -depth -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt'
Dec 16 13:10:57 Tower root: mvlogger: find "/mnt/downloads/Media" -depth -type f -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt' | grep -v -e '!qB' -e '.!qB' | du -shc --files0-from - | tail -n1
Dec 16 13:10:57 Tower root: mvlogger: Running in Test Mode: find "/mnt/downloads/Media" -depth -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt' | grep -v -e '!qB' -e '.!qB'
Dec 16 13:10:57 Tower root: /mnt/downloads/Media/Downloads/sonarr/[SubsPlease] One Piece - 1003 (1080p) [8637D29D]/[SubsPlease] One Piece - 1003 (1080p) [8637D29D].mkv
Dec 16 13:10:57 Tower root: /mnt/downloads/Media/Downloads/sonarr/[SubsPlease] One Piece - 1003 (1080p) [8637D29D]

 

mover_ignore.txt attempt

/mnt/downloads/Media/Downloads/
/mnt/downloads/Media/Downloads/*

 

Link to comment
1 hour ago, captaincu said:

 

I attempted the above but I can't seem to get the skip file list to skip files inside a directory. Maybe it's not looking recursively at sub-directories? Any suggestions?

 

Log snippet

Dec 16 13:10:57 Tower root: mvlogger: Adding Skip File List
Dec 16 13:10:57 Tower root: mvlogger: Skip File List Path: /mnt/user/appdata/mover_ignore.txt
Dec 16 13:10:57 Tower root: mvlogger: Skipfiletypes string: find "/mnt/downloads/Media" -depth -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt'
Dec 16 13:10:57 Tower root: mvlogger: find "/mnt/downloads/Media" -depth -type f -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt' | grep -v -e '!qB' -e '.!qB' | du -shc --files0-from - | tail -n1
Dec 16 13:10:57 Tower root: mvlogger: Running in Test Mode: find "/mnt/downloads/Media" -depth -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt' | grep -v -e '!qB' -e '.!qB'
Dec 16 13:10:57 Tower root: /mnt/downloads/Media/Downloads/sonarr/[SubsPlease] One Piece - 1003 (1080p) [8637D29D]/[SubsPlease] One Piece - 1003 (1080p) [8637D29D].mkv
Dec 16 13:10:57 Tower root: /mnt/downloads/Media/Downloads/sonarr/[SubsPlease] One Piece - 1003 (1080p) [8637D29D]

 

mover_ignore.txt attempt

/mnt/downloads/Media/Downloads/
/mnt/downloads/Media/Downloads/*

 

 

I would suggest testing in the console.  Start with..

find "/mnt/downloads/Media" -depth -type f -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt'

 

Maybe, add 1 specific easy to name file. (No spaces) to see if it excludes it in the results.  Once you get one file working, shortening the path in the exclude file.

 

I'll also test tonight with my cache paths/files.

Link to comment
17 hours ago, hugenbdd said:

 

I would suggest testing in the console.  Start with..

find "/mnt/downloads/Media" -depth -type f -mtime +0 | grep -vFf '/mnt/user/appdata/mover_ignore.txt'

 

Maybe, add 1 specific easy to name file. (No spaces) to see if it excludes it in the results.  Once you get one file working, shortening the path in the exclude file.

 

I'll also test tonight with my cache paths/files.

 

SOLUTION AT THE END OF MY POST

 

I did my own little testing and it seems like only the last line in mover_ignore.txt gets evaluated. This is for directories at least in my use case. Not sure what happens if you sprinkle files into mover_ignore.txt.

 

My test setup:

test_folder/
├── test/
│   ├── test.txt
├── test2/
│   ├── test2.txt
├── test3/
│   ├── test3.txt

 

Test command

find "/mnt/downloads/Media/Downloads/test_folder" -depth -type f | grep -vFf '/mnt/user/appdata/mover_ignore.txt'

 

Single top level directory in mover_ignore.txt

mover_ignore.txt

/mnt/downloads/Media/Downloads/test_folder/

No files found as expected

 

Multiple directories in mover_ignore.txt

When mover_ignore.txt has multiple directories including parent directory test_folder/ and test3/ is on the last line

/mnt/downloads/Media/Downloads/test_folder/
/mnt/downloads/Media/Downloads/test_folder/test/
/mnt/downloads/Media/Downloads/test_folder/test2/
/mnt/downloads/Media/Downloads/test_folder/test3/

Files found in output

/mnt/downloads/Media/Downloads/test_folder/test/test.txt
/mnt/downloads/Media/Downloads/test_folder/test2/test2.txt

 

EDIT: Figured it out using this page. Having my mover_ignore.txt save with CRLF broke the patterns for preceding lines so only my last pattern in the file would work. Solution was to save the file with LF as the end of line sequence.

Edited by captaincu
found solution
  • Like 2
Link to comment

hello guys.

i love this plug-in, but from now on I moved on to two cache pools.

How does this plug-in handle that scenario?

As I read the rpevious post, the settings are the same for both cache pools, but what about the trigger conditions?

Is it checked for the different pools at a different time?

I have a 2TB and a 500G pools with different habits.

If I set a 75% trigger, what happens?

is the 2TB and 500GB condition evaluated different? and mover only starts for the triggered cache?

or it checks the sum of the 2 caches reaches 75%, which might mean, that my 500GB is full, and the 2TB is only 50%, it would mean this script is never triggered?

 

Thanks for your help!

Link to comment
On 12/18/2021 at 12:36 PM, LSL1337 said:

hello guys.

i love this plug-in, but from now on I moved on to two cache pools.

How does this plug-in handle that scenario?

As I read the rpevious post, the settings are the same for both cache pools, but what about the trigger conditions?

Is it checked for the different pools at a different time?

I have a 2TB and a 500G pools with different habits.

If I set a 75% trigger, what happens?

is the 2TB and 500GB condition evaluated different? and mover only starts for the triggered cache?

or it checks the sum of the 2 caches reaches 75%, which might mean, that my 500GB is full, and the 2TB is only 50%, it would mean this script is never triggered?

 

Thanks for your help!

For the time being it's just one level except for disk used %.  Which can be set manually by editing the config file. (see 1-08-21 release notes)

- age_mover bash script now check for global threshold percent or the a manual entry in the ca.mover.tuning.cfg file. (example: /boot/config/plugins/ca.mover.tuning cachetv="65")

- "cachetv" being the name of the pool.

- if using single digits leading zero required. (i.e. cachetv="01" for 1 percent)

 

The way the plug-in script works, it loops through each share and checks against that shares cache settings/pool.  If the setting is reached/surpassed it will run the find script on that share and send the files to the binary mover.  The percentage used is calculated for each pool when each share is looped through it.

Link to comment

Is there a way to force the mover to move only one file at a time?  It seems to move more than one file simultaneously, often to different physical disks.  When doing this, write speeds tank from 200MB/sec+ to maybe 20MB/sec (due to the simultaneous reads needed for parity calculations).  

Edited by dyno
Link to comment

at the moment, no.

The plug-in follows the same type of find command that the normal mover script uses.  i.e. find <cache path> | /usr/local/sbin/move

 

I do plan to update the plug-in to work off a file list, instead of the find command.  i.e. loop through the file list, and possibly add a user selectable "wait" between files.

 

You could try to set the two following items.

Priority for mover process: <Low, or Very Low>

Priority for disk I/O: <Low, or idle>

  • Thanks 1
Link to comment

I have been using Mover Tuning with days old and percent enabled for about 6 months now. When setting this in mover tuning the expected result would be when cache reaches percent used it will trigger a move only files older than a specified days old. Instead what happens is mover will trigger at percent used and move all files regardless of age.

Edited by ItsRaining
Link to comment
9 minutes ago, ItsRaining said:

I have been using Mover Tuning with days old and percent enabled for about 6 months now. When setting this in mover tuning the expected result would be when cache reaches percent used it will trigger a move only files older than a specified days old. Instead what happens is mover will trigger at percent used and move all files regardless of age.

Please enable logging.

 

You may also want to play with this in test mode.

 

Please post a few of the log entries with the "find" command also.

 

One thing to note, is this is not event based, but time based.  The trigger is the "scheduled time" you have set.  at each time, it just checks the percentage of the cache.  Then create the find command to send to the binary mover.  That find command should have a time component to it if you select "days old".

Link to comment
  • 2 weeks later...
On 10/28/2021 at 7:59 AM, i1mran92 said:

thank you very sir. i was able to make the changes. it detects the correct folder now.

 

also, if you dont mind, do you know how i can use the user script so that i can close qbitorrent before the mover start and open qbitorrent after the mover ends. i tried it, and it doesnt seem to work. any advice would be awesome.

 

i added this into the mover tuning before and after mover start script

/boot/config/plugins/user.scripts/scripts/stopping_qbittorrent

/boot/config/plugins/user.scripts/scripts/starting_qbitorrent

 

some screenshot of the settings.

 

unraid 1.PNG

unraid 2.PNG

unraid 3.PNG

late responding to this, but yes, I was able to rename the file in /boot from Media to media with no issues. I backed it up to be safe and it took a reboot to apply, but it worked.

Link to comment
53 minutes ago, Orishas said:

Thanks for the advice, i thought that would only work with files.

 

let me know if it works for you, i followed this rule in my .txt file for excluding 3 folders

 

/mnt/cache/Media/BILDER/
/mnt/cache/Media/BILDER/*
/mnt/cache/Media/EBOOKS/
/mnt/cache/Media/EBOOKS/*
/mnt/cache/Media/RECORD/
/mnt/cache/Media/RECORD/*

 

the mover didnt had to run yet as i also use the fillrate rule ... i ll see what happens ;)

  • Thanks 1
Link to comment

One question...

 

I don't get it, but why does the Input-field for file types is greyed out if I choose Yes for "ignore file types"?!

 

Yes:

image.png.75728e42a809ae0744ef112fc3db7a1f.png

 

No:

image.png.3c0a87ddd8f0a7c20f07436f63d6fa2a.png

 

Same happens when I choose Yes for "Move files off cache based on age?"

 

image.png.107275b91d0ab3a2b88cead1b8fbbba2.png

 

image.png.ab2c6c718380871fdae01041cc0626ed.png

 

 

Its all the wrong way for anything which can be set to Yes or No.

 

What's the point I don't get?!

Link to comment
8 minutes ago, enJOyIT said:

One question...

 

I don't get it, but why does the Input-field for file types is greyed out if I choose Yes for "ignore file types"?!

 

Yes:

image.png.75728e42a809ae0744ef112fc3db7a1f.png

 

No:

image.png.3c0a87ddd8f0a7c20f07436f63d6fa2a.png

 

Same happens when I choose Yes for "Move files off cache based on age?"

 

image.png.107275b91d0ab3a2b88cead1b8fbbba2.png

 

image.png.ab2c6c718380871fdae01041cc0626ed.png

 

 

Its all the wrong way for anything which can be set to Yes or No.

 

What's the point I don't get?!

Is this your first time opening the schedules page?  i.e. nothing saved yet?

 

There is a bug with some of the javascript I can't track down that may do this sometimes.  I would say, make a small change somewhere, apply, and reload the page.  Then see what happen.

  • Thanks 1
Link to comment
3 minutes ago, hugenbdd said:

Is this your first time opening the schedules page?  i.e. nothing saved yet?

 

There is a bug with some of the javascript I can't track down that may do this sometimes.  I would say, make a small change somewhere, apply, and reload the page.  Then see what happen.

 

Ah, thank you for that advice! It is working now!

 

I'm new to unraid and thought there is a sort of reverse unraid logic :D

Edited by enJOyIT
  • Like 1
Link to comment

Newbie here: I'm trying to add a script before and after Mover runs. The goal is to stop the qbittorrent docker while Mover does its thing and restart it after it finishes.
 

The script running BEFORE is:

 

#!/bin/bash
docker stop --time=60 qbittorrent

 

I have uploaded the script to the path: /mnt/user/appdata/script-mover-before/script.sh

 

The script running AFTER is:

 

#!/bin/bash
docker start qbittorrent

 

I have uploaded the script to the path: /mnt/user/appdata/script-mover-after/script.sh

 

I don't know if the script file should go in a particular path or in a different format/extension. At the moment, I can't get docker to stop. Any help would be much appreciated!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.