[Plugin] Mover Tuning


Recommended Posts

4 hours ago, DanielPT said:

HI all

EDIT: I can see many outhers have thees problems. 

Can i provide something?

I been using this for some time and all working graat.

 

I have followed the trash guide:
https://trash-guides.info/Downloaders/qBittorrent/Tips/How-to-run-the-unRaid-mover-for-qBittorrent/

 

But today i saw that the mover has moved my files but no the folders? 

Can i make a rollback?

image.thumb.png.1c1c135bc560c2c7fa179a6936c116a4.png

My settings:

image.thumb.png.55695cb8e6c734a429963cea9b137500.png

I'll have a fix in the next few days addressing.

 

1.) The echo statements in the hardlink section.

2.) Moving empty folders.

3.) Fix a "testing" if statement in the hardlink section of code.

  • Like 1
Link to comment
12 hours ago, Andiroo2 said:

 

More on this one...it looks like Mover ignored the Tuning settings for file ages.  It moved everything available, and not just the files older than 30 days.  I'm not over the "move all cache-yes" files threshold either.  If you are interested, the logs I DM'd show the mover using a 30-day cut-off, but many more files were moved. 

I'm not able to reproduce.  I'm willing to do a google meeting sometime next week if you want me to look at it.

Link to comment
18 hours ago, hugenbdd said:

I'm not able to reproduce.  I'm willing to do a google meeting sometime next week if you want me to look at it.

 

Not urgent.  I will watch for this next time and if it's a recurring problem then I'll reach out.  Thanks for the offer.

Link to comment

Hey all,

 

I've found that the mover script is not working correctly on a schedule.

It hasn't moved any files for over 1.5 months as far as I can tell.

 

However, when I manually ran it today, it moved all the files no problem.

 

Screenshot of my settings below - can someone please help me understand if I've done something wrong?

 

 

chrome_iLi1SIFjx5.png

chrome_1IIUCed3vp.png

Link to comment
21 hours ago, Deadringers said:

Hey all,

 

I've found that the mover script is not working correctly on a schedule.

It hasn't moved any files for over 1.5 months as far as I can tell.

 

However, when I manually ran it today, it moved all the files no problem.

 

Screenshot of my settings below - can someone please help me understand if I've done something wrong?

I would need to see Logs.

 

Link to comment
On 5/28/2023 at 7:31 PM, hugenbdd said:

I'll have a fix in the next few days addressing.

 

1.) The echo statements in the hardlink section.

2.) Moving empty folders.

3.) Fix a "testing" if statement in the hardlink section of code.

Looking for a few to test the changes above before I release.  Just replace the age_mover file with the one attached.

 

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

age_mover

Link to comment

Feature request (is this the right place to ask?):  A option to invoke the mover once I hit a certain threshold (95% in my case) and move the OLDEST files from the cache to the array until I hit a free space threshold on the cache (say 80% free) and then stop moving.

 

I like to keep as many files on my cache as possible for fast access and to keep my array spun down whenever I can. I do this today with the mover tuning plugin by only running the mover once I hit 95% cache utilization and only moving flies over a certain age, but with the above feature I would be able to keep the cache exactly where I want it at all times. 

Edited by Andiroo2
Spelling
  • Like 1
Link to comment
On 5/12/2023 at 1:41 AM, a632079 said:

Hello, I am using Unraid v6.12.0-rc5 with ZFS based cache pool. The `Pool Pct Used` value always be `1%` because this

command just fetch `/mnt/cache`'s percentage usage.

The `df -h` messae:

```

cache           142G  128K  142G   1% /mnt/cache
cache/appdata   142G  165M  142G   1% /mnt/cache/appdata
cache/system    147G  5.9G  142G   4% /mnt/cache/system
cache/domains   165G   24G  142G  15% /mnt/cache/domains

```

So, we can't get zpool usage just by df.

 

Maybe we can try zpool command to get usage:

```

zpool get -o value capacity cache | tail -n1

```

it returns

```

86%

```

 

 

I have temporary changed this line in mover_age:

```

POOLPCTUSED=`df --output=pcent /mnt/$CACHEPOOLNAME | tail -n 1 | tr -d '%'`

```

to

```

POOLPCTUSED=`zpool get -o value capacity cache | tail -n 1 | tr -d '%'`

```

 

Maybe we should determine which command should be used by judging which fs is used.

 

@hugenbdd cc.

 

Hello, since the last time I shared the limitations of ZFS support, I have made further modifications to the plugin and added support for file system detection. I believe that Unraid 6.12-rc6 is now stable enough to ensure that the content of the ZFS capacity query will not be altered (even if it is altered, it will not affect the functionality of the zpool).

 

Here is the function I use to determine the file system type using the findmnt command.

If there is a conflict between CACHEPOOLNAME and the name of the mount point, please let me know! I will fix this issue.

  

function is_zfs_mountpoint_fstype()
{
        local result
        result=`findmnt $1 -o fstype | tail -n 1`
        [ "$result" = "zfs" ]
        return
}

`is_zfs_mountpoint_fstype "cache"`

echo $? # it should be return 0 or 1

 

I added this function to the file /usr/local/emhttp/plugins/ca.mover.tuning/age_mover.

 

image.thumb.png.e0afce489e49c7e57cc0d8c827d41b93.png

 

In addition, I made modifications in two places regarding the usage of POOLPCTUSED.

 

		POOLPATH=/mnt/$(basename "$POOL" .cfg)
                if is_zfs_mountpoint_fstype "$POOLPATH" ; then
                        mvlogger "$POOLPATH is zfs."
                        POOLPCTUSEDA=`zpool get -o value capacity $(basename "$POOL" .cfg) | tail -n 1 | tr -d '%'`
                else
                        POOLPCTUSEDA=`df --output=pcent $POOLPATH | tail -n 1 | tr -d '%'`
                fi

 

image.thumb.png.4c29af54e4aaf9a498fbe7a8ea9dd543.png

 

		if is_zfs_mountpoint_fstype "/mnt/$CACHEPOOLNAME" ; then
                        mvlogger "$CACHEPOOLNAME is zfs."
                        POOLPCTUSED=`zpool get -o value capacity $CACHEPOOLNAME | tail -n 1 | tr -d '%'`
                else
                        POOLPCTUSED=`df --output=pcent /mnt/$CACHEPOOLNAME | tail -n 1 | tr -d '%'`
                fi

 

 

image.thumb.png.54f762d3c3622bad8f0c715db351ad99.png

 

By modifying the above content, mover tuning can now correctly identify the cache partition usage of ZFS and provide compatibility features for Btrfs and XFS. The following is the log content using this configuration.

 

mvlogger: *********************************MOVER START*******************************
mvlogger: Age supplied 0
mvlogger: Size supplied
mvlogger: Sparness supplied
mvlogger: No Skipfiles Argument Supplied
mvlogger: No Skipfiles Argument Supplied
mvlogger: BEFORESCRIPT: /mnt/user/appdata/Scripts/on_mover_start.sh
mvlogger: AFTERSCRIPT: /mnt/user/appdata/Scripts/on_mover_stop.sh
mvlogger: CTIME Argument: no
mvlogger: CACHE THRESH to Move all Cache-Yes shares to array: 85
mvlogger: No Test Mode Argument Supplied
mvlogger: No Ignore Hidden Files Argument Supplied
mvlogger: Before script will be executed: /mnt/user/appdata/Scripts/on_mover_start.sh
mvlogger: CACHETHRESH: 85
mvlogger: Checking to see if we should all files from share.
mvlogger: Move All Cache Threshold: 85
mvlogger: /mnt/cache is zfs.
mvlogger: /mnt/cache PCT USED: 82
mvlogger: Done checking Move All from Share
mvlogger: Share Name Only: NAS
mvlogger: Cache Pool Name: cache
mvlogger: cache Threshold Pct:
mvlogger: OVERALL Threshold: 35
mvlogger: Share Path: /mnt/cache/NAS
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 82 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 35
mvlogger: Threshold Used: 35
mvlogger: Adding Size
mvlogger: Size 100
mvlogger: Skipfiletypes string: find "/mnt/cache/NAS" -depth -size +100M
mvlogger: Share Name Only: isos
mvlogger: Cache Pool Name: cache
mvlogger: cache Threshold Pct:
mvlogger: OVERALL Threshold: 35
mvlogger: Share Path: /mnt/cache/isos
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 82 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 35
mvlogger: Threshold Used: 35
mvlogger: Adding Size
mvlogger: Size 100
mvlogger: Skipfiletypes string: find "/mnt/cache/isos" -depth -size +100M
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------

 

Edited by a632079
  • Like 1
Link to comment

 

By the way, can we add an additional option to the "Script to run before mover (No checks, always runs)" scripts, or simply introduce a new setting called "Script to run before mover (if filters are not triggered)"? I have a PT (Private Tracker) client running on my server, and the mover regularly checks for mover conditions every day. Since I have allocated a large cache pool, in most cases, the mover is not triggered. However, when the script runs, it pauses my BT (BitTorrent) software, causing me to lose out on continuous seeding rewards. This has been a bit of a concern for me.

  • Like 1
Link to comment
On 6/3/2023 at 12:31 PM, Andiroo2 said:

Feature request (is this the right place to ask?):  A option to invoke the mover once I hit a certain threshold (95% in my case) and move the OLDEST files from the cache to the array until I hit a free space threshold on the cache (say 80% free) and then stop moving.

 

I like to keep as many files on my cache as possible for fast access and to keep my array spun down whenever I can. I do this today with the mover tuning plugin by only running the mover once I hit 95% cache utilization and only moving flies over a certain age, but with the above feature I would be able to keep the cache exactly where I want it at all times. 

Same mate. This is my dream feature:

If threshold is greater than %, move oldest files until threshold is under %.

  • Like 1
Link to comment
6 hours ago, NGHTCRWLR said:

Same mate. This is my dream feature:

If threshold is greater than %, move oldest files until threshold is under %.

If I wrote this, it would be an independent script and not part of the plug-in.  But this could be done as a cron entry.

Link to comment
On 6/7/2023 at 8:32 AM, hugenbdd said:

If I wrote this, it would be an independent script and not part of the plug-in.  But this could be done as a cron entry.

 

+1. This is also my preferred scenario.  keep the most recent files cached, while at 85% full or whatever. 

  • Like 1
Link to comment
10 minutes ago, dopeytree said:

I would love to be able to keep ONLY my recent DOWNLOADS (data share) cached while at 75 / 85%.

 

Would still like other normal new share folders to get moved regularly to the array.

I'm starting to work on a seperate script for this.  Once I release it, I'll post a new forum thread separate from Mover Tuning.  At first it will just move based on age of files (oldest first).  I would expect later it will include other filters from Mover tuning.

  • Like 2
Link to comment

It sounds like the incoming 6.12 only has primary & secondary storage options for mover so thanks this will be much appreciated. 

 

If it makes it any easier I run downloads in a separate cache pool. 

Everything else happens on the normal cache pool.

 

To make dates work for newly copied over folders on normal cache (non downloads) I guess I'd want those to use dates that refer to when the item was copied rather than its original creation date which happened on another PC.

 

Anyway

e.g

 

'when share on cache/s (primary) reaches 75% capacity' = move 'anything over 90days old' to array (secondary). { but never move: APPDATA, DOMAINS, SYSTEM } 

Edited by dopeytree
Link to comment

Getting a strange error:

Jun 13 15:02:32 NAS root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 214: /usr/local/sbin/move: No such file or directory

 

Which makes sense because move is in bin, not sbin.

which move
/usr/local/bin/move

 

It's a simple fix but I wonder if it indicates something more serious if I'm the only one seeing it.

age_mover was last modified May 23:

ls -l /usr/local/emhttp/plugins/ca.mover.tuning/age_mover
-rwxr-xr-x 1 root root 23309 May 23 09:01 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover*

 

My plugins are updated/running 6.12.0-rc8

 

EDIT: I see the age_mover in this post also references sbin. Huh.

EDIT 2: It appear all scripts in /usr/local/emhttp/plugins/ca.mover.tuning reference sbin/move, which doesn't exist. Maybe the recently-released RC8 moved it from sbin to bin.

Edited by CS01-HS
Link to comment
19 minutes ago, CS01-HS said:

Getting a strange error:

Jun 13 15:02:32 NAS root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 214: /usr/local/sbin/move: No such file or directory

 

Which makes sense because move is in bin, not sbin.

which move
/usr/local/bin/move

 

It's a simple fix but I wonder if it indicates something more serious if I'm the only one seeing it.

age_mover was last modified May 23:

ls -l /usr/local/emhttp/plugins/ca.mover.tuning/age_mover
-rwxr-xr-x 1 root root 23309 May 23 09:01 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover*

 

My plugins are updated/running 6.12.0-rc8

 

EDIT: I see the age_mover in this post also references sbin. Huh.

 

 

CS01-HS i'm also seeing the same error.

I'm running 6.12.0-rc8 with the plugins updated. I've also tried the recent test file.
 

Jun 13 20:45:41 Server move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 214: /usr/local/sbin/move: No such file or directory
Jun 13 20:45:41 Server move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 214: echo: write error: Broken pipe

 

Link to comment
3 minutes ago, 2simguy said:

 

CS01-HS i'm also seeing the same error.

I'm running 6.12.0-rc8 with the plugins updated. I've also tried the recent test file.
 

Jun 13 20:45:41 Server move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 214: /usr/local/sbin/move: No such file or directory
Jun 13 20:45:41 Server move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 214: echo: write error: Broken pipe

 

 

RC8 seems to have relocated the move binary to /usr/local/bin from /usr/local/sbin

  • Like 1
  • Upvote 1
Link to comment
Just now, CS01-HS said:

 

I've fixed it (until the next reboot) with a symbolic link:

ln -s /usr/local/bin/move /usr/local/sbin/move

 

Seems to be running properly but hasn't finished yet.

Yes looks like they changed locations of it for that release.
 

  • Thanks 1
Link to comment

getting a ton of these lately

 

move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/[etc]

 

why?

 

When I track down the problematic directories on the cache drive I find that indeed the files do not exist.  Just the empty folders they were living in.

Edited by DontWorryScro
further clarification
Link to comment

New Release (And I already found a bug which will be fixed tomorrow)

 

###2023.06.15
- Delete empty directories ***(Bug when no empty directories found)
- Add Softlink for binary mover file change if on 6.12 RC8 in install script. (CS01-HS Found 6/13)
- Add function for compatibility with RC8+ with zfs % full (a632079 - 6/3)

Link to comment

Hi there! I've got an issue on 6.12 RC8 and 6.12 final with the plugin.

When invoking the mover the only thing it does is this : 

2023-06-16 10:25:27	Jun 16 10:25:27 Vili emhttpd: shcmd (163): /usr/local/sbin/mover |& logger -t move &
2023-06-16 10:25:27	Jun 16 10:25:27 Vili root: ionice -c 2 -n 7 nice -n 5 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 /mnt/user/tools/mover-ignore.txt '' /mnt/user/tools/mover-pre-script.sh /mnt/user/tools/mover-post-script.sh no 90 '' ''
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: Log Level: 1
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: mover: started
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: find: '/tmp/Mover/*': No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: find: '/tmp/Mover/*': No such file or directory
2023-06-16 10:27:05	Jun 16 10:27:05 Vili move: cat: /tmp/Mover/Hard_Mover_Tuning_2023-06-16T102527.list: No such file or directory
2023-06-16 10:28:48	Jun 16 10:28:48 Vili move: mover: finished

 

And nothing gets move. Is that the bug you're talking about in your previous post?

 

Thanks!

Link to comment
4 hours ago, foux said:

Hi there! I've got an issue on 6.12 RC8 and 6.12 final with the plugin.

When invoking the mover the only thing it does is this : 

2023-06-16 10:25:27	Jun 16 10:25:27 Vili emhttpd: shcmd (163): /usr/local/sbin/mover |& logger -t move &
2023-06-16 10:25:27	Jun 16 10:25:27 Vili root: ionice -c 2 -n 7 nice -n 5 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 /mnt/user/tools/mover-ignore.txt '' /mnt/user/tools/mover-pre-script.sh /mnt/user/tools/mover-post-script.sh no 90 '' ''
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: Log Level: 1
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T102527.log: No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: mover: started
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: find: '/tmp/Mover/*': No such file or directory
2023-06-16 10:25:27	Jun 16 10:25:27 Vili move: find: '/tmp/Mover/*': No such file or directory
2023-06-16 10:27:05	Jun 16 10:27:05 Vili move: cat: /tmp/Mover/Hard_Mover_Tuning_2023-06-16T102527.list: No such file or directory
2023-06-16 10:28:48	Jun 16 10:28:48 Vili move: mover: finished

 

And nothing gets move. Is that the bug you're talking about in your previous post?

 

Thanks!

No, can you verify the folder was created?

/tmp/Mover

 

from the console

ls -ltr /tmp/Mover

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.