[Plugin] Mover Tuning

[RonneBlaze]

18 minutes ago, fk_muck1 said:

You can find the Mover_action_"timestamp".list at /tmp/ca.mover.tuning/

 

ahh ok i was looking for a file with the word concerned in it lol..  but it seems onced i fixed my appdata zfs issue i have no other current errors.

[Reynald]

1 hour ago, RonneBlaze said:

fixed the zfs list -Hpo used cache/appdata issue it now reports a size back

Great, how?

 

1 hour ago, RonneBlaze said:

where is the Mover_action_concerned.list located at? 

It's in /tmp/ca.mover.tuning, but thanks to @fk_muck1 I got what I need to debug, and I'm going to push a new version in a couple of hours.

The 'integer expected', 'unary operator expected', and 'warning: no action for:' are safe to ignore when seen together

The reason is that I optimized memory and cpu thread usage for moving loop, and there can be a "shoot to blank" loop for certain shares without files to move.

[RonneBlaze]

2 hours ago, Reynald said:

Great, how?

 

It's in /tmp/ca.mover.tuning, but thanks to @fk_muck1 I got what I need to debug, and I'm going to push a new version in a couple of hours.

The 'integer expected', 'unary operator expected', and 'warning: no action for:' are safe to ignore when seen together

The reason is that I optimized memory and cpu thread usage for moving loop, and there can be a "shoot to blank" loop for certain shares without files to move.

I have the ZFS Master PLugin also installed and for some reason my appdata was not setup as a dataset,  i clicked the actions butting next to it and it had the option to create dataset, it did what ever it is that it does and now there is a file size next to the folder and when running zfs list -Hpo used cache/appdata it reports a size back also.. 

[Reynald]

6 hours ago, RonneBlaze said:

I have the ZFS Master PLugin also installed and for some reason my appdata was not setup as a dataset,  [...]

 

Interesting, thanks for the report.

 

The case where the storage is zfs but not the share is not handled by the plugin script (yet). I'll add that plus a warning in the log, in a next version.

 

Speaking of next version, I won't be able to develop for a couple of days, as I'm moving right now from Paris to Lyon (France)

[rogales]

Hello,

 

I know there is a lot of posts about this awesome plugin, but it also hard to find answer for my question (tried with search feature).

So...

 

moverDisabled="no"
moverNice="0"
moverIO="-c 2 -n 0"
threshold="95"
age="no"
sizef="no"
sparsnessf="no"
filelistf="no"
filetypesf="no"
parity="yes"
enableTurbo="yes"
logging="no"
force="no"
ignoreHidden="no"
beforeScript="/boot/config/plugins/ca.mover.tuning/dockeroff.sh"
afterScript="/boot/config/plugins/ca.mover.tuning/dockeron.sh"
omovercfg="yes"
movenow="no"
testmode="no"
omoverthresh="100"

 

 

These are my settings for mover tuning.

I would like to stop before moving files few dockers, and after process is complete I want to start them back.

 

Here is my script for that:

 

#!/bin/bash
docker stop binhex-deluge
sleep 1
docker stop binhex-lidarr
sleep 1
docker stop bazarr
sleep 1
docker stop binhex-plexpass
sleep 1
docker stop binhex-radarr
sleep 1
docker stop binhex-sonarr
exit 0

 

and of course same code is for start (stop word has been replaced with start obviously).

 

Why my dockers don't want to stop, or even if I manually stop them, they won't start after moving process?

 

thank You.

Edited August 19 by rogales

[alturismo]

1 hour ago, rogales said:

Why my dockers don't want to stop, or even if I manually stop them, they won't start after moving process?

you can't run scripts off the boot stick ... copy them somewhere ... on your cache as sample ... try from there.

[rogales]

Thank You for Your answer.

To be honest I've had it earlier in my appdata folder (cache drive for appdata files, no the one I want to move files to array) and it did not work either.

Would like to add, that I've created another script in notepad++ in unix system with .sh format:

 

#!/bin/bash
docker stop binhex-deluge
sleep 1
docker stop binhex-lidarr
sleep 1
docker stop bazarr
sleep 1
docker stop binhex-plexpass
sleep 1
docker stop binhex-radarr
sleep 1
docker stop binhex-sonarr

/usr/local/emhttp/plugins/dynamix/scripts/notify -s "Mover Tuning" -d "Mover Tuning has started @ `date +%H:%M:%S`."

 

and it did not work.

any tips?

Edited August 19 by rogales

[Reynald]

Once in appdata, you have to 'chmod +x /mnt/appdata/bothscripts.sh'

[rogales]

Ok, will do that.

How can I now check if it's working now?
Should I run test mode, and see the logs?

[de Techneut]

For some reason since the last update, mover tuning settings is empty:

 

What could be wrong? 

[Soulplayer]

3 hours ago, de Techneut said:

For some reason since the last update, mover tuning settings is empty:

 

What could be wrong? 

Not seeing this issue:

[ramjam824]

14 hours ago, Reynald said:

Speaking of next version, I won't be able to develop for a couple of days, as I'm moving right now from Paris to Lyon (France)

 

Good luck, I hope the move goes smoothly!

[drumtechphoto]

Hello all, thought I would share since the recent plugin update as of today, 8/19/2024:

I thank all that are working on this CA Mover Tuning Plugin, most recent @Reynald.  I noticed a bug when running the Mover manually on a share, in my example "/mnt/user/media".  The command below has a slight typo near the end, where "move" should be "mover".

 

Original/Before:
"find "/mnt/cache/media" -depth | /usr/local/sbin/move -d 1"

 

Correction/After:

"find "/mnt/cache/media" -depth | /usr/local/sbin/mover -d 1"

 

The script as it was (below), gave me the following output and made me wonder, "hey, this plugin uses mover, not move, so let me investigate":

 

mvlogger: *********************************MOVER -SHARE- START*******************************
mvlogger: Mon Aug 19 17:23:47 EDT 2024
mvlogger: Share supplied media
Sharecfg: /boot/config/shares/media.cfg
mvlogger: Cache Pool Name: cache 
mvlogger: Share Path: /mnt/cache/media
mvlogger: Complete Mover Command: find "/mnt/cache/media" -depth | /usr/local/sbin/move -d 1
mvlogger: Mon Aug 19 17:23:47 EDT 2024
mvlogger: ********************************Mover Finished*****************************

_________________________________________________________________________________________________

##Edit:

The bug is located on line # 96 of file "share_mover", snippet below of what line the issue was found on:

 

File & File Location: "/usr/local/emhttp/plugins/ca.mover.tuning/share_mover"

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#81#SHAREPATH="/mnt/$CACHEPOOLNAME/$SHARE"
        mvlogger "Share Path: $SHAREPATH"

        if [ "$SHAREPATH" = "/mnt/cache" ] ; then
                echo "Exiting, can't determine cache name used"
                exit 2
        else
                if [ -d $SHAREPATH ] ; then
                        FINDSTR="find \"$SHAREPATH\" -depth"

                        if [ "$TESTMODE" = "yes" ] ; then
                                mvlogger "Running in Test Mode: $FINDSTR"
                                eval "$FINDSTR"
                        else
                                #Add Mover to find string
#96: Fix ME-->#        FINDSTR+=" | /usr/local/sbin/move -d $LOGLEVEL"
                                mvlogger "Complete Mover Command: $FINDSTR"
                                eval " $FINDSTR"
                        fi
                else
                        mvlogger "Directory does not exist: $SHAREPATH"
                fi
        fi

  fi

rm -f $PIDFILE
$NOW=`date`
mvlogger "$NOW"
mvlogger "********************************Mover Finished*****************************"

Edited August 19 by drumtechphoto
Found config and line # where the bug is located at

[BeefIsForDinner]

11 hours ago, de Techneut said:

For some reason since the last update, mover tuning settings is empty:

 

What could be wrong? 

Same thing for me.

[Raider_M]

 

3 hours ago, BeefIsForDinner said:

Same thing for me.

Same for me. Mover settings are gone.

V7 rc2

Edited August 20 by Raider_M

[alturismo]

15 hours ago, de Techneut said:

For some reason since the last update, mover tuning settings is empty:

 

 

3 hours ago, BeefIsForDinner said:

Same thing for me.

 

5 minutes ago, Raider_M said:

Same for me. Mover settings are gone.

 

here it changed the layout, more strctured in seperate ... may also on your mashines and didnt noticed ? scrolled down ?

 

[Raider_M]

 

 

Sorry for being so zoomed out, but just to show there is no mover tuner section at all in schedule.
The plug in is installed, I also tried a reinstall.

[rogales]

18 hours ago, rogales said:

Ok, will do that.

How can I now check if it's working now?
Should I run test mode, and see the logs?

Guys? Any kind of help? 

[ramjam824]

7 hours ago, rogales said:

Guys? Any kind of help? 

100%, run in test mode, plug-in filters set to YES, and select options you feel align with what you are seeking as output.  Keep tinkering until you see the output match your expectations.  Then, turn test mode off and let it rip!

[taalas]

I moved from the "old" to the "new" Mover Tuning plugin yesterday and after setting it up and running it I immediately got a minor heart attack: it seems that on my system the plugin does not evaluate the mover action that is set in the share settings. I (historically) have some shares that are set to cache primary, array secondary, but with mover direction set from secondary to primary. The mover plugin ignored this and started moving system files, syslogs and vm disks off the cache to the array.

 

I agree that setting the share like this does not make a lot of sense, but am also certain the mover should not ignore that setting

[Faceman]

You may have to re-do those settings if they were configured before installing the plugin.

 

If unsure, use the test mode dry run and check the logs to see if it does what you expect, then run it for real and monitor it for a few runs.

 

 

 

 

[Freender]

Hi @Reynald, thanks for your work!

 

I noticed the following errors in the logs:

 

Aug 21 13:20:29 NAS move: Deleting empty dir /mnt/cache/backup/docker/ab_20240816_040001
Aug 21 13:20:29 NAS move: 0 files remaining from caches to array  0B
Aug 21 13:20:29 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected
Aug 21 13:20:29 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected
Aug 21 13:20:29 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected
Aug 21 13:20:29 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected
Aug 21 13:20:29 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected
Aug 21 13:20:29 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected
Aug 21 13:20:29 NAS move: ------------------------------------------------------------------- Cleaning up -----------------------------------------------------------------
Aug 21 13:20:29 NAS move: Restoring original turbo write mode Auto (read/modify/write)
Aug 21 13:20:29 NAS move: Cleaning lock and stop files
Aug 21 13:20:29 NAS kernel: mdcmd (54): set md_write_method auto
Aug 21 13:20:29 NAS kernel: 
Aug 21 13:20:29 NAS move: ****************************************************************** WE ARE DONE ! ****************************************************************

 

            # Skip filesize = 0 (hardlinks already handled in a previous loop)
            if [ "$NBLINKS" -gt 1 ] && [ "$FILESIZE" = 0 ] ; then
                continue

 

Not sure if it is a data issue or we need to check if $NBLINKS is not null.

I can attach Mover_action_2024-08-21T124119.list  and Filtered_files_2024-08-21T124119.list if required. 

Edited August 21 by Freender
updated logs

[Faceman]

I've been seeing that as well.

 

I've also noticed it seemingly ignoring my fill settings, moving files when the pool is only at 20% full, when I have the move threshold set at 70%

[Yusseiin]

I have installed the "Reynalds" one but it seems not working.
In the log i can see that it is saying
 

0:15:16.521 Primary storage: cache - size: 692GiB - used: 85 % (611GiB)
10:15:16.522 Secondary storage: user0
10:15:16.531 Share Information: Name: downloads - Path: /mnt/cache/downloads
10:15:16.537 Moving threshold: 75% (519GiB) ; Freeing threshold: 50% (346GiB)
10:15:16.539 Mover action: cache->user0 (cache:yes). Pool is above moving threshold percentage:  85% >= 75%.
10:15:16.540 => Will smart move old files from cache to user0. Nothing will be moved from user0 to cache
10:15:16.553 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-08-22T101515.list for downloads

In the "Mover_action" all the files has the

move from primary|/mnt/cache|/mnt/user0

but in the end no file where moved.
the settings are basic ones


Can someone guide me to throubleshooting this?

[de Techneut]

How can I read the mover log file?