[Plugin] Mover Tuning

Yusseiin · August 22

20 minutes ago, de Techneut said:

How can I read the mover log file?

You can go with an FTP client or use the terminal with “nano” cmd.

the default folder is in /tmp/ca.mover.tuning

taalas · August 22

On 8/21/2024 at 4:18 AM, Faceman said:

You may have to re-do those settings if they were configured before installing the plugin.

If unsure, use the test mode dry run and check the logs to see if it does what you expect, then run it for real and monitor it for a few runs.

Not exactly sure what you mean. Aren't those share settings regarding the default mover? Why should I have to re-do those settings (and how would I have known)?

I also did a couple of test runs and checked the logs before running it, the test runs and the productive run behaved completely different though...

drumtechphoto · August 22

I have noticed as well that the settings I have applied to CA Mover Tuning are not being used by the plugin:

Only move if above this threshold of used Primary (cache) space: 70%

Free down/prime up to this level of used Primary (cache) space: 55%

I got home from work and noticed my Unraid Server's Cache share is at 100% Full with only 197KB free...

We understand that @Reynald is busy moving, and hope that the moving process is going well.

Hopefully someone else can figure out what is wrong in the code to give some idea of how to fix what is going on/wrong.

Faceman · August 22

3 hours ago, taalas said:

Not exactly sure what you mean. Aren't those share settings regarding the default mover? Why should I have to re-do those settings (and how would I have known)?

Was just a guess, something to test

Gareth321 · August 26

Mover schedule is no longer running for me. I use a 70% full threshold, but the cache fills up now every time. It looks like many others are experiencing the same. I will uninstall this plugin for now.

Faceman · August 26

I just did a test run with the settings adjusted to 5% below my current actual fill level and it worked.

So something seems screwy if some people have it working and others dont, we will have to wait for @Reynald to return.

stelks · August 27

It is also broken for me on unraid 7 beta 2. Uninstalling fixes the issue. Here are the logs:

Aug 27 11:36:18 LilNASX move: mover: started
Aug 27 11:37:43 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 206: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:43 LilNASX move: cat: write error: Broken pipe
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: echo: write error: Broken pipe
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: echo: write error: Broken pipe
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: mover: finished

Faceman · August 27

I think it should be calling /usr/local/sbin/mover not /usr/local/sbin/move

I assume you could probably fix it by dropping a good copy of the mover script into the sbin folder

There is also mover.old which has the original mover script backed up.

My "mover" script looks like this, and once that fires it calls /usr/local/emhttp/plugins/ca.mover.tuning/age_mover script that does the real job.

#!/bin/bash
PPPID=`ps h -o ppid= $PPID 2>/dev/null`
P_COMMAND=`ps h -o %c $PPPID 2>/dev/null`
/usr/local/emhttp/plugins/ca.mover.tuning/mover.php $P_COMMAND $1

solidno8 · August 27

Could you add in the option for the script to honor hidden folders in the root directory like the old mover script by setting the option below?

shopt -s dotglob

sergiu.topan · August 29

I am trying to run some script before and after the mover with this plugin and with no luck.

I created using User Scripts plugin 2 scripts:

/boot/config/plugins/user.scripts/scripts/Mover_before/script

/boot/config/plugins/user.scripts/scripts/Mover_after/script

In the logs is get the error

Before script file sh /boot/config/plugins/user.scripts/scripts/Mover_before/script does not exist or is not executable

Where should I put the script to be executable for the plugin

In other scripts I can execute the above ones if I add sh before the path.

alturismo · August 29

9 hours ago, sergiu.topan said:

Where should I put the script to be executable for the plugin

as Unraid is blocking scripts from the boot drive ... anywhere on the system besides /boot and +x

SimonF · August 29

9 hours ago, sergiu.topan said:

I am trying to run some script before and after the mover with this plugin and with no luck.

I created using User Scripts plugin 2 scripts:

/boot/config/plugins/user.scripts/scripts/Mover_before/script

/boot/config/plugins/user.scripts/scripts/Mover_after/script

In the logs is get the error

Before script file sh /boot/config/plugins/user.scripts/scripts/Mover_before/script does not exist or is not executable

Where should I put the script to be executable for the plugin

In other scripts I can execute the above ones if I add sh before the path.

As stated file system does not support executables so need s to be somewhere. But I thought you could specify a user script within the plugin. Did you add via user scripts,

sergiu.topan · August 30

12 hours ago, SimonF said:

As stated file system does not support executables so need s to be somewhere. But I thought you could specify a user script within the plugin. Did you add via user scripts,

The scripts mentioned by me are created using user script plugin and that is the location that is show in the app where the script is located.

There is another location where I can find the scrips created by user scripts plugin?

ZGief · September 1

On 8/29/2024 at 11:26 PM, sergiu.topan said:

There is another location where I can find the scrips created by user scripts plugin?

It looks like the scripts stored by User Scripts aren't ready to be run outside of the plugin. The files aren't executable (see permissions col) and the script itself doesn't have that first line that "bash" line to instruct the os how to run it.

What I did was put my before and after run scripts in:

/mnt/ssd_cache_docker/appdata/qbittorrent/qBittorrent/scripts/pause-qbit.sh

/mnt/ssd_cache_docker/appdata/qbittorrent/qBittorrent/scripts/resume-qbit.sh

I made sure these two files were executable (chmod command) and that the file had #!/bin/bash on the first line.

If you're hoping to only have one copy of the script that can be run by mover tuner and in the user scripts UI, i'm not sure if that is possible.

alturismo · September 1

On 8/30/2024 at 8:26 AM, sergiu.topan said:

There is another location where I can find the scrips created by user scripts plugin?

yes there is, as of the boot restriction ... they are copied to /tmp

sample, my script called "Encode"

root@AlsServerII:~# ls -la /tmp/user.scripts/tmpScripts/Encode/
total 16
drwxr-xr-x  2 root root    80 Sep  1 16:57 ./
drwxr-xr-x 15 root root   300 Sep  1 16:57 ../
-rw-r--r--  1 root root  3876 Sep  1 18:02 log.txt
-rwxr-xr-x  1 root root 10130 Sep  1 16:57 script*
root@AlsServerII:~#

so running /tmp/user.scripts/tmpScripts/Encode/script will execure it (and its original including /bin/bash)

23 minutes ago, ZGief said:

and the script itself doesn't have that first line that "bash" line to instruct the os how to run it.

just as note see upper comment

ZGief · September 1

36 minutes ago, alturismo said:

yes there is, as of the boot restriction ... they are copied to /tmp

Interesting! Learned something new today. Thanks!

ZGief · September 1

Reading through my latest mover tuner logs (doesn't happen too often because it only triggers when cache is 70% full) I noticed a few errors I wanted to report in case they are bugs.

First one:

Sep  1 03:49:58 Tower root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1050: [: : integer expression expected

Around line 1050 the code is

if [ "$NBLINKS" -gt 1 ] && [ "$FILESIZE" = 0 ] ; then
    continue
fi

The second one was:

Sep  1 03:49:58 Tower root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1231: 0: command not found

#Run mover after script if specified.
if [ ! -z "$AFTERSCRIPT" ] && [ ! -e $SOFTSTOPFILE ]; then
    if [ -f "$AFTERSCRIPT" ] && [ -x "$AFTERSCRIPT" ]; then
        mvlogger "Launching after script: $AFTERSCRIPT"
        eval \"$AFTERSCRIPT\"
        mvlogger "After script finished with result: $( $? && "OK" || $?)" <---- This is line 1231
    else
        mvlogger "After script file $AFTERSCRIPT does not exist or is not executable."
        mvlogger "Skipping"
    fi
fi

I do run before and after scripts which seems to be running just fine. So I'm not sure what this is trying to tell me.

Reynald · September 4

On 8/19/2024 at 10:03 PM, ramjam824 said:

Good luck, I hope the move goes smoothly!

Hi all,

I'm back online, a coolant leak from my AIO ruined my motherboard and thus I've been with no server, no dev machine, and so on, plus fiber internet took some time to come, but back on business now, I'm going to catch the backlog from everyone's messages

Edited September 4 by Reynald

Notwist · September 4

Is there any chance you could add an option to allow hard-link breaking when mover runs? I know that was recently patched out and it's actually useful for cases where you want to keep a file on both the cache and array (such as maintaining seeding from cache but moving primary file to array), but you want to maintain hard links until mover runs to prevent cache storage overfilling in the meantime.

Edited September 4 by Notwist
improved description of situation

Reynald · September 4

On 8/22/2024 at 10:25 AM, Yusseiin said:
I have installed the "Reynalds" one but it seems not working.
In the log i can see that it is saying
0:15:16.521 Primary storage: cache - size: 692GiB - used: 85 % (611GiB)
10:15:16.522 Secondary storage: user0
10:15:16.531 Share Information: Name: downloads - Path: /mnt/cache/downloads
10:15:16.537 Moving threshold: 75% (519GiB) ; Freeing threshold: 50% (346GiB)
10:15:16.539 Mover action: cache->user0 (cache:yes). Pool is above moving threshold percentage:  85% >= 75%.
10:15:16.540 => Will smart move old files from cache to user0. Nothing will be moved from user0 to cache
10:15:16.553 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-08-22T101515.list for downloads
In the "Mover_action" all the files has the
move from primary|/mnt/cache|/mnt/user0
but in the end no file where moved.
the settings are basic ones

Can someone guide me to throubleshooting this?

Test mode will not move any files, that's a "dry run". You can check from log if action intended are what you want, and then proceed without test mode. Here is a good explanation:

On 8/20/2024 at 3:34 PM, ramjam824 said:

100%, run in test mode, plug-in filters set to YES, and select options you feel align with what you are seeking as output. Keep tinkering until you see the output match your expectations. Then, turn test mode off and let it rip!

Reynald · September 4

On 8/27/2024 at 6:42 PM, stelks said:

It is also broken for me on unraid 7 beta 2. Uninstalling fixes the issue. Here are the logs:

Aug 27 11:36:18 LilNASX move: mover: started
Aug 27 11:37:43 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 206: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:43 LilNASX move: cat: write error: Broken pipe
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: echo: write error: Broken pipe
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: echo: write error: Broken pipe
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:44 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 119: /usr/local/sbin/move: No such file or directory
Aug 27 11:37:45 LilNASX move: mover: finished

This is from the old plugin as the newest age_mover version does not use move binary anymore.

Edited September 4 by Reynald

Reynald · September 4

On 8/19/2024 at 11:38 PM, drumtechphoto said:

Hello all, thought I would share since the recent plugin update as of today, 8/19/2024:

I thank all that are working on this CA Mover Tuning Plugin, most recent @Reynald. I noticed a bug when running the Mover manually on a share, in my example "/mnt/user/media". The command below has a slight typo near the end, where "move" should be "mover".

Original/Before:
"find "/mnt/cache/media" -depth | /usr/local/sbin/~~move~~ -d 1"

Correction/After:

"find "/mnt/cache/media" -depth | /usr/local/sbin/mover -d 1"

The script as it was (below), gave me the following output and made me wonder, "hey, this plugin uses mover, not move, so let me investigate":

mvlogger: *********************************MOVER -SHARE- START*******************************
mvlogger: Mon Aug 19 17:23:47 EDT 2024
mvlogger: Share supplied media
Sharecfg: /boot/config/shares/media.cfg
mvlogger: Cache Pool Name: cache
mvlogger: Share Path: /mnt/cache/media
mvlogger: Complete Mover Command: find "/mnt/cache/media" -depth | /usr/local/sbin/move -d 1
mvlogger: Mon Aug 19 17:23:47 EDT 2024
mvlogger: ********************************Mover Finished*****************************

_________________________________________________________________________________________________

##Edit:

The bug is located on line # 96 of file "share_mover", snippet below of what line the issue was found on:

File & File Location: "/usr/local/emhttp/plugins/ca.mover.tuning/share_mover"

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#81#SHAREPATH="/mnt/$CACHEPOOLNAME/$SHARE"
mvlogger "Share Path: $SHAREPATH"

if [ "$SHAREPATH" = "/mnt/cache" ] ; then
echo "Exiting, can't determine cache name used"
exit 2
else
if [ -d $SHAREPATH ] ; then
FINDSTR="find \"$SHAREPATH\" -depth"

if [ "$TESTMODE" = "yes" ] ; then
mvlogger "Running in Test Mode: $FINDSTR"
eval "$FINDSTR"
else
#Add Mover to find string
#96: Fix ME--># FINDSTR+=" | /usr/local/sbin/move -d $LOGLEVEL"
mvlogger "Complete Mover Command: $FINDSTR"
eval " $FINDSTR"
fi
else
mvlogger "Directory does not exist: $SHAREPATH"
fi
fi

fi

rm -f $PIDFILE
$NOW=`date`
mvlogger "$NOW"
mvlogger "********************************Mover Finished*****************************"

Nice investigation and fix proposal! Unfortunatly piping to mover will execute mover.php that will launch share_mover that will probably pipe again to mover which will execute mover.php that will launch share_mover... Or the loop won't be continued thanks to the pidfile.

I have not updated share_mover yet (which is exactly how it was when hugenbdd maintained this plugin and it's not ready for Unraid 7.x yet. I'm working on it tonight

Edit: The fact is that move executable for unraid 7 is located elsewhere. Line 96 (you spotted the right place ;)) now looks like:

FINDSTR+=" | $([ -f /usr/libexec/unraid/move ] && echo /usr/libexec/unraid/move || which move) -d $LOGLEVEL"

Which will use the unraid 7 /usr/libexec/unraid/move binary or look in /usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin to cover previous unraid versions.

Edited September 5 by Reynald

Reynald · September 4

15 hours ago, Notwist said:

Is there any chance you could add an option to allow hard-link breaking when mover runs? I know that was recently patched out and it's actually useful for cases where you want to keep a file on both the cache and array (such as maintaining seeding from cache but moving primary file to array), but you want to maintain hard links until mover runs to prevent cache storage overfilling in the meantime.

I see but... I don't see the point of having the same file with 2 different names/paths on 2 different storages. Moving just the primary file to secondary storage won't save any space on primary storage, so better to keep the hardlinked files on the same place.

However, if I understand your use case correctly you can keep a file on both the cache and the array by using this option:

I'd like feedback on this feature from those who use it, regarding deletion in particular.

Edited September 5 by Reynald
grammar

Reynald · September 4

2024.09.05

Add compatibility with unraid 7.x for share_mover ([R3yn4ld](https://github.com/R3yn4ld))
Fix "integer expression expected" Thanks to: [RonaldJerez] (don't know his member screenname here)
and "0: command not found" bugs [R3yn4ld]

Reynald · September 4

On 8/28/2024 at 1:01 AM, solidno8 said:
Could you add in the option for the script to honor hidden folders in the root directory like the old mover script by setting the option below?
shopt -s dotglob

In fact, if it doesn't work it's because you spotted a bug...

Hidden Files are handled here (if the option is set to yes, then hidden files are excluded):

https://github.com/R3yn4ld/ca.mover.tuning/blob/471734ae898962c0449f51f8876554188539d601/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L619

        #Add Ignore Hidden Files to find string
        if ([ -z "$OMOVERTHRESH" ] || [ $POOLPCTUSED -le $OMOVERTHRESH ]) && ([ "$SHAREUSECACHE" = "prefer" ] || [ "$SHAREUSECACHE" = "yes" ]) && [ "$HIDDENFILES" == "yes" ]; then
            mvlogger "Skipping Hidden Files. File size are taken into account in the calculation of the threshold"
            FINDSTR+=" -not -path '*/\.*'"
            HIDDENFILES_RESERVED_SPACE=$(find "/mnt/$PRIMARYSTORAGENAME/$SHARENAME" -type f -path '*/\.*' -print0 | xargs -0 du -sh | awk '{ sum += $1 } END { print sum }')
            echo "$PRIMARYSTORAGENAME|$PRIMARYSTORAGENAME|$SHARENAME|skipped|9999999999|$PRIMARYSIZETHRESH|$SKIPPED_PATH_RESERVED_SPACE|0|0|0|/mnt/$PRIMARYSTORAGENAME/$SHARENAME hidden files" >>$FILTERED_FILELIST
            mvlogger "    Hidden files are using $(bytes_to_human $SKIPPED_PATH_RESERVED_SPACE)"
        fi

But... I broke the bash globbing pattern option while adding nullglob:

Quote

#Let find find hidden files.
shopt -s nullglob

added back dotglob, it's working:

testing with ignoring hidden files yes and pushing the update

With ignore hidden files options set to yes, their total size is computed and they are not listed:

Publishing update in a couple of minutes, thank you @solidno8 for the head up !

Edited September 5 by Reynald

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation