[Plugin] Mover Tuning

[UnknownWitcher]

On 2/25/2024 at 2:33 PM, hugenbdd said:

the mover.old file is so that we can restore the original mover binary.  The location changed several releases ago but I'm not certain what version.  If we remove this then it will not work for older releases.  I will try and work on a way around this to satisfy older and newer releases this week.  Which I think will be a "longer" if statement, and to check for the file before mv'ing it and also creating another if to create a copy for the sbin location file.

 

You could check the unraid version then define the location based on the minimum version required for the change.

mover_path="/usr/local/bin/"
if awk "BEGIN {exit !($(cat cat /etc/unraid-version | grep -Po '([0-9]+\.?)+') >= 6.12.6)}"; then
    mover_path="/usr/local/sbin/"
fi
if [[ ! -f "${mover_path}/mover.old" ]]; then mv "${mover_path}/mover" "${mover_path}/mover.old"; fi

 

or run a function to return a boolean if changes are required for multiple parts of the script.

 

function min_unraid_version {
    local min_version="6.12.6"
    if awk "BEGIN {exit !($(cat cat /etc/unraid-version | grep -Po '([0-9]+\.?)+') >= $min_version)}"; then
        true; return
    fi
    false
}
if min_unraid_version; then
    echo "/usr/local/sbin/"
else
    echo "/usr/local/bin/"
fi

 

[noonamer]

On 3/5/2024 at 9:27 PM, Swarles said:

In that case you should be able to echo the list of file paths and pipe them into "/usr/local/sbin/move -d".

that's what I do, but this won't be able to handle hardlinks

[wuudogg]

Appreciate all of the effort provided by the devs to improve this script and your consideration for multiple firmware versions. It seems I have to reinstall and reconfigure the Mover Tuning plugin after boot with 6.12.8 in order for it to work.   A minor inconvenience and worth it to get the full power of Mover Tuning.  If the soft link is the only issue, was thinking of putting a section of your post-install script in 'User Scripts" at 'Startup of Array'. Does this seem a reasonable temporary workaround?

 

#!/bin/bash
echo "The 'post-install' script"
echo "Backup current mover binary, command may fail depending on version of unRAID"
echo ""
echo "Creating Soft link for 6.12 RC8 or newer since /usr/local/sbin/move is now lcoated in /usr/local/bin/move"
if [[ ! -e /usr/local/sbin/move ]]; then ln -s /usr/local/bin/move /usr/local/sbin/move; fi
echo "Done"

 

[wuudogg]

On 3/14/2024 at 10:01 AM, wuudogg said:

Appreciate all of the effort provided by the devs to improve this script and your consideration for multiple firmware versions. It seems I have to reinstall and reconfigure the Mover Tuning plugin after boot with 6.12.8 in order for it to work.   A minor inconvenience and worth it to get the full power of Mover Tuning.  If the soft link is the only issue, was thinking of putting a section of your post-install script in 'User Scripts" at 'Startup of Array'. Does this seem a reasonable temporary workaround?

 

#!/bin/bash
echo "The 'post-install' script"
echo "Backup current mover binary, command may fail depending on version of unRAID"
echo ""
echo "Creating Soft link for 6.12 RC8 or newer since /usr/local/sbin/move is now lcoated in /usr/local/bin/move"
if [[ ! -e /usr/local/sbin/move ]]; then ln -s /usr/local/bin/move /usr/local/sbin/move; fi
echo "Done"

 

Confirmed the softlink alone doesn't work. Have to remove plugin and then reinstall plugin and adjust any settings to get mover working after reboot.

[Spritzup]

Apologies in advance, I'm usually one to read the entirety of a thread, but I threw in the towel around page 37. In any case, either it's not possible, or I can't wrap my mind around how to bend mover to my will.

 

I am currently using a 3TB ZFS pool as cache and would like (in a perfect world), mover to do the following:

 

- ignore hidden files / folders (DONE!)

- ignore specific directories (Read the FAQ, just need to implement)

- Move my media items after a set # of days OR when usage hits a pre-defined threshold (but still respecting the first two points)

- Move all other shares at a set interval / time (at least daily) OR when usage hits a pre-defined threshold (but still respecting the first two points)

 

I realize the irony of being picky about how mover works, when back in the 4.x days, you got what you got and you were happy to have it; but any help or guidance would be appreciated!

 

~Spritz

[Terebi]

18 hours ago, Spritzup said:

Apologies in advance, I'm usually one to read the entirety of a thread, but I threw in the towel around page 37. In any case, either it's not possible, or I can't wrap my mind around how to bend mover to my will.

 

I am currently using a 3TB ZFS pool as cache and would like (in a perfect world), mover to do the following:

 

- ignore hidden files / folders (DONE!)

- ignore specific directories (Read the FAQ, just need to implement)

- Move my media items after a set # of days OR when usage hits a pre-defined threshold (but still respecting the first two points)

- Move all other shares at a set interval / time (at least daily) OR when usage hits a pre-defined threshold (but still respecting the first two points)

 

I realize the irony of being picky about how mover works, when back in the 4.x days, you got what you got and you were happy to have it; but any help or guidance would be appreciated!

 

~Spritz

#3 can be dealt with using either mover tuning settings, or by adding in this script 

 

[Swarles]

On 3/20/2024 at 4:04 AM, wuudogg said:

Confirmed the softlink alone doesn't work. Have to remove plugin and then reinstall plugin and adjust any settings to get mover working after reboot.

Interesting, so are you having issues running Mover Tuning entirely? What version of unraid are you using?

[Swarles]

On 3/20/2024 at 10:21 AM, Spritzup said:

Apologies in advance, I'm usually one to read the entirety of a thread, but I threw in the towel around page 37. In any case, either it's not possible, or I can't wrap my mind around how to bend mover to my will.

 

I am currently using a 3TB ZFS pool as cache and would like (in a perfect world), mover to do the following:

 

- ignore hidden files / folders (DONE!)

- ignore specific directories (Read the FAQ, just need to implement)

- Move my media items after a set # of days OR when usage hits a pre-defined threshold (but still respecting the first two points)

- Move all other shares at a set interval / time (at least daily) OR when usage hits a pre-defined threshold (but still respecting the first two points)

 

I realize the irony of being picky about how mover works, when back in the 4.x days, you got what you got and you were happy to have it; but any help or guidance would be appreciated!

 

~Spritz

Unfortunately your exact requirements are not currently supported. The primary issue is that in point #3 the threshold will take precedence, so if the threshold is not exceeded, files that are older than X days will not be moved. They will only be moved if the threshold has been exceeded AND they are older than X days.

 

Furthermore, the threshold operates on a pool level not a share level, as a share typically does not have a set maximum capacity. So the threshold takes into account the usage of all shares on that pool. 

 

You might be able to implement your desired logic by slightly modifying the before script Terebi posted above. Otherwise I recommend setting the threshold to a value that is suitable for all your shares. And then understand that it will only ever move the shares if that threshold is exceeded and then if it is going to move files it will use all your other settings.

 

I'm remaking mover tuning from the ground up and it will support your desired functionality, but it is quite a ways off at the moment.

[Spritzup]

Thank you @Terebi and @Swarles for the help and guidance, I really appreciate it. There was a time (pre-4 kids) that I would have loved to dive into scripting a custom solution for myself, but now many of these things are a balance of functionality and time investment. That said, looking at the options available to me already, and the guidance you provided, would the following not mostly work -->

 

- Set mover to run every x Hours

- Set mover tuner to move at x% (applicable for Media)

- Set mover tuner to use a forced schedule to move all (this covers my "critical" shares)

- For my media share, have it override mover settings so it doesn't move on a schedule

- Have it only move files that are older than x days for the media share

 

It is a bit of a compromise to what I had originally wanted, but it seems to mostly check all my boxes... unless I'm missing something?

 

Thanks!

~Spritz

[Pete]

It is nice that I can choose what filetypes should stay on the cache for faster access and less disc spin up.

 

But it would be nice if I could move these filetypes from the array to the cache as well. I omit the cache when doing  large bulk transfers to prevent wearing out the cache SSDs and then these types of files never reach the cache. 

 

Is this possible?

 

If not I'd suggest this as a feature request for a future update. It would probably be enough to add a checkbox ("move to cache") behind the "Ignore files listed inside of a text file" and "Ignore file types" menu options because a "move to cache" would make no sense without excluding these files from the normal "move to array" operation. Otherwise the files would be moved back and forth forever. 

[Terebi]

On 3/21/2024 at 7:07 PM, Spritzup said:

Thank you @Terebi and @Swarles for the help and guidance, I really appreciate it. There was a time (pre-4 kids) that I would have loved to dive into scripting a custom solution for myself, but now many of these things are a balance of functionality and time investment. That said, looking at the options available to me already, and the guidance you provided, would the following not mostly work -->

 

- Set mover to run every x Hours

- Set mover tuner to move at x% (applicable for Media)

- Set mover tuner to use a forced schedule to move all (this covers my "critical" shares)

- For my media share, have it override mover settings so it doesn't move on a schedule

- Have it only move files that are older than x days for the media share

 

It is a bit of a compromise to what I had originally wanted, but it seems to mostly check all my boxes... unless I'm missing something?

 

Thanks!

~Spritz

 

I honestly think my script will do what you want with only variable tweaks. 

 

Set my script to your media dir. 

 

Run mover hourly. disable all other tuning settings (age/size)

 

Every hour, everything NOT on the media dir will move to the array. 

 

Every hour the media share will get moved to the array, BUT skip the newest files up to the threshhold. each time a new file is downloaded, the next hour, an equal amount of the oldest data should move. 

 

[Terebi]

7 hours ago, Pete said:

It is nice that I can choose what filetypes should stay on the cache for faster access and less disc spin up.

 

But it would be nice if I could move these filetypes from the array to the cache as well. I omit the cache when doing  large bulk transfers to prevent wearing out the cache SSDs and then these types of files never reach the cache. 

 

Is this possible?

 

If not I'd suggest this as a feature request for a future update. It would probably be enough to add a checkbox ("move to cache") behind the "Ignore files listed inside of a text file" and "Ignore file types" menu options because a "move to cache" would make no sense without excluding these files from the normal "move to array" operation. Otherwise the files would be moved back and forth forever. 

 

During bulk transfers where you don't want to use the cache, either disable the cache temporarily, or copy into the disk shares directly to skip past the cache. 

 

Mover tuner cannot control what happens with the cache at any time OTHER than moving. So all of your bulk imports or whatever, mover tuning is completely irrelevant. It has no interaction whatsoever with file creation/copies etc going on. 

 

Currently there is no "read cache" to move some files off of the array onto the cache, while simultaneously moving other files from the cache to the array (for the same share)

[Renegade605]

7 hours ago, Pete said:

It is nice that I can choose what filetypes should stay on the cache for faster access and less disc spin up.

 

But it would be nice if I could move these filetypes from the array to the cache as well. I omit the cache when doing  large bulk transfers to prevent wearing out the cache SSDs and then these types of files never reach the cache. 

 

Is this possible?

 

If not I'd suggest this as a feature request for a future update. It would probably be enough to add a checkbox ("move to cache") behind the "Ignore files listed inside of a text file" and "Ignore file types" menu options because a "move to cache" would make no sense without excluding these files from the normal "move to array" operation. Otherwise the files would be moved back and forth forever. 

 

This is pretty far outside of the scope of what the mover (and the cache itself) is intended for. If there is specific data that you want on the cache at all times, I suggest making a cache-only share for that data and keeping it separate from the other files that should be moved.

[Spritzup]

19 hours ago, Terebi said:

 

I honestly think my script will do what you want with only variable tweaks. 

 

Set my script to your media dir. 

 

Run mover hourly. disable all other tuning settings (age/size)

 

Every hour, everything NOT on the media dir will move to the array. 

 

Every hour the media share will get moved to the array, BUT skip the newest files up to the threshhold. each time a new file is downloaded, the next hour, an equal amount of the oldest data should move. 

 

I'll have a closer look at your script in the next few days, thanks for that! Would I put in your script to ignore hidden files / certain directories, or would that still be taken care of by mover tuning?

[Terebi]

3 hours ago, Spritzup said:

I'll have a closer look at your script in the next few days, thanks for that! Would I put in your script to ignore hidden files / certain directories, or would that still be taken care of by mover tuning?

 

Mover tuning.  The only thing my script does is tell mover to ignore up to X newest bytes of content in a particular directory.  Its to keep my media share full of newer content to keep drives spun down. Other shares, or other settings from mover or mover tuning are not affected. 

[Swarles]

On 3/25/2024 at 10:39 PM, Pete said:

It is nice that I can choose what filetypes should stay on the cache for faster access and less disc spin up.

 

But it would be nice if I could move these filetypes from the array to the cache as well. I omit the cache when doing  large bulk transfers to prevent wearing out the cache SSDs and then these types of files never reach the cache. 

 

Is this possible?

 

If not I'd suggest this as a feature request for a future update. It would probably be enough to add a checkbox ("move to cache") behind the "Ignore files listed inside of a text file" and "Ignore file types" menu options because a "move to cache" would make no sense without excluding these files from the normal "move to array" operation. Otherwise the files would be moved back and forth forever. 

Yeah so at the moment that's not possible with the raw plugin and I totally understand the desire for it. Infact at the moment, there is no mover tuning settings for array->cache moves at all.

 

I have been working on creating a new plugin from the ground up which will bring all the tuning settings to array->cache moves as well even though it probably won't be used by 90% of people. Unfortunately because the mover only really moves stuff one way depending on how the share is configured, what you desire is not easily achievable by manipulating the built in mover. So there may be circumstances where files on the array that your tuning settings would normally keep on the cache will remain on the array. This would be likely to occur if you set up mover tuning after having already established your server file system, or changing the mover tuning settings considerably.

 

What I will think about though, is creating a button that will extract files on the destination drive that match  the source drives mover tuning settings (but the opposite), for the purpose of ensuring all files on that share are following the tuned settings. That way you would only need to run it once after making changes to mover tuning. I think this would address your feature request 

 

In saying this, the plugin is still a fair way off and this feature will be lower down on my priority list. 

[ezek1el3000]

I am still getting the following error on boot:

 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

Server boots up fine and only hangs for a couple of seconds on this error. Latest CA Version 2023.12.19
The error has only showed up on (re)boot in the console

 

ls -l /usr/local/bin/mover

Output:
-rwxr-xr-x 1 root root 161 Mar 26 15:01 /usr/local/bin/mover*

 

cat /etc/cron.d/root

Output: # Generated mover schedule:
0 8 * * * /usr/local/sbin/mover &> /dev/null

 

[wgstarks]

1 hour ago, ezek1el3000 said:

am still getting the following error on boot:

 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

 

Same here.

[MaverickMacker]

6 hours ago, ezek1el3000 said:

I am still getting the following error on boot:

 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

Server boots up fine and only hangs for a couple of seconds on this error. Latest CA Version 2023.12.19
The error has only showed up on (re)boot in the console

 

ls -l /usr/local/bin/mover

Output:
-rwxr-xr-x 1 root root 161 Mar 26 15:01 /usr/local/bin/mover*

 

cat /etc/cron.d/root

Output: # Generated mover schedule:
0 8 * * * /usr/local/sbin/mover &> /dev/null

 

Same here. GUI was locking up and unresponsive every three days and mover did not work. Had to uninstall mover tuner now have no issues. This only happens on my 6.12.8 server; my 6.12.4 server does not have this issue. I will not be upgrading that server anytime soon.

[imsoz0]

I recently added a new hard drive to my array and manually balanced the usage of two hard drives using unbalance. Today, I found that small files such as posters and subtitles from my original movie library in the cache were all moved to the array, presumably because I didn't pay attention to this issue during the unbalance operation.

 

Now, I want to reverse the process and move the small files from the array back to the cache, while keeping the folder structure unchanged.

 

I noticed that the mover can be set to "move only files larger than xx MB." However, I'm only moving small files like posters, subtitles, and nfo files, so I need an operation like "move only files smaller than xx MB," but this is not available in the mover.

 

I want to know if there are any plugins or other methods to achieve the above purpose.

[Terebi]

On 3/27/2024 at 1:45 PM, imsoz0 said:

I recently added a new hard drive to my array and manually balanced the usage of two hard drives using unbalance. Today, I found that small files such as posters and subtitles from my original movie library in the cache were all moved to the array, presumably because I didn't pay attention to this issue during the unbalance operation.

 

Now, I want to reverse the process and move the small files from the array back to the cache, while keeping the folder structure unchanged.

 

I noticed that the mover can be set to "move only files larger than xx MB." However, I'm only moving small files like posters, subtitles, and nfo files, so I need an operation like "move only files smaller than xx MB," but this is not available in the mover.

 

I want to know if there are any plugins or other methods to achieve the above purpose.

 

If you are talking about a plex/emby/kodi library, the general recommendation is that it should be on the appdata share, and most people keep appdata as cache only (and exclusive) for performance reasons.  If that data is part of the appdata share,  just set the mover to move from array to cache, then run the mover.  After everything is moved, disable secondary storage and make sure the appdata share is set to exclusive. 

 

To get a subset of files back to cache you could also set the mover from array->cache, and then manually run the mover on a given subdirectory by using the following command

 

 

find /mnt/user/DirectoryNameGoesHere* -type f | /usr/local/sbin/move&

                      ^^^^^^^^^^^^^^^^^^^^^^^^

[fritolays]

On 10/27/2022 at 12:09 PM, hugenbdd said:

How to ignore a SINGLE file

1.) Find the path of the file you wish to ignore.
    ls -ltr /mnt/cache/Download/complete/test.txt
        root@Tower:/# ls -ltr /mnt/cache/Download/complete/test.txt
        -rwxrwxrwx 1 root root 14 Oct 27 11:32 /mnt/cache/Download/complete/test.txt*
2.) Copy the complete path used.  
    /mnt/cache/Download/complete/test.txt
3.) Create a text file to hold the ignore list.
    vi /mnt/user/appdata/mover-ignore/mover_ignore.txt
4.) Add a file path to the mover_ignore.txt file
    While still in vi press the i button on your keyboard.
    Right click mouse.
5.) Exit and Save
    ESC key, then : key, then w key, then q key
6.) Verify file was saved
    cat /mnt/user/appdata/mover-ignore/mover_ignore.txt
    "This should print out 1 line you just entered into the mover_ignore.txt"
7.) Verify the find command results does not contain the ignored file
    find "/mnt/cache/Download" -depth | grep -vFf '/mnt/user/appdata/mover-ignore/mover_ignore.txt'
    /mnt/cache/Download is the share name on the cache. (Note, cache name could be different if you have multiple caches or changed the default name)

 

How to ignore Muliple files.

1.) Find the paths of the files you wish to ignore
    ls -ltr /mnt/cache/Download/complete/test.txt
    ls -ltr /mnt/cache/Download/complete/Second_File.txt
2.) Copy the complete paths used to a separate notepad or text file.
    /mnt/cache/Download/complete/test.txt
    /mnt/cache/Download/complete/Second_File.txt
3.) Copy the paths in your notepad to the clip board. 
    Select, the right click copy
4.) Create a text file to hold the ignore list.
    vi /mnt/user/appdata/mover-ignore/mover_ignore.txt
5.) Add the file paths to the mover_ignore.txt file
    While still in vi press the i button on your keyboard.
    Right click mouse.

    You should now have two file paths in the file.
6.) Exit and Save
    ESC key, then : key, then w key, then q key
7.) Verify file was saved
    cat /mnt/user/appdata/mover-ignore/mover_ignore.txt
    "This should print out 2 lines you just entered into the mover_ignore.txt"
8.) Verify the find command results does not contain the ignored files
    find "/mnt/cache/Download" -depth | grep -vFf '/mnt/user/appdata/mover-ignore/mover_ignore.txt'
    /mnt/cache/Download is the share name on the cache. (Note, cache name could be different if you have multiple caches or changed the default name)

How to Ignore a directory
instead of a file path, use a directory path.  no * or / at the end.
This may cause issues if you have other files or directories named the similar but with extra text.

/mnt/cache/Download/complete

*Note /mnt/cache/Download/complete  will also ignore /mnt/cache/Download/complete-old 

 

 

*I use vi in this example instead of creating a file in windows, as windows can add ^m characters to the end of the line, causing issues in Linux.   This would not be an issue if dos2unix was included in unRAID.

 

**Basic vi commands

https://www.cs.colostate.edu/helpdocs/vi.html

 

So I wrote a script to help manage which files and folders to exclude.

When the mover runs it will convert line endings to unix format, validate file and directory paths, then compile a complete exclusion list which is then passed to the mover.

Its progress and any errors can be viewed in Unraid's system log (Tools -> System Log).

 

https://gist.github.com/fritolays/da5b835080ad0a4f8a48013b7b7745d8

[danimal86]

I've been using the mover tuning to keep new media on my ssd cache pool as long as possible to keep drives from spinning up.  It works great, takes a little bit of time to set the duration just right....the only time i run into problems is when i want to grab a bunch of new media, then i need to lower the duration to move data off to be ready for the new. 

 

Is it possible to have it so that the mover just moves the oldest media on the cache off to the array once the cache pool hits a certain % full?

[Terebi]

14 minutes ago, danimal86 said:

I've been using the mover tuning to keep new media on my ssd cache pool as long as possible to keep drives from spinning up.  It works great, takes a little bit of time to set the duration just right....the only time i run into problems is when i want to grab a bunch of new media, then i need to lower the duration to move data off to be ready for the new. 

 

Is it possible to have it so that the mover just moves the oldest media on the cache off to the array once the cache pool hits a certain % full?

 

 

[Swarles]

On 3/28/2024 at 5:45 AM, imsoz0 said:

I recently added a new hard drive to my array and manually balanced the usage of two hard drives using unbalance. Today, I found that small files such as posters and subtitles from my original movie library in the cache were all moved to the array, presumably because I didn't pay attention to this issue during the unbalance operation.

 

Now, I want to reverse the process and move the small files from the array back to the cache, while keeping the folder structure unchanged.

 

I noticed that the mover can be set to "move only files larger than xx MB." However, I'm only moving small files like posters, subtitles, and nfo files, so I need an operation like "move only files smaller than xx MB," but this is not available in the mover.

 

I want to know if there are any plugins or other methods to achieve the above purpose.

Depending on how many files you have on your array at the moment you could just set it to move from array->cache run the mover then set it back to cache->array and make it use your settings. I'd you have lots of files that'll either take a long time, waste write cycles, or be impossible. So what you should do then is come up with your own find string to find the files you want to move from the array to the cache, eg.

find /mnt/disk*/ShareName -depth -size -50M

That will search all of the disks that contain the relevant share and find files that are less than (and equal to I think) 50MB. Use ChatGPT to fine tune it to the exact parameters you want if you're not familiar with the find command. It will print out a list of files and folders it finds. 

 

Then if you want to manually move them you can do that, or you can ask ChatGPT how to write your own command with eg rsync to do it, or you can temporarily set the mover for that share to array->cache (DON'T RUN THE MOVER NOW) you will then manually invoke the mover by adding the following to end of your find string. 

| /usr/local/sbin/move

That will pipe all the file paths the find string finds into the built in mover which will then see the share is set to move from the array to the cache and it will do the moves for you

 

Don't forget to change your share mover setting back to what you want.

Edited April 1 by Swarles