[Plugin] Mover Tuning

[Autchirion]

It is made to handle bursts, but sometimes these bursts are to big in size to be handled by the cache. I was hoping to find a solution without throwing money (aka bigger cache, not reusing my first SSD) at it.

[Kilrah]

Well it's either accept the temporary slowdown or shell out for bigger drives, no miracle solution...

[Schaka]

First of all, I'm sorry for hijacking this thread. I tried to find a solution in other threads, but everything seems outdated. I'm still on 6.11.5.

 

I was looking to manually trigger the mover via cli/script and expected it to be blocking, so that I may shut down docker containers before running it so files won't be in use.

 

There seem to be two problems where calling mover via ssh just seems to do nothing (no disk activity), using the GUI it does run but then calling mover stop does nothing again.

 

What is the correct way (maybe the way this plugin does it) to call the mover? Can I just call one of the scripts the plugin uses and it will be blocking? I'd love to utilize the cache again, but unless I can get the mover to run after my nightly backup, I don't see another way of doing this atm.

 

 

Edited January 28 by Schaka

[Revan335]

58 minutes ago, Schaka said:

it to be blocking

Any Error/Message do you become?

[Kilrah]

43 minutes ago, Revan335 said:

Any Error/Message do you become?

What he's after is to be able to start the mover in a script and the script not continuing until the mover is done

[hugenbdd]

3 hours ago, Schaka said:

First of all, I'm sorry for hijacking this thread. I tried to find a solution in other threads, but everything seems outdated. I'm still on 6.11.5.

 

I was looking to manually trigger the mover via cli/script and expected it to be blocking, so that I may shut down docker containers before running it so files won't be in use.

 

There seem to be two problems where calling mover via ssh just seems to do nothing (no disk activity), using the GUI it does run but then calling mover stop does nothing again.

 

What is the correct way (maybe the way this plugin does it) to call the mover? Can I just call one of the scripts the plugin uses and it will be blocking? I'd love to utilize the cache again, but unless I can get the mover to run after my nightly backup, I don't see another way of doing this atm.

 

 

Should be able to call it from the command line. 

 

"SoftStop" tries to stop the mover once the current file has been transfered. (i.e. the code checks to see if softstop was triggered before moving to the next file in the list.). As long as you are not using hardlinks.  Hardlinks does not loop through a filelist to move files but rather sends the full filelist to the binary mover file.

 

root@Tower:~# mover status
Log Level: 1
mover: not running

root@Tower:~# mover stop
Log Level: 1
mover: not running

root@Tower:~# mover softstop
Log Level: 1
Soft Stop Requested
mover: not running

root@Tower:~# mover start
Log Level: 1
mover: started
mover: finished

[Schaka]

I had "Disable Mover running on a schedule" checked. This seems to break the functionality of just calling mover via the CLI.
I never made the connection, but your examples helped me knowing what to look for.

 

Now the real question is, how do I both disable the the regular mover schedule AND still trigger the mover manually?
My best guess is that this plugin modifies the mover script to do nothing and then implements its own functionality. 

 

When disabling the schedule, how do I trigger what the plugin does via cli or the original mover script?

Your help is much appreciated!

[Indi]

When mover is done moving files, it's leaving the directory/folder on the cache drive, so I have hundreds and hundreds of empty folders leftover on the cache 

How can I get mover tuner to remove folders after it's moved the file?

[Revan335]

1 hour ago, Indi said:

When mover is done moving files, it's leaving the directory/folder on the cache drive, so I have hundreds and hundreds of empty folders leftover on the cache 

Its ZFS or Datasets involved?

[Indi]

11 minutes ago, Revan335 said:

Its ZFS or Datasets involved?

This cache drive is ZFS (3 m.2 in raid0) with only a single dataset, the folders are a few levels down from the root share

[Revan335]

I have problems sometimes? too. ZFS and Datasets on Raid 1. 2x NVMes.

Then I must remove this empty Folders/Datasets manually.

[Terebi]

Like many others, I have wanted to use mover tuner to keep my cache full, so that more recent media is coming off of cache. This can give better performance, and reduce disk spin ups.  @hugenbdd said he was going to work on a more fully featured script to do this, here. But I am impatient. 

 

Mover tuner already supports a ignoring files which are listed in a config file. Tuner also already supports running a script before running the mover.  So, All we need is a script which produces a list of files we wish to not move. 

 

Details of how to do this moved to the below link for better visibility and maintenance. 

 

[GUIDE] HOW TO KEEP CACHE DRIVE FULL OF MEDIA

 

 

Edited February 17 by Terebi
move to standalone guide

[Indi]

6 hours ago, Terebi said:

Like many others, I have wanted to use mover tuner to keep my cache full, so that more recent media is coming off of cache. This can give better performance, and reduce disk spin ups.  @hugenbdd said he was going to work on a more fully featured script to do this, here. But I am impatient. 

 

Mover tuner already supports a ignoring files which are listed in a config file. Tuner also already supports running a script before running the mover.  So, All we need is a script which produces a list of files we wish to not move. 

 

I am a developer, but unfortunately not a linux one, but logic is logic. And there is ChatGPT, which knows bash commands.  I was (through quite a few iterations) able to produce a script which I think gets us into the ballpark.   Yes, This code was produced by ChatGPT.  USE AT YOUR OWN RISK. But I think its pretty safe...   

 

The one gap I think that there might be is in how hardlinked files are processed. Its possible one copy of the hardlink could be listed and not the other. Its also possible hardlinked files are double counted.  I tried to get chatgpt to fix these problems, but it ended up just getting confused and breaking things. Since I personally do not use hardlinks much, I'm going to use the script as is for now.

 

Setup

Copy the script into your appdata or somewhere (preferably someplace that won't get moved by mover). You may need to chmod the file to make it executable.  I recommend appdata, because if you put it in data it may either get moved by the mover, or reading the file will spin up a drive when maybe there is nothing to actually move. 

 

Modify the variables at the top of the file as needed.

 

Set mover tuner to ignore files contained in the output filename : 

 

Add the script to mover tuner to run before moves. 

 

If you run the script by hand (bash moverignore.sh) you can see the files that it will keep on the cache. 

You can then run the following command to test if mover will not move the files in question (any files in the ignore file, should not appear in the output of this command)   " find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/appdata/moverignore.txt'"

 

You can also run mover in the commandline, and verify that it does not move any of the listed files (but should continue to move unlisted files)

 

#!/bin/bash

# Define variables
TARGET_DIR="/mnt/cache/data"
OUTPUT_DIR="/mnt/user/appdata"
OUTPUT_FILE="$OUTPUT_DIR/moverignore.txt"
MAX_SIZE="500000000000"  # 500 gigabytes in bytes
EXTENSIONS=("mkv" "srt")

# Ensure the output directory exists
mkdir -p "$OUTPUT_DIR"

# Cleanup previous temporary files
rm -f "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_filtered_metadata.txt"

# Step 1: Change directory to the target directory
cd "$TARGET_DIR" || exit

# Step 2: Find files with specified extensions and obtain metadata

#find via mtime
#find "$(pwd)" -type f \( -iname "*.${EXTENSIONS[0]}" -o -iname "*.${EXTENSIONS[1]}" \) -exec stat --printf="%Y %s %N\0" {} + > "$OUTPUT_DIR/temp_metadata.txt"

#find via ctime
find "$(pwd)" -type f \( -iname "*.${EXTENSIONS[0]}" -o -iname "*.${EXTENSIONS[1]}" \) -exec stat --printf="%Z %s %N\0" {} + > "$OUTPUT_DIR/temp_metadata.txt"


# Step 3: Sort metadata by date in descending order
sort -z -rn -o "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_metadata.txt"

# Step 4: Get the newest files up to the specified size limit (part 1)
total_size=0
while IFS= read -r -d $'\0' line; do
    size=$(echo "$line" | cut -d ' ' -f2)
    if ((total_size + size <= MAX_SIZE)); then
        echo "$line"
        total_size=$((total_size + size))
    else
        break
    fi
done < "$OUTPUT_DIR/temp_metadata.txt" > "$OUTPUT_DIR/temp_filtered_metadata.txt"

# Step 5: Get the newest files up to the specified size limit (part 2)
#cut -d ' ' -f3- "$OUTPUT_DIR/temp_filtered_metadata.txt" | tr '\0' '\n' > "$OUTPUT_FILE"

# Step 5: Get the newest files up to the specified size limit (part 2)
cut -d ' ' -f3- "$OUTPUT_DIR/temp_filtered_metadata.txt" | tr -d '\0' | sed "s/^'\(.*\)'$/\1/" > "$OUTPUT_FILE"




# Step 6: Cleanup temporary files
rm "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_filtered_metadata.txt"

echo "File list generated and saved to: $OUTPUT_FILE"

 

There is another script to modify mover tuner to exclude files and/or folders, I haven't tested this myself but plan to tonight

https://gist.github.com/fritolays/da5b835080ad0a4f8a48013b7b7745d8

[Terebi]

2 hours ago, Indi said:

There is another script to modify mover tuner to exclude files and/or folders, I haven't tested this myself but plan to tonight

https://gist.github.com/fritolays/da5b835080ad0a4f8a48013b7b7745d8

 

The point of my script that it will dynamically exclude the newest files.  to get similar behavior out of the other script, you would have to constantly edit the script to exclude the new files. 

[Terebi]

On 2/2/2024 at 10:39 AM, Terebi said:

 

 

The one gap I think that there might be is in how hardlinked files are processed. Its possible one copy of the hardlink could be listed and not the other. Its also possible hardlinked files are double counted.  I tried to get chatgpt to fix these problems, but it ended up just getting confused and breaking things. Since I personally do not use hardlinks much, I'm going to use the script as is for now.

 

 

 

 

I believe I have solved the hardlinks problem.  Their space will not be double counted, and all copies will be listed

 

Pinging people from this thread that expressed interest in this idea previously. @NGHTCRWLR @Andiroo2@dopeytree See two posts up for more details

 

<script moved to standalone guide post

Edited February 23 by Terebi
removing scipt

[hugenbdd]

5 minutes ago, Terebi said:

 

 

I believe I have solved the hardlinks problem.  Their space will not be double counted, and all copies will be listed

 

Pinging people from this thread that expressed interest in this idea previously. @NGHTCRWLR @Andiroo2@dopeytree See two posts up for more details

 

#!/bin/bash

# Define variables
TARGET_DIR="/mnt/cache/data"
OUTPUT_DIR="/mnt/user/appdata"
OUTPUT_FILE="$OUTPUT_DIR/moverignore.txt"
MAX_SIZE="500000000000"  # 500 gigabytes in bytes
EXTENSIONS=("mkv" "srt")

# Ensure the output directory exists
mkdir -p "$OUTPUT_DIR"

# Cleanup previous temporary files
rm -f "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_filtered_metadata.txt"

# Step 1: Change directory to the target directory
cd "$TARGET_DIR" || exit

# Step 2: Find files with specified extensions and obtain metadata
find "$(pwd)" -type f \( -iname "*.${EXTENSIONS[0]}" -o -iname "*.${EXTENSIONS[1]}" \) -printf "%i %n %p\0" > "$OUTPUT_DIR/temp_metadata.txt"

# Step 3: Sort metadata by date in descending order
sort -z -rn -o "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_metadata.txt"

# Step 4: Get the newest files up to the specified size limit (part 1)
total_size=0
while IFS= read -r -d $'\0' line; do
    size=$(echo "$line" | cut -d ' ' -f2)
    if ((total_size + size <= MAX_SIZE)); then
        #echo "$line"
        total_size=$((total_size + size))

        # Step 4a: List hardlinks for the current file
        inode=$(echo "$line" | cut -d ' ' -f1)
        find "$(pwd)" -type f -inum "$inode" -not -path "$line" -printf "%i %n %p\0" | \
            while IFS= read -r -d $'\0' hardlink; do
                echo "$hardlink"
            done
    else
        break
    fi
done < "$OUTPUT_DIR/temp_metadata.txt" > "$OUTPUT_DIR/temp_filtered_metadata.txt"

# Step 5: Get the newest files up to the specified size limit (part 2)
cut -d ' ' -f3- "$OUTPUT_DIR/temp_filtered_metadata.txt" | tr '\0' '\n' > "$OUTPUT_FILE"

# Step 6: Cleanup temporary files
rm "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_filtered_metadata.txt"

echo "File list generated and saved to: $OUTPUT_FILE"

 

Is this something we could incorporate into the general plug-in to change the way we handle hardlinks today?

[Terebi]

5 minutes ago, hugenbdd said:

Is this something we could incorporate into the general plug-in to change the way we handle hardlinks today?

 

Hrm, Im not sure what the plugin is doing today, but if anything I came up with is useful to you, please do incorporate it.  

 

This is just sorting all of the files in the relevant dir by date, then listing the files up until the quota to keep on cache is filled. 

 

Step 4 first pass keeps track of hardlinks so they aren't double counted against the quota. 

 

Then step 4a lists all the files, including all of the hardlinks to be ignored by the mover. 

 

I believe the mover is breaking the hardlinks, moving all the files (twice for hardlinks?) then repairing the hardlinks. It doesn't seem like what I came up with helps with that, but im just dipping my toes in. 

Edited February 3 by Terebi

[hugenbdd]

6 minutes ago, Terebi said:

 

Hrm, Im not sure what the plugin is doing today, but if anything I came up with is useful to you, please do incorporate it.  

 

This is just sorting all of the files in the relevant dir by date, then listing the files up until the quota to keep on cache is filled. 

 

Step 4 first pass keeps track of hardlinks so they aren't double counted against the quota. 

 

Then step 4a lists all the files, including all of the hardlinks to be ignored by the mover. 

 

I believe the mover is breaking the hardlinks, moving all the files (twice for hardlinks?) then repairing the hardlinks. It doesn't seem like what I came up with helps with that, but im just dipping my toes in. 

Okay, I'll look at it closer when I get a chance.  

Currently, because I do not know how the unRAID binary is moving hardlinks, I just pipe the whole filelist into the mover binary and it takes care of it.  As opposed to the file list looping through each one (for non hardlinked).

[Anon]

Hello,

 

I am currently facing a problem where the mover is seemingly not working anymore.

I have also posted a detailed description with logs etc. here:

TLDR: Mover runs as a process. Errors are sometimes thrown. But no write activity to array.

In the logs I get the following error multiple times:

 

Feb  5 11:21:17 Tower move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 356: ((: TOTALCACHESIZE += : syntax error: operand expected (error token is "+= ")

 

Can anyone tell me what the cause of this could be? Or in general just some more information on what I could do to hunt down the underlying problem that is preventing the mover from actually moving stuff.

 

 

Edit: After manually moving most of the files the mover should have moved via "unbalance" I restarted the mover to further troubleshoot. It brought the same error as above a few times but then actually started moving files.

Can someone maybe explain how this could have happend? Does the Mover first have to go through and check each file before it starts moving? Or why did it not move a single file for over a hour when running with lots of files needing to be moved vs. starting to move files after only a few seconds when the cache is not very full?

 

Edited February 5 by Anon
Update

[Swarles]

17 hours ago, Anon said:

Hello,

 

I am currently facing a problem where the mover is seemingly not working anymore.

I have also posted a detailed description with logs etc. here:

TLDR: Mover runs as a process. Errors are sometimes thrown. But no write activity to array.

In the logs I get the following error multiple times:

 

Feb  5 11:21:17 Tower move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 356: ((: TOTALCACHESIZE += : syntax error: operand expected (error token is "+= ")

 

Can anyone tell me what the cause of this could be? Or in general just some more information on what I could do to hunt down the underlying problem that is preventing the mover from actually moving stuff.

 

 

Edit: After manually moving most of the files the mover should have moved via "unbalance" I restarted the mover to further troubleshoot. It brought the same error as above a few times but then actually started moving files.

Can someone maybe explain how this could have happend? Does the Mover first have to go through and check each file before it starts moving? Or why did it not move a single file for over a hour when running with lots of files needing to be moved vs. starting to move files after only a few seconds when the cache is not very full?

 

There could be a couple reasons this is happening, it sounded very similar to an issue posted a week or so ago but I have checked and it is not quite the same.

It seems as though it is encountering an non-numerical value and thus cannot perform the numerical operation. While we could simply put in some code to handle and ignore these cases it would be beneficial to understand where it is coming from. If you are still occasionally receiving the error, could you please turn on mover logging and test mode and then provide the latest file located at

"/tmp/Mover/Custom_Mover_Tuning_[TIMESTAMP].list"

If you don't know how to enable these settings:

Spoiler

1. Go to Settings>Scheduler>Mover Settings
2. Set "Mover Logging" to Enabled.
3. Go to the Mover Tunning settings and set "Test Mode" to "Yes"
4. Set "Move Now button follows plug-in filters" to "Yes".
5. On the Main tab press the "Move" button to run the mover. Running in test mode will generate logs/lists but not make actual moves.
6. Go to "/tmp/Mover" and get the latest version of the requested file.
7. You can revert test mode and logging after this is done.

 

[Sptz87]

Hi all, just a quick clarification.

 

When using the "move files in cache at X percentage" basically means that there's no daemon or anything monitoring when the cache drive reaches X percentage right? It means that when the mover runs periodically at what is set on the default mover schedule, that it will only actually move IF it's over X percentage right?

[wgstarks]

19 minutes ago, Sptz87 said:

Hi all, just a quick clarification.

 

When using the "move files in cache at X percentage" basically means that there's no daemon or anything monitoring when the cache drive reaches X percentage right? It means that when the mover runs periodically at what is set on the default mover schedule, that it will only actually move IF it's over X percentage right?

Correct. If you have the schedule set at hourly and the percentage at 75% then once an hour the mover will check to see if the cache is above 75% and start moving if it is.

[g.strange42]

I've just had to remove this, it's killing the mover transfer speed, from MB/s to Kb/s. It only seems to have started doing this since Unraid 6.12.8. 

Are there any known issues around this, and updates on the horizon?
 

[hugenbdd]

14 minutes ago, g.strange42 said:

I've just had to remove this, it's killing the mover transfer speed, from MB/s to Kb/s. It only seems to have started doing this since Unraid 6.12.8. 

Are there any known issues around this, and updates on the horizon?
 

I just installed 6.12.8 this morning.  I will keep an eye on my transfer speeds.

[g.strange42]

39 minutes ago, g.strange42 said:

I've just had to remove this, it's killing the mover transfer speed, from MB/s to Kb/s. It only seems to have started doing this since Unraid 6.12.8. 

Are there any known issues around this, and updates on the horizon?
 

Just to give more Info, I tried reinstalling it twice, and stopping all the dockers that use the cache.