[Guide] How to keep cache drive full of media

[Terebi]

8 minutes ago, danimal86 said:

Thanks for linking to this from the movertuning page.  Looks exactly what i am looking for. 

 

So for the simple minded like me:

Copy the scrpt to notepad++, change the "MAX_SIZE" to 3.5tb (in bytes, i have a 4tb cache), save as "moverignore.sv" and place it \appdata\scripts.  Do i need to change anything in TARGET_DIR or OUTPUT_DIR?  I created moverignore.txt and placed it in \appdata\scripts

In mover tuning path to moverignore.sv in "Script to run before mover (No checks, always runs):"  and moverignore.txt for the "Ignore files listed inside of a text file:"

 

.sh not .sv

 

you don't need to create moverignore.txt, it will be created by the script.  you only need to change target dir if your media share/directory name is different.  

[danimal86]

49 minutes ago, Terebi said:

 

.sh not .sv

 

you don't need to create moverignore.txt, it will be created by the script.  you only need to change target dir if your media share/directory name is different.  

oops...good catch.  I changed it to SH.  Looks like our directories are the same (trash's guide?)

 

Alright, i've got everything setup:

Ignore files listed inside of a text file: /mnt/user/appdata/scripts/moverignore.txt

Script to run before mover (No checks, always runs): /mnt/user/appdata/scripts/moverignore.sh

 

I've got the rest of the mover tuning settings for moving after a certain amount of days, should i change anything?

[Terebi]

you don't need to use the age or size threshholds in mover tuner anymore unless you want them for reasons outside my script.  If you run mover every day, it will run my script, and as long as you are below the script threshhold, nothing will be moved.  If you are above the threshhold, oldest files will be moved off to get you under the threshhold. 

 

 

Edited April 1 by Terebi

[danimal86]

Ok, i've got mover set to run hourly and i've got about 500gb of space before i hit the 3.5tb i set in the script. 

Once the mover runs i should see the moverignore.txt file appear in /mnt/user/appdata/scripts?

[Terebi]

3 minutes ago, danimal86 said:

Ok, i've got mover set to run hourly and i've got about 500gb of space before i hit the 3.5tb i set in the script. 

Once the mover runs i should see the moverignore.txt file appear in /mnt/user/appdata/scripts?

 

If thats what the output directory is set to, yes

[danimal86]

6 minutes ago, Terebi said:

 

If thats what the output directory is set to, yes

Thanks for pointing that out.  I had the output as /mnt/user/appdata and i added /script to the end since thats where i keep my upgradinatorr script.  

I'll be excited to see how it works

[Terebi]

Just now, danimal86 said:

Thanks for pointing that out.  I had the output as /mnt/user/appdata and i added /script to the end since thats where i keep my upgradinatorr script.  

I'll be excited to see how it works

 

you can run the moverignore script manually before the mover runs, to generate the file and make sure it appears as you expect.  Instructions for doing so are in the first post. 

[danimal86]

17 hours ago, Terebi said:

 

you can run the moverignore script manually before the mover runs, to generate the file and make sure it appears as you expect.  Instructions for doing so are in the first post. 

Should the script create the text file even if its not hit the MAX_SIZE threshold?  

 

sorry for the dumb question, do i need a "/" at the end of the OUTPUT_DIR?

[Terebi]

1 minute ago, danimal86 said:

Should the script create the text file even if its not hit the MAX_SIZE threshold?  

 

sorry for the dumb question, do i need a "/" at the end of the OUTPUT_DIR?

 

You do not need an end slash in the output dir. You can see in the next line (output file) that a slash is added there between the directory and filename. 

 

The file should be created every time mover runs.  Otherwise the mover would move anything not listed in the file.  You can run the moverignore script manually to make sure the file gets generated by going into the directory in a shell in the sh file is in, then running "bash moverignore.sh" .   If you still have the movertuning threshholds in place for size and age, then mover may not be running at all, which would mean my script would also not run. 

[danimal86]

11 minutes ago, Terebi said:

 

You do not need an end slash in the output dir. You can see in the next line (output file) that a slash is added there between the directory and filename. 

 

The file should be created every time mover runs.  Otherwise the mover would move anything not listed in the file.  You can run the moverignore script manually to make sure the file gets generated by going into the directory in a shell in the sh file is in, then running "bash moverignore.sh" .   If you still have the movertuning threshholds in place for size and age, then mover may not be running at all, which would mean my script would also not run. 

Ok, bare with me....here's what i got.  Command line work is like a foreign language to me, actually surprised i got this far, but learning every time. 

[Terebi]

18 minutes ago, danimal86 said:

Ok, bare with me....here's what i got.  Command line work is like a foreign language to me, actually surprised i got this far, but learning every time. 

 

It looks like you saved the file with Windows line endings.  (\n\r). It needs to have linux endings (\n). 

 

Some more advanced text editors (notepad++) can convert line endings for you.  Or there are various linux command lines you could google to do the replacements. 

[danimal86]

3 minutes ago, Terebi said:

 

It looks like you saved the file with Windows line endings.  (\n\r). It needs to have linux endings (\n). 

 

Some more advanced text editors (notepad++) can convert line endings for you.  Or there are various linux command lines you could google to do the replacements. 

Interesting.  i used notepad++ to create the file......I'll recreate it again from scratch and try it.  

[Terebi]

Just now, danimal86 said:

Interesting.  i used notepad++ to create the file......I'll recreate it again from scratch and try it.  

 

You don't need to recreate the file.  In notepad++ you can see the line endings (see the LFs in my file)

 

 

You can then convert using notepad++ too in the edit menu.

  

[danimal86]

4 minutes ago, Terebi said:

 

You don't need to recreate the file.  In notepad++ you can see the line endings (see the LFs in my file)

 

 

You can then convert using notepad++ too in the edit menu.

  

Thanks for that.  I would not have figured that out on my own! 🙂
 

Holy moly.  It just processed a crap ton of files.  I had to run it twice because apparently i cant spell "scripts" correctly 

 

I'm pretty shocked how much i have on the cache after looking at the txt file.....

 

[Terebi]

3 minutes ago, danimal86 said:

Thanks for that.  I would not have figured that out on my own! 🙂
 

Holy moly.  It just processed a crap ton of files.  I had to run it twice because apparently i cant spell "scripts" correctly 

I'm pretty shocked how much i have on the cache after looking at the txt file.....

 

 

Yeah, the purpose of the script is so you can ignore this and it self manages based on time.  But I admit I do manually move things off of cache that I know won't get watched soon/frequently to make room for other things to stay on cache longer

[danimal86]

5 minutes ago, Terebi said:

 

Yeah, the purpose of the script is so you can ignore this and it self manages based on time.  But I admit I do manually move things off of cache that I know won't get watched soon/frequently to make room for other things to stay on cache longer

So if you were to manually delete an item from the txt file then the mover wouldn't ignore it and it would get moved off the cache?

 

Just so i know how it works.  Once the cache hits the MAX_SIZE then it starts deleting items off the list starting at the bottom and then the mover does not ignore those anymore and moves them off the cache to the array?

[Terebi]

The file will get recreated from scratch every time mover runs, so you can't really edit it to control what happens. In the first post I give a command that lets you manually move a file/directory immediately. 

 

It doesn't "delete" oldest files really, because it recreates the file from scratch every time. But that is the effect. Files will be added newest to oldest until the size is reached, and then the oldest files won't be added to the file again.  Then the mover will run, and since those oldest files aren't listed, they will be moved. 

[Spritzup]

Thanks for this! I created a hybrid of @Terebi and @ronia script, and then added an additional variable to append the contents of a static ignore file to dynamically generated one. Why? Just to keep everything contained in one area.

 

To that end, @ronia, I'm using multiple directories as per your script. While it works, it will fail out at step 1 if the directories don't exist. If I have some free time, I may look at adding a check in there so that if say one of three directories is missing, it will continue on... or create the missing directory.

 

~Spritz

[Unraidmule]

If torrents are actively being seeded, can you safely run this script? Or do they need to be paused, moved and resumed?

[Terebi]

26 minutes ago, Unraidmule said:

If torrents are actively being seeded, can you safely run this script? Or do they need to be paused, moved and resumed?

 

My script itself is safe to run if something is seeding. But the regular problems the mover has if something is seeding are still applicable. 

 

My script helps with this problem, because it will tend to extend how long a file lives on the cache, which will reduce the frequency of a torrent being actively seeded  by the time the files get moved to array. 

 

The biggest problem is if you have the torrents hardlinked (from radarr/sonarr), because if that file is one that should be moved based on time/threshhold the "plex" copy will be able to be moved, but the torrent copy might not move it the file is in use.  That will break the hardlink and cause space duplication. 

 

If you aren't using hardlinks, then it probalby doesn't matter, because if the torrent fails to move it will just retry it the next day. 

 

Because of this, its probably safest to continue to pause/resume torrents as part of the mover process, even though it may work ok 9/10 times. 

 

You may need to modify my script to add in the code to pause torrents, because I think the tuner setting used to enable my script is the same setting used to pause torrents. 

Edited April 19 by Terebi

[Unraidmule]

I only set up the cache drive today — before downloads were going to array directly, so I really don't have a method to pause torrents yet.

 

I will have to look into this as I never really pause torrents and keep them seeded permanently, and I don't really know how to modify the script, unfortunately.

 

I tried to run the script manually with two files inside the cache and got this result:

 

 

The data dir exists on the cache now IDK why it can't find it and returns that it does not exist.

 

Also, I use hard links — this is how I permanently seed all torrents. Using a cache drive will cause a big issue?

Edited April 19 by Unraidmule
Added image

[Unraidmule]

I fixed the line endings, I realized that was causing the error of the script executing.

 

I have 3 different clients, so I am still figuring out how to get them to pause the torrents which only need to be moved and then resumed after the move.

[Unraidmule]

I’ve been running this for three days but it’s only keeping the files which are on the txt file when I executed the script manually. Is the text file meant to be created in the script dir or another location?

 

How do I check if the mover tuner plug-in has permissions to execute the ignore script?

 

EDIT:

 

I used chmod +x command then tested mover manually, and I think now the .txt file is being created. Is this the correct command, or is there a safer way, so only root can execute the script?

Edited April 22 by Unraidmule

[Terebi]

10 hours ago, Unraidmule said:

I’ve been running this for three days but it’s only keeping the files which are on the txt file when I executed the script manually. Is the text file meant to be created in the script dir or another location?

 

How do I check if the mover tuner plug-in has permissions to execute the ignore script?

 

EDIT:

 

I used chmod +x command then tested mover manually, and I think now the .txt file is being created. Is this the correct command, or is there a safer way, so only root can execute the script?

 

Security is not really unraids strong suit, almost everything in the system is running as root.  The text file will go where ever the output directory is set in the script. You just have to point mover tuners ignore file to the same file/path

[flyize]

On 3/13/2024 at 7:41 PM, ronia said:

Hey thanks for this!  This was pretty much exactly what I needed the mover to do with some slight modifications.  I've attached my version below in case this is useful for anyone else:

 

#!/bin/bash

START_TIME=`date +%s`
DATE=`date`
# Define variables
TARGET_DIRS=("/mnt/cache/plex_lib")
OUTPUT_DIR="/dev/shm/cache_mover"
OUTPUT_FILE="$OUTPUT_DIR/ignore.txt"
MOVE_FILE="$OUTPUT_DIR/moved.log"
LOG_FILE="$OUTPUT_DIR/verbose.log"
MAX_SIZE="500000000000"  # 500 gigabytes in bytes
#EXTENSIONS=("mkv" "srt" "mp4" "avi" "rar")
VERBOSE=false

# Ensure the output directory exists
mkdir -p "$OUTPUT_DIR"

# Ensure the moved log exists
touch $MOVE_FILE

# Cleanup previous temporary files
rm -f "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_all_files.txt"
rm -f $OUTPUT_FILE
# MOVE_FILE and LOG_FILE intentionally kept persistent

for target_dir in "${TARGET_DIRS[@]}"; do
    # Step 1: Change directory to the target directory
    cd "$target_dir" || exit

    # Step 2: Find files with specified extensions and obtain metadata (loop through extensions)
    #for ext in "${EXTENSIONS[@]}"; do
    #    find "$(pwd)" -type f -iname "*.$ext" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt"
    #done
    # Step 2(alt): Find all files.  No filter.
    find "$(pwd)" -type f -iname "*" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt"
done

# Step 3: Sort metadata by ctime (second column) in descending order
sort -z -k 2,2nr -o "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_metadata.txt"


# Step 4: Get the newest files up to the specified size limit
total_size=0
move_size=0
processed_inodes=()
while IFS= read -r -d $'\0' line; do
    read -r inode ctime path <<< "$line"
    
    # Keep track of all files
    echo $path >> $OUTPUT_DIR/temp_all_files.txt
    
    # Skip if the inode has already been processed
    if [[ "${processed_inodes[*]}" =~ $inode ]]; then
        continue
    fi
    
    size=$(stat --printf="%s" "$path")

    if ((total_size + size <= MAX_SIZE)); then
        if $VERBOSE; then
            echo "$DATE: Processing file: $total_size $path" >> $LOG_FILE  # Debug information to log
        fi
        #echo "$path" >> "$OUTPUT_FILE"  # Appending only path and filename to the file
        total_size=$((total_size + size))
        
        # Mark the current inode as processed
        processed_inodes+=("$inode")

        # Step 4a: List hardlinks for the current file
        #hard_links=$(find "$TARGET_DIR" -type f -samefile "$path")
        #if [ -n "$hard_links" ]; then
        #       echo "$hard_links" >> "$OUTPUT_FILE"
        #else
        #   echo $path >> $OUTPUT_FILE
        #fi
        # Step 4a(alt): Script does not support hardlinks, but is significantly faster and supports multiple TARGET_DIR
        echo $path >> $OUTPUT_FILE

    else
        if $VERBOSE; then
            echo "$DATE: Moving file: $move_size $path" >> $LOG_FILE  # Debug information to log
        fi
        move_size=$((move_size + size))
        
        # Do not add to the move file log if previously added
        if ! grep -q "$path" $MOVE_FILE ; then
            echo "$DATE: $path" >> $MOVE_FILE
        fi
        continue
        #break
    fi
done < "$OUTPUT_DIR/temp_metadata.txt"

# Step 5: Cleanup temporary files
rm "$OUTPUT_DIR/temp_metadata.txt"

END_TIME=`date +%s`

if $VERBOSE; then
    echo "$DATE: File list generated and saved to: $OUTPUT_FILE" >> $LOG_FILE
    echo "$DATE: Execution time: $(($END_TIME - $START_TIME)) seconds." >> $LOG_FILE
fi

Summary:
- Removed hardlinks from step 4a.  I will not have hardlinks in my share, and it significantly improves execution time

- Removed looping over an extension list.  I didn't feel like there was anything in my share that couldn't be moved between cache and array

- Added execution time logging

- Added a 'temp_all_files.txt" file.  This was useful in convincing myself that the ignore.txt was capturing everything on the share and nothing was missed.  You can just diff the two files together when the cache is below MAX_SIZE and there should be identical.

- Added a verbose logging mode

- Added a 'moved.log' for tracking what has been moved between cache/array

- Changed TARGET_DIR to TARGET_DIRS to loop over all potential shares.  At the moment this is just my plex library, but I have some ideas where else this could be used.

- Moved the output directory to /dev/shm.  My mover is setup to run on the hour, so this reduces the number of read/writes on my cache drive

I finally got around to adding this. I used this version. I have 1.3TB on my 'cache_media' drive and set the script MAX_SIZE at 1000000000000 (1TB). Right now its at 600GB used. It made a list of files to ignore, and has then started moving files anyway. What am I missing here?

 

edit: Oops, I think it was self inflicted. I see that by default using the Move Now button does not use the Mover Tuning settings.

Edited Saturday at 07:29 PM by flyize