Actually, not sure what I've done, but the mover now moves everything off.

2 hours ago, flyize said:

Actually, not sure what I've done, but the mover now moves everything off.

 

Are you sure you have the second setting set to ignore things named in the file?

Yeah, I knew I had forgotten something.

Just wanted to come back and say thank you for this. It's working perfectly!

Have we considered using (60s just a sample timer)

inotifywatch -v -e access -e modify -t 60 -r /mnt/user0

which basically collects statistics of all non-cached (/mnt/user0) files and folders.

 

And then using a script to move the most accessed files to the (unraid) cache (/mnt/user).

 

This would effectively create a "read cache" which could be iterated over and over, for X amount of time, untill the cache is X % full.

 

Please note this records literal disk reads and memory cached content and application cached content is not included.

 

Random example excerpt from the depths of the interwebz...

% inotifywatch -v -e access -e modify -t 60 -r ~/.beagle
Establishing watches...
Setting up watch(es) on /home/rohan/.beagle
OK, /home/rohan/.beagle is now being watched.
Total of 302 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 60 seconds.
total  access  modify  filename
1436   1074    362     /home/rohan/.beagle/Indexes/FileSystemIndex/PrimaryIndex/
1323   1053    270     /home/rohan/.beagle/Indexes/FileSystemIndex/SecondaryIndex/
303    116     187     /home/rohan/.beagle/Indexes/KMailIndex/PrimaryIndex/
261    74      187     /home/rohan/.beagle/TextCache/
206    0       206     /home/rohan/.beagle/Log/
42     0       42      /home/rohan/.beagle/Indexes/FileSystemIndex/Locks/
18     6       12      /home/rohan/.beagle/Indexes/FileSystemIndex/
12     0       12      /home/rohan/.beagle/Indexes/KMailIndex/Locks/
3      0       3       /home/rohan/.beagle/TextCache/54/
3      0       3       /home/rohan/.beagle/TextCache/bc/
3      0       3       /home/rohan/.beagle/TextCache/20/
3      0       3       /home/rohan/.beagle/TextCache/62/
2      2       0       /home/rohan/.beagle/Indexes/KMailIndex/SecondaryIndex/

 

2 hours ago, Samsonight said:

all non-cached (/mnt/user0) files and folders.

 

And then using a script to move the most accessed files to the (unraid) cache (/mnt/user).

/mnt/user is the user shares including all pools. You would have to specify /mnt/poolname, such as /mnt/cache, for the move destination.

 

You definitely don't want to try to move anything from /mnt/user0 to /mnt/user. That would almost certainly result in lost data.

1 hour ago, trurl said:

/mnt/user is the user shares including all pools. You would have to specify /mnt/poolname, such as /mnt/cache, for the move destination.

 

You definitely don't want to try to move anything from /mnt/user0 to /mnt/user. That would almost certainly result in lost data.

 

Afaik. If a share has its primary storage on a Pool (cache) and you write to /mnt/user/ShareX it goes to the Pool (instead of the Array).

If you write to the /mnt/user0 it goes to the Array (disk).

 

If /mnt/user0 isn't the (array) disk-only path and /mnt/user is the top layer then what does Mover do when moving to / from array / pool. However it seems Lime is depricating user0, but equally they have not clearly stated before, nor stating now where this functionality would occur. Basically causing threads and confusions like this to fill the forums.

 

Documentation search for user0 https://docs.unraid.net/search/?q=user0

returns only https://docs.unraid.net/unraid-os/manual/shares/#user-shares

while the actual reference is https://docs.unraid.net/unraid-os/manual/shares/user-shares/

 

Please, since you imply factual knowledge of the absolute state of the FUSE layering, could you explain in detail why/how "That would almost certainly result in lost data". Preferrably citing documentation. Just to avoid obscure fud and misunderstandings like this... and how you would suggest a better result achieved. ❤️

Edited May 22, 20242 yr by Samsonight
clarification seems needed

It certainly might be possible to expand my script to include recently read item into things to preserve on cache, more than just things that were added recently. 

 

For my purposes, recently added and recently viewed overlap a lot though, so it might not be as useful as imagined. 

 

And without more complicated work would not handle promoting things from the array to the cache. 

I deleted the moverignore.txt file, since it was keeping the entries of files I had deleted manually from the cache drive which were no longer present. Instead of recreating the .txt file, it emptied the whole cache drive last night. Is the .txt file always meant to be created before a first run? I don't think the script was able to create it itself.

 

EDIT: I created the moverignore.txt file manually with the command but, seems like it is not being updated any more automatically either via the mover plugin.

Edited May 28, 20242 yr by Unraidmule

On 5/27/2024 at 2:22 AM, Unraidmule said:

I deleted the moverignore.txt file, since it was keeping the entries of files I had deleted manually from the cache drive which were no longer present. Instead of recreating the .txt file, it emptied the whole cache drive last night. Is the .txt file always meant to be created before a first run? I don't think the script was able to create it itself.

 

EDIT: I created the moverignore.txt file manually with the command but, seems like it is not being updated any more automatically either via the mover plugin.

 

Every time my script runs it should create the file.  Perhaps the script is not in the same location the mover tuner setting is looking for it in? Or the permissions on the script may need to be fixed?

6 hours ago, Terebi said:

 

Every time my script runs it should create the file.  Perhaps the script is not in the same location the mover tuner setting is looking for it in? Or the permissions on the script may need to be fixed?

 

I think it is a permission issue — it lost rights to execute, IDK what the original file permission was I did chmod 777 because I forgot about the +x command. Will let it run tonight and hopefully get the desired results, when it was running it was actually really cool.

Just now, Unraidmule said:

 

I think it is a permission issue — it lost rights to execute, IDK what the original file permission was I did chmod 777 because I forgot about the +x command. Will let it run tonight and hopefully get the desired results, when it was running it was actually really cool.

 

in a console if you just run mover, you can see if it triggers my script or not

12 hours ago, Terebi said:

 

in a console if you just run mover, you can see if it triggers my script or not

 

I can test it, what do I type in the console? Just “mover”?

 

EDIT: It executed overnight and the .txt file seems to be updating now. IDK what happened, but the executing permissions got removed somehow. If anyone else has issues, that would be the first thing to check.

Edited May 29, 20242 yr by Unraidmule

On 5/28/2024 at 5:17 PM, Unraidmule said:

 

I can test it, what do I type in the console? Just “mover”?

 

EDIT: It executed overnight and the .txt file seems to be updating now. IDK what happened, but the executing permissions got removed somehow. If anyone else has issues, that would be the first thing to check.

Are you using user-scripts to make it?  Is your script located on your boot device?

I had same problem, found out that unraid wont let you make anything on the boot drive executable.  I made a copy to cache disk and I believe its working right now.

 

Had another issue with some files not being added to the moverignore.txt file, specifically some files with 3 digit inodes.

If it was currently looking at a file with inode 416, and inode 5416 had already been processed and added to the array, it would count it as a match.  Fixed it by changing the line to:

 

if  [[ "${processed_inodes[*]}" =~ (^|[[:space:]])"$inode"($|[[:space:]]) ]]; then

 

Great idea with this, I was looking for exactly this behavior in the Mover Tuning Plugin!

Have you considered allowing us to specify a desired amount of free space to maintain, rather than specifying the size we want to TARGET_DIR to be?

 

Unless I am missing something, it would be pretty easy to calculate the MAX_SIZE from the user specified free space requirement: (total size of cache pool - desired free space) - ((total current size of data in /mnt/cache) - (total current size of data in TARGET_DIR ))

 

 

1 minute ago, crafty35a said:

Great idea with this, I was looking for exactly this behavior in the Mover Tuning Plugin!

Have you considered allowing us to specify a desired amount of free space to maintain, rather than specifying the size we want to TARGET_DIR to be?

 

Unless I am missing something, it would be pretty easy to calculate the MAX_SIZE from the user specified free space requirement: (total size of cache pool - desired free space) - ((total current size of data in /mnt/cache) - (total current size of data in TARGET_DIR ))

 

 

Its probably doable,  but since the two values are mostly equivalent (unless appdata grows suddenly) it doesn't matter enough to me to fix

Hi,

 

for the extensions part i want to keep rar files - but in this case the parts are r01-r99 for eg. so is there a way to add a wild card? like r*? because after r99 its going to for eg s01 - s99 etc

 

and its too long to add all the different numbers in

 

Thanks

On 6/10/2024 at 1:18 PM, crafty35a said:

Great idea with this, I was looking for exactly this behavior in the Mover Tuning Plugin!

Have you considered allowing us to specify a desired amount of free space to maintain, rather than specifying the size we want to TARGET_DIR to be?

 

Unless I am missing something, it would be pretty easy to calculate the MAX_SIZE from the user specified free space requirement: (total size of cache pool - desired free space) - ((total current size of data in /mnt/cache) - (total current size of data in TARGET_DIR ))

 

 

I wanted something similar to this as well and this is what I came up with:

 

#!/bin/bash

START_TIME=`date +%s`
DATE=`date`
# Define variables
CACHE_DRIVE="/mnt/cache"
TARGET_DIRS=("/mnt/cache/plex_lib")
DUPLICATES_LOG="/dev/shm/duplicates.log" # Note, this file must be created by another script that tracks duplicated files (ie. tv_series_to_cache script)
OUTPUT_DIR="/dev/shm/cache_mover"
OUTPUT_FILE="$OUTPUT_DIR/ignore.txt"
OUTPUT_FILE_SHADOW="$OUTPUT_DIR/ignore_shadow.txt" # Shadow file so OUTPUT_FILE is always in valid state
MOVE_FILE="$OUTPUT_DIR/move_queue.log" # Note, shadow file not needed as move file is not used by any process
MOVE_LOG="$OUTPUT_DIR/moved.log" # List of files that are moved if mover is expected to be invoked
LOG_FILE="$OUTPUT_DIR/verbose.log"
TRIM_RATIO_TARGET=50 # Ratio of used space to target after moving/cleaning
MOVER_RATIO_TARGET=80 # Must match the value setup in mover scheduler
#EXTENSIONS=("mkv" "srt" "mp4" "avi" "rar")
VERBOSE=false

### Perform size calculations on cache drive, target directories and trim target ###
cd $CACHE_DRIVE
# Get remaining size of cache
remaining_size=$(df --output=avail $CACHE_DRIVE | awk 'FNR ==2 {print}')
remaining_size=$(( $remaining_size * 1024 ))

# Get cache usage
cache_used=0
target_dirs_used=0
non_target_dirs_used=0
for dir in */; do
    dsize=$(du -bs $dir | awk '{print $1}')
    cache_used=$(($cache_used + $dsize))
    for target_dir in "${TARGET_DIRS[@]}"; do
        full_path=$CACHE_DRIVE/$dir
        if [ $full_path = $target_dir/ ]; then
            target_dirs_used=$((target_dirs_used + $dsize))
        else
            non_target_dirs_used=$((non_target_dirs_used + $dsize))
        fi
    done
    #echo "$dir $dsize"
done

total_size=$(($cache_used + $remaining_size))
#echo "total_size $total_size"


# Get ratio of target_dirs to total size of cache
target_dirs_ratio=$(( 100 * $target_dirs_used / $total_size ))
remaining_size_ratio=$(( 100 * $remaining_size / $total_size ))
max_possible_target_ratio=$(( target_dirs_ratio + remaining_size_ratio ))
cache_used_ratio=$(( 100* $cache_used / $total_size ))
#echo "target_dirs_ratio=$target_dirs_ratio"
if (( max_possible_target_ratio < 100-TRIM_RATIO_TARGET )); then
    # notification
    /usr/local/emhttp/webGui/scripts/notify -i alert -s "cache_mover script failed!" -d "Trim ratio target ($((100-TRIM_RATIO_TARGET))%) is greater than possible free space of monitored targets ($max_possible_target_ratio%).  Ignore file could not be created."
    exit
fi

trim_size=$(( total_size * TRIM_RATIO_TARGET / 100 ))
#echo "TARGET_DIRS needs to be trimmed to $trim_size bytes"

# Ensure the output directory exists
mkdir -p "$OUTPUT_DIR"

# Ensure the moved log exists
touch $MOVE_FILE

# Ensure the duplicates log exists
touch $DUPLICATES_LOG

# Cleanup previous temporary files
rm -f "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_all_files.txt" "$OUTPUT_DIR/temp_deleted.txt"
touch "$OUTPUT_DIR/temp_deleted.txt"
rm -f $OUTPUT_FILE_SHADOW
rm -f $MOVE_FILE
# LOG_FILE intentionally kept persistent

for target_dir in "${TARGET_DIRS[@]}"; do
    # Step 1: Change directory to the target directory
    cd "$target_dir" || exit

    # Step 2: Find files with specified extensions and obtain metadata (loop through extensions)
    #for ext in "${EXTENSIONS[@]}"; do
    #    find "$(pwd)" -type f -iname "*.$ext" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt"
    #done
    # Step 2(alt): Find all files.  No filter.
    find "$(pwd)" -type f -iname "*" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt"
done

# Step 3: Sort metadata by ctime (second column) in descending order
sort -z -k 2,2nr -o "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_metadata.txt"


# Step 4: Get the newest files up to the specified size limit
total_size=$non_target_dirs_used
move_size=0
processed_inodes=()
while IFS= read -r -d $'\0' line; do
    read -r inode ctime path <<< "$line"
    
    # Check if folder was deleted
    deleted=false
    while IFS=':' read -r ln_num; do
        ln_num=$(echo "$ln_num" | xargs)
        
        if [ -z "$ln_num" ]; then
            break
        else
            deleted=true  
        fi
    done <<< "$(grep -n "$(dirname "${path}")" "$OUTPUT_DIR/temp_deleted.txt")"
    if [ "$deleted" = true ]; then
        continue
    fi
    
    # Keep track of all files
    echo $path >> $OUTPUT_DIR/temp_all_files.txt
    
    # Skip if the inode has already been processed
    if [[ "${processed_inodes[*]}" =~ $inode ]]; then
        continue
    fi
    
    size=$(stat --printf="%s" "$path")

    if ((total_size + size <= trim_size)); then
        if $VERBOSE; then
            echo "$DATE: Processing file: $total_size $path" >> $LOG_FILE  # Debug information to log
        fi
        #echo "$path" >> "$OUTPUT_FILE_SHADOW"  # Appending only path and filename to the file
        total_size=$((total_size + size))
        
        # Mark the current inode as processed
        processed_inodes+=("$inode")

        # Step 4a: List hardlinks for the current file
        #hard_links=$(find "$TARGET_DIR" -type f -samefile "$path")
        #if [ -n "$hard_links" ]; then
        #       echo "$hard_links" >> "$OUTPUT_FILE_SHADOW"
        #else
        #   echo $path >> $OUTPUT_FILE_SHADOW
        #fi
        # Step 4a(alt): Script does not support hardlinks, but is significantly faster and supports multiple TARGET_DIR
        echo $path >> $OUTPUT_FILE_SHADOW

    else
        # TODO: I don't actually think the duplicates logic is necessary.  rsync seems to be able to reconcile duplicates when performed manually.  Mover uses rsync so it shouldn't be a problem?
        # Check if file is already duplicated on the array
        #duplicated=false
        #while IFS=':' read -r ln_num array_path cache_path; do
        #    # Trim trailing spaces
        #    ln_num=$(echo "$ln_num" | xargs)
        #    array_path=$(echo "$array_path" | xargs)
        #    cache_path=$(echo "$cache_path" | xargs)
        #    
        #    if [ -z "$ln_num" ]; then
        #        break
        #    else
        #        duplicated=true  
        #    fi
        #    
        #    # Rsync cache version based on crc.  Skip mod-time & size.
        #    rsync -avcP -X "$cache_path"/ "$array_path"/
        #done <<< "$(grep -n "$(dirname "${path}")" $DUPLICATES_LOG)"
        
        #if [ "$duplicated" = true ]; then
        #    # Remove log entry
        #    sed -i "\#$(dirname "${path}")#d" $DUPLICATES_LOG
        #    
        #    size=$(du -b "$(dirname "${path}")" | awk '{print $1}')
        #    
        #    # Delete cache folder
        #    rm -rf "$(dirname "${path}")"
        #    echo "$(dirname "${path}")" >> "$OUTPUT_DIR/temp_deleted.txt"
        #fi
        
        #if $VERBOSE; then
        #    if [ "$duplicated" = true ]; then
        #        echo "$DATE: Removing duplicate folder: $move_size "$(dirname "${path}")"" >> $LOG_FILE  # Debug information to log
        #    else
        #        echo "$DATE: Moving file: $move_size $path" >> $LOG_FILE  # Debug information to log
        #    fi
        #fi
        
        move_size=$((move_size + size))
        
        ## Do not add to the move file log if previously added
        #if ! grep -q "$path" $MOVE_FILE ; then
        #    echo "$DATE: $path" >> $MOVE_FILE
        #fi
        # Add the path to the move file
        echo "$path" >> $MOVE_FILE
        
        continue
        #break
    fi
done < "$OUTPUT_DIR/temp_metadata.txt"

# Step 5: Cleanup temporary files
rm "$OUTPUT_DIR/temp_metadata.txt"

# Step 6: Overwrite the output file with the shadow file.  
rm -f $OUTPUT_FILE
mv $OUTPUT_FILE_SHADOW $OUTPUT_FILE

# Step 7: Update move log if mover is expected to run
if [[ $cache_used_ratio -ge $MOVER_RATIO_TARGET ]]; then
    cat $MOVE_FILE >> $MOVE_LOG
fi

END_TIME=`date +%s`

if $VERBOSE; then
    echo "$DATE: File list generated and saved to: $OUTPUT_FILE" >> $LOG_FILE
    echo "$DATE: Execution time: $(($END_TIME - $START_TIME)) seconds." >> $LOG_FILE
fi

 

Notes:

- ignore the "duplicates" log logic.  I use this in conjunction with an "array to cache" script that polls tautulli for in progress media and sends it back to the cache.  It also probably doesn't work.

- my version does not deal with hard links, unlike OPs.  It's assumed that the media library is hard link free or being taken care of externally.

- I think all you would need to do is cut the portions involving "TRIM_RATIO_TARGET" and "MOVER_RATIO_TARGET" to OPs script.  (his is also cleaner because I'm still working/debugging mine)

Edited July 1, 20242 yr by ronia

Hey @Terebi

I had your script running for a while and it works great.  I just recently doubled my cache drives size, is there anything other than changing the MAX_SIZE= in the moverignore.sh file?  

 

Obviously there isn't any media in the cache drive since i had to move it over to the array to replace the drives.  I'm ok with just filling it up as i go.  

Edited October 2, 20241 yr by danimal86

13 hours ago, danimal86 said:

Hey @Terebi

I had your script running for a while and it works great.  I just recently doubled my cache drives size, is there anything other than changing the MAX_SIZE= in the moverignore.sh file?  

 

Obviously there isn't any media in the cache drive since i had to move it over to the array to replace the drives.  I'm ok with just filling it up as i go.  

No, just changing the size is all you should need to do. 

 

However, I will likely no longer be maintaining this script, since the new unraid 7 version of mover tuning has auto-age threshhold moving available out of the box, which replicates this scripts functionality in a more robust way.  If you have moved to unraid 7, I would suggest using that. 

5 hours ago, Terebi said:

No, just changing the size is all you should need to do. 

 

However, I will likely no longer be maintaining this script, since the new unraid 7 version of mover tuning has auto-age threshhold moving available out of the box, which replicates this scripts functionality in a more robust way.  If you have moved to unraid 7, I would suggest using that. 

Thanks for the reply! 

 

Dang I didn't know that 7 had that.  That's super cool.  Ill probably wait till it's stable and probably a revision or two in before I update so I'll keep your script running in the mean time. 

 

Thanks for all you did on this

On 10/2/2024 at 5:36 PM, Terebi said:

No, just changing the size is all you should need to do. 

 

However, I will likely no longer be maintaining this script, since the new unraid 7 version of mover tuning has auto-age threshhold moving available out of the box, which replicates this scripts functionality in a more robust way.  If you have moved to unraid 7, I would suggest using that. 

Oh that's good news! Your script has been a lifesaver so far! Don't know when the official public release will be but I'll wait till then!

In the spirit this thread I had our friend ChatGPT write us this:

 

You schedule it with Cron to run every 10 minutes. */10 * * * *

https://unraid/Settings/Userscripts

 

And add /mnt/user/appdata/cache-filler/cache-filler-exclusion.txt the Mover settings of files to ignore

https://unraid/Settings/Scheduler

 

The same file can be used by other scripts to move detected content to the cache.

 

#!/bin/bash

# Script to build a persistent exclusion file for Unraid mover script
# Monitors /mnt/user/Media for most accessed files using inotify
# Outputs filenames to /mnt/user/appdata/cache-filler/cache-filler-exclusion.txt
# Only includes files up to 80% of disk space of /mnt/turbo
# Exits after 540 seconds to ensure commits
# Provides real-time status updates and detailed statistics
# Persists all data between runs
# Includes concurrent execution protection with PID check

########################################
# Configuration Variables
########################################

WATCH_DIR="/mnt/user/Media"
EXCLUSION_FILE="/mnt/user/appdata/cache-filler/cache-filler-exclusion.txt"
PERSISTENT_DATA_FILE="/mnt/user/appdata/cache-filler/cache-filler-data.txt"
DISK_LIMIT_PATH="/mnt/turbo"
SCRIPT_RUNTIME=540           # Script runs for 540 seconds, 9 minutes
UPDATE_INTERVAL=120          # Update exclusion file every X seconds
DISK_USAGE_THRESHOLD=80      # Percentage of disk usage allowed (80%) DISK_LIMIT_PATH
LOCKFILE="/tmp/cache-filler.lock"

########################################
# Initialization
########################################

# Initialize associative arrays to store file data
declare -A FILE_ACCESS_COUNT  # Stores access count for files
declare -A FILE_SIZES         # Stores sizes for files
declare -A FILE_LAST_ACCESSED # Stores last accessed times

# Ensure the exclusion file's directory exists
EXCLUSION_DIR=$(dirname "$EXCLUSION_FILE")
mkdir -p "$EXCLUSION_DIR"

# Ensure the persistent data file's directory exists
PERSISTENT_DIR=$(dirname "$PERSISTENT_DATA_FILE")
mkdir -p "$PERSISTENT_DIR"

########################################
# Concurrent Execution Protection
########################################

# Function to check if a process is running
is_running() {
    pid=$1
    if [ -z "$pid" ]; then
        return 1
    fi
    if [ -d "/proc/$pid" ]; then
        return 0
    else
        return 1
    fi
}

# Check if the lock file exists
if [ -e "$LOCKFILE" ]; then
    existing_pid=$(cat "$LOCKFILE")
    if is_running "$existing_pid"; then
        echo "Another instance of the script is already running with PID $existing_pid."
        exit 1
    else
        echo "Found stale lock file with PID $existing_pid. Removing it."
        rm -f "$LOCKFILE"
    fi
fi

# Create a lock file with the current PID
echo $$ > "$LOCKFILE"

# Set trap to remove lock file on exit
trap 'echo "Script terminating..."; rm -f "$LOCKFILE"; update_exclusion_file; save_persistent_data; exit' EXIT SIGINT SIGTERM

########################################
# Function Definitions
########################################

# Function to load persisted data
load_persistent_data() {
    if [ -f "$PERSISTENT_DATA_FILE" ]; then
        while IFS='|' read -r filepath access_count size last_accessed; do
            if [[ -n "$filepath" && -f "$filepath" ]]; then
                normalized_path=$(realpath "$filepath" 2>/dev/null)
                if [[ -n "$normalized_path" ]]; then
                    FILE_ACCESS_COUNT["$normalized_path"]="$access_count"
                    FILE_SIZES["$normalized_path"]="$size"
                    FILE_LAST_ACCESSED["$normalized_path"]="$last_accessed"
                fi
            fi
        done < "$PERSISTENT_DATA_FILE"
        echo "Loaded persisted data with ${#FILE_ACCESS_COUNT[@]} files."
    else
        echo "No persisted data found. Starting fresh."
    fi
}

# Function to save persisted data
save_persistent_data() {
    echo "Saving persisted data..."
    > "$PERSISTENT_DATA_FILE"  # Clear the file
    for file in "${!FILE_ACCESS_COUNT[@]}"; do
        access_count="${FILE_ACCESS_COUNT["$file"]}"
        size="${FILE_SIZES["$file"]}"
        last_accessed="${FILE_LAST_ACCESSED["$file"]}"
        echo "$file|$access_count|$size|$last_accessed" >> "$PERSISTENT_DATA_FILE"
    done
    echo "Persisted data saved with ${#FILE_ACCESS_COUNT[@]} files."
}

# Function to calculate total size of tracked files
calculate_total_size() {
    total_size=0
    for file in "${!FILE_SIZES[@]}"; do
        total_size=$((total_size + FILE_SIZES["$file"]))
    done
}

# Function to update the exclusion file
update_exclusion_file() {
    echo "Updating exclusion file..."
    > "$EXCLUSION_FILE"  # Clear the file
    for file in "${!FILE_ACCESS_COUNT[@]}"; do
        echo "$file" >> "$EXCLUSION_FILE"
    done
    echo "Exclusion file updated with ${#FILE_ACCESS_COUNT[@]} files."
}

# Function to provide real-time status updates
status_update() {
    elapsed_time=$((SECONDS - start_time))
    remaining_time=$((SCRIPT_RUNTIME - elapsed_time))
    echo "----------------------------------------"
    echo "Status Update at $(date +'%Y-%m-%d %H:%M:%S'):"
    echo "Time Elapsed: ${elapsed_time}s, Remaining: ${remaining_time}s"
    disk_usage=$(df -h "$DISK_LIMIT_PATH" | awk 'NR==2 {print $5}')
    echo "Disk Usage of $DISK_LIMIT_PATH: $disk_usage"
    echo "Total Disk Size: $(df -h "$DISK_LIMIT_PATH" | awk 'NR==2 {print $2}')"
    echo "Available Disk Space: $(df -h "$DISK_LIMIT_PATH" | awk 'NR==2 {print $4}')"
    echo "Files Tracked: ${#FILE_ACCESS_COUNT[@]}"
    echo "Total Size of Tracked Files: $(numfmt --to=iec "$total_size")"

    # Display top 5 most accessed files
    echo "Top 5 Most Accessed Files:"
    printf "%-10s %-50s %-10s\n" "Count" "Filename" "Size"

    # Generate and process data using awk
    for f in "${!FILE_ACCESS_COUNT[@]}"; do
        echo "${FILE_ACCESS_COUNT["$f"]}|$f|${FILE_SIZES["$f"]}"
    done | sort -t'|' -k1,1nr 2>/dev/null | awk -F'|' 'NR<=5 {
        count = $1
        file = $2
        size = $3
        n = split(file, a, "/")
        filename = a[n]
        # Convert size to human-readable format
        cmd = "numfmt --to=iec " size
        cmd | getline size_human
        close(cmd)
        printf "%-10s %-50s %-10s\n", count, filename, size_human
    }'

    echo "----------------------------------------"
}


########################################
# Main Script Execution
########################################

# Load persisted data at script start
load_persistent_data

echo "Starting monitoring of $WATCH_DIR for ACCESS events."

# Record script start time
start_time=$SECONDS

# Initialize last update time
last_update_time=$SECONDS

# Start inotifywait and process events using process substitution
while read -r file; do

    # Check if script runtime has been exceeded
    elapsed_time=$((SECONDS - start_time))
    if (( elapsed_time >= SCRIPT_RUNTIME )); then
        echo "Script runtime of $SCRIPT_RUNTIME seconds reached."
        break  # Exit the loop gracefully
    fi

    # Normalize the file path
    real_file=$(realpath "$file" 2>/dev/null)
    if [[ -z "$real_file" || ! -f "$real_file" ]]; then
        continue
    fi

    # Get the filename without the path for user output
    file_name=$(basename "$real_file")

    # Update access count and file size without triggering ACCESS events
    if [[ -z "${FILE_ACCESS_COUNT["$real_file"]}" ]]; then
        FILE_ACCESS_COUNT["$real_file"]=1
        FILE_SIZES["$real_file"]=$(stat -c%s "$real_file" 2>/dev/null)
        FILE_LAST_ACCESSED["$real_file"]=$(date +"%Y-%m-%d %H:%M:%S")
        echo "New file accessed: $file_name (Access Count: ${FILE_ACCESS_COUNT["$real_file"]})"
    else
        FILE_ACCESS_COUNT["$real_file"]=$((FILE_ACCESS_COUNT["$real_file"] + 1))
        FILE_LAST_ACCESSED["$real_file"]=$(date +"%Y-%m-%d %H:%M:%S")
        # echo "File accessed: $file_name (Access Count: ${FILE_ACCESS_COUNT["$real_file"]})"
    fi

    # Update exclusion file and status every UPDATE_INTERVAL seconds
    if (( SECONDS - last_update_time >= UPDATE_INTERVAL )); then

        # Calculate total size of tracked files
        calculate_total_size

        # Get total disk size and calculate maximum allowed size
        total_disk_size=$(df --output=size -k "$DISK_LIMIT_PATH" | awk 'NR==2 {print $1 * 1024}')
        max_allowed_size=$((total_disk_size * DISK_USAGE_THRESHOLD / 100))

        # Remove least accessed files if total size exceeds limit
        while [ "$total_size" -gt "$max_allowed_size" ]; do
            # Find the least accessed file
            least_accessed_file=$(printf "%s\n" "${!FILE_ACCESS_COUNT[@]}" | \
            while read -r f; do
                echo "${FILE_ACCESS_COUNT["$f"]}|$f"
            done | sort -n | head -n1 | awk -F'|' '{print $2}')

            # Get the filename without the path for user output
            least_accessed_file_name=$(basename "$least_accessed_file")
            echo "Removing file due to size limit: $least_accessed_file_name"

            # Remove file from tracking
            unset "FILE_ACCESS_COUNT[$least_accessed_file]"
            unset "FILE_SIZES[$least_accessed_file]"
            unset "FILE_LAST_ACCESSED[$least_accessed_file]"

            # Recalculate total size
            calculate_total_size
        done

        # Update exclusion file and status
        update_exclusion_file
        status_update

        # Update last update time
        last_update_time=$SECONDS
    fi

done < <(inotifywait -m -r -e access --format '%w%f' "$WATCH_DIR" 2>/dev/null)

# Save persisted data before exiting (in case trap did not catch)
save_persistent_data

echo "Script execution completed."

comes with a "Ai" logo too

Edited October 7, 20241 yr by Samsonight
typo

On 10/2/2024 at 9:36 AM, Terebi said:

No, just changing the size is all you should need to do. 

 

However, I will likely no longer be maintaining this script, since the new unraid 7 version of mover tuning has auto-age threshhold moving available out of the box, which replicates this scripts functionality in a more robust way.  If you have moved to unraid 7, I would suggest using that. 

So i just made the jump to Unraid 7, i'm having a hard time finding the Auto-Age Threshold moving in Mover Tuning plugin.  Maybe i'm looking past it.  

 

I see there's a Mover Tuning from Squid and Reynald's.  Not sure which one or both have it. 

 

Edit: So on Reynalds, i'm trying this settings which seem to work

Edited January 18, 20251 yr by danimal86

On 1/12/2025 at 5:16 PM, danimal86 said:

So i just made the jump to Unraid 7, i'm having a hard time finding the Auto-Age Threshold moving in Mover Tuning plugin.  Maybe i'm looking past it.  

 

I see there's a Mover Tuning from Squid and Reynald's.  Not sure which one or both have it. 

 

Edit: So on Reynalds, i'm trying this settings which seem to work

Where are you finding these settings? I think I have the right Mover Tuner plugin installed, but don't see this.

 

edit: Oh, it seems I had to uninstall/reinstall the plugin. Weird.

Edited January 22, 20251 yr by flyize