Terebi Posted April 1 Author Share Posted April 1 8 minutes ago, danimal86 said: Thanks for linking to this from the movertuning page. Looks exactly what i am looking for. So for the simple minded like me: Copy the scrpt to notepad++, change the "MAX_SIZE" to 3.5tb (in bytes, i have a 4tb cache), save as "moverignore.sv" and place it \appdata\scripts. Do i need to change anything in TARGET_DIR or OUTPUT_DIR? I created moverignore.txt and placed it in \appdata\scripts In mover tuning path to moverignore.sv in "Script to run before mover (No checks, always runs):" and moverignore.txt for the "Ignore files listed inside of a text file:" .sh not .sv you don't need to create moverignore.txt, it will be created by the script. you only need to change target dir if your media share/directory name is different. Quote Link to comment
danimal86 Posted April 1 Share Posted April 1 49 minutes ago, Terebi said: .sh not .sv you don't need to create moverignore.txt, it will be created by the script. you only need to change target dir if your media share/directory name is different. oops...good catch. I changed it to SH. Looks like our directories are the same (trash's guide?) Alright, i've got everything setup: Ignore files listed inside of a text file: /mnt/user/appdata/scripts/moverignore.txt Script to run before mover (No checks, always runs): /mnt/user/appdata/scripts/moverignore.sh I've got the rest of the mover tuning settings for moving after a certain amount of days, should i change anything? Quote Link to comment
Terebi Posted April 1 Author Share Posted April 1 (edited) you don't need to use the age or size threshholds in mover tuner anymore unless you want them for reasons outside my script. If you run mover every day, it will run my script, and as long as you are below the script threshhold, nothing will be moved. If you are above the threshhold, oldest files will be moved off to get you under the threshhold. Edited April 1 by Terebi Quote Link to comment
danimal86 Posted April 1 Share Posted April 1 Ok, i've got mover set to run hourly and i've got about 500gb of space before i hit the 3.5tb i set in the script. Once the mover runs i should see the moverignore.txt file appear in /mnt/user/appdata/scripts? Quote Link to comment
Terebi Posted April 1 Author Share Posted April 1 3 minutes ago, danimal86 said: Ok, i've got mover set to run hourly and i've got about 500gb of space before i hit the 3.5tb i set in the script. Once the mover runs i should see the moverignore.txt file appear in /mnt/user/appdata/scripts? If thats what the output directory is set to, yes Quote Link to comment
danimal86 Posted April 1 Share Posted April 1 6 minutes ago, Terebi said: If thats what the output directory is set to, yes Thanks for pointing that out. I had the output as /mnt/user/appdata and i added /script to the end since thats where i keep my upgradinatorr script. I'll be excited to see how it works Quote Link to comment
Terebi Posted April 1 Author Share Posted April 1 Just now, danimal86 said: Thanks for pointing that out. I had the output as /mnt/user/appdata and i added /script to the end since thats where i keep my upgradinatorr script. I'll be excited to see how it works you can run the moverignore script manually before the mover runs, to generate the file and make sure it appears as you expect. Instructions for doing so are in the first post. Quote Link to comment
danimal86 Posted April 2 Share Posted April 2 17 hours ago, Terebi said: you can run the moverignore script manually before the mover runs, to generate the file and make sure it appears as you expect. Instructions for doing so are in the first post. Should the script create the text file even if its not hit the MAX_SIZE threshold? sorry for the dumb question, do i need a "/" at the end of the OUTPUT_DIR? Quote Link to comment
Terebi Posted April 2 Author Share Posted April 2 1 minute ago, danimal86 said: Should the script create the text file even if its not hit the MAX_SIZE threshold? sorry for the dumb question, do i need a "/" at the end of the OUTPUT_DIR? You do not need an end slash in the output dir. You can see in the next line (output file) that a slash is added there between the directory and filename. The file should be created every time mover runs. Otherwise the mover would move anything not listed in the file. You can run the moverignore script manually to make sure the file gets generated by going into the directory in a shell in the sh file is in, then running "bash moverignore.sh" . If you still have the movertuning threshholds in place for size and age, then mover may not be running at all, which would mean my script would also not run. Quote Link to comment
danimal86 Posted April 2 Share Posted April 2 11 minutes ago, Terebi said: You do not need an end slash in the output dir. You can see in the next line (output file) that a slash is added there between the directory and filename. The file should be created every time mover runs. Otherwise the mover would move anything not listed in the file. You can run the moverignore script manually to make sure the file gets generated by going into the directory in a shell in the sh file is in, then running "bash moverignore.sh" . If you still have the movertuning threshholds in place for size and age, then mover may not be running at all, which would mean my script would also not run. Ok, bare with me....here's what i got. Command line work is like a foreign language to me, actually surprised i got this far, but learning every time. Quote Link to comment
Terebi Posted April 2 Author Share Posted April 2 18 minutes ago, danimal86 said: Ok, bare with me....here's what i got. Command line work is like a foreign language to me, actually surprised i got this far, but learning every time. It looks like you saved the file with Windows line endings. (\n\r). It needs to have linux endings (\n). Some more advanced text editors (notepad++) can convert line endings for you. Or there are various linux command lines you could google to do the replacements. Quote Link to comment
danimal86 Posted April 2 Share Posted April 2 3 minutes ago, Terebi said: It looks like you saved the file with Windows line endings. (\n\r). It needs to have linux endings (\n). Some more advanced text editors (notepad++) can convert line endings for you. Or there are various linux command lines you could google to do the replacements. Interesting. i used notepad++ to create the file......I'll recreate it again from scratch and try it. Quote Link to comment
Terebi Posted April 2 Author Share Posted April 2 Just now, danimal86 said: Interesting. i used notepad++ to create the file......I'll recreate it again from scratch and try it. You don't need to recreate the file. In notepad++ you can see the line endings (see the LFs in my file) You can then convert using notepad++ too in the edit menu. 1 Quote Link to comment
danimal86 Posted April 2 Share Posted April 2 4 minutes ago, Terebi said: You don't need to recreate the file. In notepad++ you can see the line endings (see the LFs in my file) You can then convert using notepad++ too in the edit menu. Thanks for that. I would not have figured that out on my own! 🙂 Holy moly. It just processed a crap ton of files. I had to run it twice because apparently i cant spell "scripts" correctly I'm pretty shocked how much i have on the cache after looking at the txt file..... Quote Link to comment
Terebi Posted April 2 Author Share Posted April 2 3 minutes ago, danimal86 said: Thanks for that. I would not have figured that out on my own! 🙂 Holy moly. It just processed a crap ton of files. I had to run it twice because apparently i cant spell "scripts" correctly I'm pretty shocked how much i have on the cache after looking at the txt file..... Yeah, the purpose of the script is so you can ignore this and it self manages based on time. But I admit I do manually move things off of cache that I know won't get watched soon/frequently to make room for other things to stay on cache longer Quote Link to comment
danimal86 Posted April 2 Share Posted April 2 5 minutes ago, Terebi said: Yeah, the purpose of the script is so you can ignore this and it self manages based on time. But I admit I do manually move things off of cache that I know won't get watched soon/frequently to make room for other things to stay on cache longer So if you were to manually delete an item from the txt file then the mover wouldn't ignore it and it would get moved off the cache? Just so i know how it works. Once the cache hits the MAX_SIZE then it starts deleting items off the list starting at the bottom and then the mover does not ignore those anymore and moves them off the cache to the array? Quote Link to comment
Terebi Posted April 2 Author Share Posted April 2 The file will get recreated from scratch every time mover runs, so you can't really edit it to control what happens. In the first post I give a command that lets you manually move a file/directory immediately. It doesn't "delete" oldest files really, because it recreates the file from scratch every time. But that is the effect. Files will be added newest to oldest until the size is reached, and then the oldest files won't be added to the file again. Then the mover will run, and since those oldest files aren't listed, they will be moved. 1 Quote Link to comment
Spritzup Posted April 15 Share Posted April 15 Thanks for this! I created a hybrid of @Terebi and @ronia script, and then added an additional variable to append the contents of a static ignore file to dynamically generated one. Why? Just to keep everything contained in one area. To that end, @ronia, I'm using multiple directories as per your script. While it works, it will fail out at step 1 if the directories don't exist. If I have some free time, I may look at adding a check in there so that if say one of three directories is missing, it will continue on... or create the missing directory. ~Spritz 1 Quote Link to comment
Unraidmule Posted April 19 Share Posted April 19 If torrents are actively being seeded, can you safely run this script? Or do they need to be paused, moved and resumed? Quote Link to comment
Terebi Posted April 19 Author Share Posted April 19 (edited) 26 minutes ago, Unraidmule said: If torrents are actively being seeded, can you safely run this script? Or do they need to be paused, moved and resumed? My script itself is safe to run if something is seeding. But the regular problems the mover has if something is seeding are still applicable. My script helps with this problem, because it will tend to extend how long a file lives on the cache, which will reduce the frequency of a torrent being actively seeded by the time the files get moved to array. The biggest problem is if you have the torrents hardlinked (from radarr/sonarr), because if that file is one that should be moved based on time/threshhold the "plex" copy will be able to be moved, but the torrent copy might not move it the file is in use. That will break the hardlink and cause space duplication. If you aren't using hardlinks, then it probalby doesn't matter, because if the torrent fails to move it will just retry it the next day. Because of this, its probably safest to continue to pause/resume torrents as part of the mover process, even though it may work ok 9/10 times. You may need to modify my script to add in the code to pause torrents, because I think the tuner setting used to enable my script is the same setting used to pause torrents. Edited April 19 by Terebi Quote Link to comment
Unraidmule Posted April 19 Share Posted April 19 (edited) I only set up the cache drive today — before downloads were going to array directly, so I really don't have a method to pause torrents yet. I will have to look into this as I never really pause torrents and keep them seeded permanently, and I don't really know how to modify the script, unfortunately. I tried to run the script manually with two files inside the cache and got this result: The data dir exists on the cache now IDK why it can't find it and returns that it does not exist. Also, I use hard links — this is how I permanently seed all torrents. Using a cache drive will cause a big issue? Edited April 19 by Unraidmule Added image Quote Link to comment
Unraidmule Posted April 20 Share Posted April 20 I fixed the line endings, I realized that was causing the error of the script executing. I have 3 different clients, so I am still figuring out how to get them to pause the torrents which only need to be moved and then resumed after the move. Quote Link to comment
Unraidmule Posted April 22 Share Posted April 22 (edited) I’ve been running this for three days but it’s only keeping the files which are on the txt file when I executed the script manually. Is the text file meant to be created in the script dir or another location? How do I check if the mover tuner plug-in has permissions to execute the ignore script? EDIT: I used chmod +x command then tested mover manually, and I think now the .txt file is being created. Is this the correct command, or is there a safer way, so only root can execute the script? Edited April 22 by Unraidmule Quote Link to comment
Terebi Posted April 22 Author Share Posted April 22 10 hours ago, Unraidmule said: I’ve been running this for three days but it’s only keeping the files which are on the txt file when I executed the script manually. Is the text file meant to be created in the script dir or another location? How do I check if the mover tuner plug-in has permissions to execute the ignore script? EDIT: I used chmod +x command then tested mover manually, and I think now the .txt file is being created. Is this the correct command, or is there a safer way, so only root can execute the script? Security is not really unraids strong suit, almost everything in the system is running as root. The text file will go where ever the output directory is set in the script. You just have to point mover tuners ignore file to the same file/path 1 Quote Link to comment
flyize Posted April 27 Share Posted April 27 (edited) On 3/13/2024 at 7:41 PM, ronia said: Hey thanks for this! This was pretty much exactly what I needed the mover to do with some slight modifications. I've attached my version below in case this is useful for anyone else: #!/bin/bash START_TIME=`date +%s` DATE=`date` # Define variables TARGET_DIRS=("/mnt/cache/plex_lib") OUTPUT_DIR="/dev/shm/cache_mover" OUTPUT_FILE="$OUTPUT_DIR/ignore.txt" MOVE_FILE="$OUTPUT_DIR/moved.log" LOG_FILE="$OUTPUT_DIR/verbose.log" MAX_SIZE="500000000000" # 500 gigabytes in bytes #EXTENSIONS=("mkv" "srt" "mp4" "avi" "rar") VERBOSE=false # Ensure the output directory exists mkdir -p "$OUTPUT_DIR" # Ensure the moved log exists touch $MOVE_FILE # Cleanup previous temporary files rm -f "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_all_files.txt" rm -f $OUTPUT_FILE # MOVE_FILE and LOG_FILE intentionally kept persistent for target_dir in "${TARGET_DIRS[@]}"; do # Step 1: Change directory to the target directory cd "$target_dir" || exit # Step 2: Find files with specified extensions and obtain metadata (loop through extensions) #for ext in "${EXTENSIONS[@]}"; do # find "$(pwd)" -type f -iname "*.$ext" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt" #done # Step 2(alt): Find all files. No filter. find "$(pwd)" -type f -iname "*" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt" done # Step 3: Sort metadata by ctime (second column) in descending order sort -z -k 2,2nr -o "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_metadata.txt" # Step 4: Get the newest files up to the specified size limit total_size=0 move_size=0 processed_inodes=() while IFS= read -r -d $'\0' line; do read -r inode ctime path <<< "$line" # Keep track of all files echo $path >> $OUTPUT_DIR/temp_all_files.txt # Skip if the inode has already been processed if [[ "${processed_inodes[*]}" =~ $inode ]]; then continue fi size=$(stat --printf="%s" "$path") if ((total_size + size <= MAX_SIZE)); then if $VERBOSE; then echo "$DATE: Processing file: $total_size $path" >> $LOG_FILE # Debug information to log fi #echo "$path" >> "$OUTPUT_FILE" # Appending only path and filename to the file total_size=$((total_size + size)) # Mark the current inode as processed processed_inodes+=("$inode") # Step 4a: List hardlinks for the current file #hard_links=$(find "$TARGET_DIR" -type f -samefile "$path") #if [ -n "$hard_links" ]; then # echo "$hard_links" >> "$OUTPUT_FILE" #else # echo $path >> $OUTPUT_FILE #fi # Step 4a(alt): Script does not support hardlinks, but is significantly faster and supports multiple TARGET_DIR echo $path >> $OUTPUT_FILE else if $VERBOSE; then echo "$DATE: Moving file: $move_size $path" >> $LOG_FILE # Debug information to log fi move_size=$((move_size + size)) # Do not add to the move file log if previously added if ! grep -q "$path" $MOVE_FILE ; then echo "$DATE: $path" >> $MOVE_FILE fi continue #break fi done < "$OUTPUT_DIR/temp_metadata.txt" # Step 5: Cleanup temporary files rm "$OUTPUT_DIR/temp_metadata.txt" END_TIME=`date +%s` if $VERBOSE; then echo "$DATE: File list generated and saved to: $OUTPUT_FILE" >> $LOG_FILE echo "$DATE: Execution time: $(($END_TIME - $START_TIME)) seconds." >> $LOG_FILE fi Summary: - Removed hardlinks from step 4a. I will not have hardlinks in my share, and it significantly improves execution time - Removed looping over an extension list. I didn't feel like there was anything in my share that couldn't be moved between cache and array - Added execution time logging - Added a 'temp_all_files.txt" file. This was useful in convincing myself that the ignore.txt was capturing everything on the share and nothing was missed. You can just diff the two files together when the cache is below MAX_SIZE and there should be identical. - Added a verbose logging mode - Added a 'moved.log' for tracking what has been moved between cache/array - Changed TARGET_DIR to TARGET_DIRS to loop over all potential shares. At the moment this is just my plex library, but I have some ideas where else this could be used. - Moved the output directory to /dev/shm. My mover is setup to run on the hour, so this reduces the number of read/writes on my cache drive I finally got around to adding this. I used this version. I have 1.3TB on my 'cache_media' drive and set the script MAX_SIZE at 1000000000000 (1TB). Right now its at 600GB used. It made a list of files to ignore, and has then started moving files anyway. What am I missing here? edit: Oops, I think it was self inflicted. I see that by default using the Move Now button does not use the Mover Tuning settings. Edited April 27 by flyize Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.