Reynald Posted May 7, 2020 Posted May 7, 2020 (edited) Hello all, Depcrecation information This script is not maintened anymore. You may find updated version in page 2 from other users. I've started to work again on it but: there is issue with ZFS I'm working on Mover Tuning script, which will include smart moving from array to cache, and is already including the smart cleaning part of this cript (well, renamed "Automatic age" and largely revamped but equivalent in termes of functionality): Background I have a 8 mechanical HDD 40TB array in a Unraid server v6.8.2 DVB. I have 40GB memory, only about 8GB are used. I don't use memory for caching, for now. I have a 2TB SSD mounted as cache and hosting docker appdata and VM domains, and until now I was using Unraid cache system to store only new files from a data share, with a script moving to array when SSD was 90% full. With this method, only latest written file were on the cache, so I rethought the whole thing, see below. I use plex to stream to several devices on LAN (gigabit ethernet) or WAN (gigabit fiber internet), and also seed torrents with transmission. Here is my share setup: So I wanted to dynamically cache file from data share to SSD. Main file consumers are plex and transmission, which have their data in a data share As a fail-safe, I set mover to only move file if cache usage is more than 95%. I wrote a script to handle automagically caching of the data share to use SSD up to 90% (including appdata and VMs). What the script needs: a RPC enabled transmission installation (optional) access to Plex web API (optional) path to a share on cache path to same share on array What the script do: When you start it, it makes basic connection and path checks and then 3 main functions are executed: Cleans selected share on cache to have at least 10% free (configurable). To free space, oldest data are copied back to array then deleted from cache. Retrieves the list of active torrents from transmission-rpc daemon and copy to cache without removing from array. (note: active-torrents are those downloading and seeding during the last minute, but also starting and stopping, that's a caveat if you start/stop a batch of torrent and launch the script in the minute) Retrieves the list of active playing sessions from Plex and copy (rsync, same as mover or unbalance) movies to cache without removing from array. For series, there are options to copy: current and next episode or all episodes from current to the end of season Cleans again Note: to copy, rsync is used, like mover or unbalance, so it syncs data (don't overwrite if existing) in addition, hard-links, if any (from radarr, sonarr, etc...), are recreated on destination (cache when caching, array when cleaning cache) if you manually send file to the share on cache, it will be cleaned when getting old you may write a side script then (for working files, libraries, etc..) Because of shfs mechanism accessing a file from /mnt/user will read/write fro cache if it exists, then from array. Duplicate data are not a problem and globally speed up things. The script is very useful when, like me, you have noisy/slow mechanical HDDs for storage, and quick and quiet SDD to serve files. Script installation: I recommend copy/paste it in a new script created with User Scripts. Script configuration: No parameters are passed to the script, so it's easy to use with User Scripts plugin. To configure, a relevant section is at the beginning of the script, parameters are pretty much self explanatory: Here is a log from execution: Pretty neat hum? Known Previous issues (update may came to fix them later): At the moment, log can become huge if, like me, you run the script every minute. This is the recommended interval because transmission-RPC active torrent list contain only the ones from last minute. Edit 13-02-2020: corrected in latest version At the moment, a orphan file (only on cache) being played or seeded is detected, but not synced to the array until it needs to be cleaned (i.e: fresh torrents, recent movies fetched by *arr and newsgroup, etc...). Edit 13-02-2020: corected in latest version: it sync back to array during set day (noisy) hours. I don't know if/how shfs will handle the file on cache. I need more investigation/testing to see if it efficiently read the file from cache instead of array. I guess transmission/plex need to close and reopen the file to pick it from new location? (my assumption is that both read chunks, so caching shall work). Edit 13-02-2020: yes, after checking with File Activity plugin, that's the case and its plex/transmission take the file on cache as soon as it is available! Conclusion, disclaimer, and link: The script run successfully in my configuration since yesterday. Before using rsync I was using rclone which has a cache back-end, a similar plex caching function, plus a remote (I used it for transmission), but it's not as smooth and quick as rsync. Please note that, even if I use it 1440 times a day (User Scripts, Custom schedule * * * * *), this script is still experimental and can: erase (or most likely fill up) your SSD, Edit 13-02-2020: but I did not experienced this, error handling improved erase (not likely) your array Edit 13-02-2020: Technically, this script never delete anything on array, it won't happen kill your cat (sorry) make your mother in law move to your home and stay (I can do nothing for that) break your server into pieces (you can keep these) Thanks for reading to this point, you deserve to get the link to the script (<- link is here). If you try it or have any comment, idea, recommendation, question, etc..., feel free to reply Take care, Reynald Edited July 11 by Reynald Correct link 2 Quote
Reynald Posted May 13, 2020 Author Posted May 13, 2020 Hello all, I updated the script 2 days ago, it's holding tight ! I have very very few spinup now, because 1.4To of most recent data are duplicated on SSD. Quote - Added max number of torrent/sessions for transmission/plex, preventing caching all torrents at transmission startup/torrent checks - Added syncing new files on cache to storage during day hours - Improved logging - Improved error handling for caching/uncaching function It's on my github: https://bit.ly/Ro11u5-GH_smart-cache Shall I make this a plugin? 1 Quote
Reynald Posted May 14, 2020 Author Posted May 14, 2020 Updated: v0/5/14: - Improvement on verbosity (new settings) - Added parameter CACHE_MAX_FREE_SPACE_PCT="85" in addition to CACHE_MIN_FREE_SPACE_PCT="90" => When cache usage exceed CACHE_MIN_FREE_SPACE_PCT (here 90%), it is freed until CACHE_MAX_FREE_SPACE_PCT is achieved, here 85% Quote
hugenbdd Posted May 14, 2020 Posted May 14, 2020 Nice script! Love the Plex section of it. Question, why are you using rsync instead of the mover provided binary? The binary just needs a file list. To move from cache to disk the file list needs contain the /mnt/cache/<SHARENAME>... and for array to cache /mnt/disk<##>/<SHARENAME> Example call from the mover bash script. # Check for objects to move from array to cache for SHAREPATH in $(ls -dv /mnt/disk[0-9]*/*/) ; do SHARE=$(basename "$SHAREPATH") if grep -qs 'shareUseCache="prefer"' "/boot/config/shares/${SHARE}.cfg" ; then find "${SHAREPATH%/}" -depth | /usr/local/sbin/move -d $LOGLEVEL fi done Quote
Reynald Posted May 14, 2020 Author Posted May 14, 2020 (edited) Hello, Thank you for your interest and warm words @hugenbdd ! This script takes me quiet some hours of thinking/scripting. 2 hours ago, hugenbdd said: Question, why are you using rsync instead of the mover provided binary? I was not aware of mover binary. If I'm not mistaken the /usr/local/sbin/mover.old script where you find your piece of script for your example was invoking rsync in the past. I recall having picked rsync options (-aH) from this mover.old script My strategy is not to move, but to archive-sync in both direction (same as mover), and to delete from cache depending on disk usage, not deleting on array. Some benefits: - File is not overwritten if identical, latest copy is on cache if it exists on cache. --> Moving from cache to array and vice-versa will take more time than duplicating data (mover will also not move, but sync and delete). --> Copying from array to cache let the data secured by parity --> Having control on deletion allows to handle hardlinks (a torrent seeded by transmission from cache is also available for plex). Mover will preserve them also as it move a whole directory, but I'm moving files. --> I can bypass cache "prefer/yes/no/only" directives, and set mover so it won't touch my "data" share until I'm short on space on cache(i.e if this smart-cache script is killed). --> Using rsync with -P parameters while debugging/testing give some status/progress info Drawbacks: - data is duplicated - deletion and modification from array using /mnt/user0 or /mnt/diskN is not synced to /mnt/cache. This is not possibble if we use /mnt/user for the 'data' share. But thanks to your suggestion (with the filelist idea), I have an idea about how to sync cache only files (i.e fresh transmission downloads during quiet hours) to array. Also, mover may do some extra checks from array to cache. From cache to array, I use unraid shfs mechanism as I sync to /mnt/user0 (and not to /mnt/diskN), same for hardlink that are well handled by shfs. If you want to use this script for plex only, you can: - set $TRANSMISSION_ENABLED to false or if you want to clean the script: - remove #Transmission parameters, transmission_cache fonction and its call ('$TRANSMISSION_ENABLED && transmission_cache') at the bottom of the script. I may extend to other torrent client later. Edited May 14, 2020 by Reynald 2 Quote
doobyns Posted June 18, 2022 Posted June 18, 2022 (edited) great script, can you modify your script to add video preloading in ram like in this script (with this thing videos cached by your scipt would start instantaneously): But it would be only for movies and tv shows cached on ssd by your script, with this addition your script will be the ultimate solution for caching and speeding up plex ! e.g. : It would be nice to have minimum parameters : MAX_FREE_RAM_USAGE_PCT (percent of free ram used for preload) VIDEO_MAX_PRELOAD_PCT (would be 1% or 2% by default) Edited June 18, 2022 by doobyns Quote
kizer Posted June 19, 2022 Posted June 19, 2022 I tried using this, but I get a path not found. I created the data share on my system, but it seems to delete it every time I run it. I'm running 6.10.3 Quote
doobyns Posted June 19, 2022 Posted June 19, 2022 18 hours ago, kizer said: I tried using this, but I get a path not found. I created the data share on my system, but it seems to delete it every time I run it. I'm running 6.10.3 can you post here your settings for : STORAGE_PATH CACHE_PATH CACHE_DISK Quote
kizer Posted June 20, 2022 Posted June 20, 2022 STORAGE_PATH="/mnt/user0/TV/Shows/" CACHE_PATH="/mnt/cache/data/" CACHE_DISK="/mnt/cache/data/" I noticed you had CACHE_DISK= /dev/mapper/ and a NVE device. I only have /dev/mapper/control Quote
doobyns Posted June 23, 2022 Posted June 23, 2022 On 6/20/2022 at 8:20 PM, kizer said: STORAGE_PATH="/mnt/user0/TV/Shows/" CACHE_PATH="/mnt/cache/data/" CACHE_DISK="/mnt/cache/data/" I noticed you had CACHE_DISK= /dev/mapper/ and a NVE device. I only have /dev/mapper/control what do you obtain wheh you type this : df -h /mnt/cache/ Quote
kizer Posted June 23, 2022 Posted June 23, 2022 df -h /mnt/cache/ Filesystem Size Used Avail Use% Mounted on /dev/sdh1 932G 153G 779G 20% /mnt/cache Quote
doobyns Posted June 23, 2022 Posted June 23, 2022 52 minutes ago, kizer said: df -h /mnt/cache/ Filesystem Size Used Avail Use% Mounted on /dev/sdh1 932G 153G 779G 20% /mnt/cache Try this : CACHE_DISK="/dev/sdh1" Quote
kizer Posted June 23, 2022 Posted June 23, 2022 Is my Storage Path Correct for Plex to read from? Its the current location of my TV shows when they are temporary before the mover moves them. Meaning I use /mnt/user/TV for my Tv shows and /mnt/user/Movies for movies. I'm not as worried about my Movies as much as my TV shows. I'll give this a try tonight. I'm currently at work and thank you for your time to look at this for me. Quote
JonathanM Posted June 23, 2022 Posted June 23, 2022 1 hour ago, doobyns said: CACHE_DISK="/dev/sdh1" That is not a permanent fix even if it works at the moment, it may not work on subsequent boots, as the sdX designations are subject to change based on many factors out of Unraid's control. Quote
trurl Posted June 24, 2022 Posted June 24, 2022 On 6/20/2022 at 2:20 PM, kizer said: STORAGE_PATH="/mnt/user0/TV/Shows This means the Shows folder in the TV share on the array only Quote
kizer Posted June 24, 2022 Posted June 24, 2022 @JonathanM @trurl Totally understand. I'm trying to peak under the curtain some. I'm trying to figure out what the magic is here since it has me really intrigued but I want to make sure there isn't anything that would cause my system or somebody else's system to go haywire. Also if it does what its supposed to it would put a pretty cool spin on what I have going on at my place. Quote
kizer Posted June 24, 2022 Posted June 24, 2022 Ok newest update. #Rsync path STORAGE_PATH="/mnt/user0/TV/" CACHE_PATH="/mnt/cache/TV/" CACHE_DISK="/dev/sdh1" CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" Delivers: Welcome to /tmp/user.scripts/tmpScripts/Plex-Caching-Script/script stat: cannot statx '/tmp/user.scripts/tmpScripts/Plex-Caching-Script/log.txt': No such file or directory Info: Log size is ---------------------------- 1 active(s) plex session(s): ---------------------------- - 1/1: Serie: Suits Season 9 - Episode 4/-1: Cairo --------------------- Cache disk usage: 16% --------------------- - Info: 16% space used, quota is 90%, nothing to do -- Info: Cleaning empty directories... It just doesn't cache anything thou. Quote
doobyns Posted July 19, 2022 Posted July 19, 2022 Great script, but i've some issues with the cache, eg when the script add a directory on the top level of the cache, a share is automatically created by unraid, when the script add : /data/tv_shows/a_tv_show/tv_show 01x01 Ep1.mkv on the cache, unraid create the share "tv_shows" automatically, my question is : can we add the hardlink in a subdirectory of the cache ? it will be simpler to organise... Quote
doobyns Posted July 23, 2022 Posted July 23, 2022 (edited) On 6/24/2022 at 6:01 AM, kizer said: Ok newest update. #Rsync path STORAGE_PATH="/mnt/user0/TV/" CACHE_PATH="/mnt/cache/TV/" CACHE_DISK="/dev/sdh1" CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" Delivers: Welcome to /tmp/user.scripts/tmpScripts/Plex-Caching-Script/script stat: cannot statx '/tmp/user.scripts/tmpScripts/Plex-Caching-Script/log.txt': No such file or directory Info: Log size is ---------------------------- 1 active(s) plex session(s): ---------------------------- - 1/1: Serie: Suits Season 9 - Episode 4/-1: Cairo --------------------- Cache disk usage: 16% --------------------- - Info: 16% space used, quota is 90%, nothing to do -- Info: Cleaning empty directories... It just doesn't cache anything thou. It's not a joke, I managed to make it work; it's also depending of the settings of your plex docker, what is the Container Path linking to your data in Plex ? Edited July 23, 2022 by doobyns Quote
kizer Posted July 24, 2022 Posted July 24, 2022 Tv Shows are at /mnt/user/TV/Shows/ Movies are at /mnt/user/Movies/All/ I'm only really concerned about my Tv Shows Plex docker shows TV shows listed at /mnt/user/TV/Shows/ Quote
kizer Posted July 24, 2022 Posted July 24, 2022 Checked my setting now I'm seeing this. Script location: /tmp/user.scripts/tmpScripts/Plex-Caching-Script/script Note that closing this window will abort the execution of this script Welcome to /tmp/user.scripts/tmpScripts/Plex-Caching-Script/script stat: cannot statx '/tmp/user.scripts/tmpScripts/Plex-Caching-Script/log.txt': No such file or directory Info: Log size is ---------------------------- 1 active(s) plex session(s): ---------------------------- - 1/1: Serie: Chuck Season 1 - Episode 13/13: Chuck Versus the Marlin -- Info: File 13/13: /TV/Chuck/Season 1/Chuck.-.S01E13.mkv --- Debug:Rsync_transfer function parameters: ---- Debug: Source file: /mnt/user0/TV/Chuck/Season 1/Chuck.-.S01E13.mkv ---- Debug: Dest. file: /mnt/cache/TV/Chuck/Season 1/Chuck.-.S01E13.mkv ---- Debug: Source path: /mnt/user0 ---- Debug: Dest. path: /mnt/cache ---- Debug: Options : --- Error: Files: /mnt/user0/TV/Chuck/Season 1/Chuck.-.S01E13.mkv /mnt/cache/TV/Chuck/Season 1/Chuck.-.S01E13.mkv does not exist --------------------- Cache disk usage: 35% --------------------- - Info: 35% space used, quota is 90%, nothing to do -- Info: Cleaning empty directories... Quote
doobyns Posted July 24, 2022 Posted July 24, 2022 (edited) On 7/24/2022 at 6:53 AM, kizer said: ---- Debug: Source path: /mnt/user0 ---- Debug: Dest. path: /mnt/cache it's weird your source path should be : "/mnt/user0/TV/", and the Dest path : "/mnt/cache/TV/". can you try with a modified version of the script ? (the same i use, a slightly modified version) i've filled some settings, at first try with STORAGE_PATH="/mnt/user0/TV/" and if it's not working try with STORAGE_PATH="/mnt/user0/TV/Shows/". In this version you must specify the container path of your data in plex, for you it's: "TV" (without slash it's important), and for CACHE_DISK it's automatically filled now... Here it is : #!/bin/bash # This script gets actives torrents list from Transmission, playing sessions from Plex, and tells rsync to copy them from array to cache drive. # It also cleans oldest modified files by rsyncing them back to array (in case of modification). # Hardlink are preserved # # By Reynald - 06 may 2020 - mailto:[email protected] # modified by quinto # v.0.5.15 # settings { #Plex PLEX_TOKEN="redacted" PLEX_HOST="192.168.0.xxx:32400" PLEX_MAX_CACHED_SESSIONS=10 PLEX_CONTAINER_PATH="TV" #Rsync path STORAGE_PATH="/mnt/user0/TV/" CACHE_PATH="/mnt/cache/TV/" CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" #Parameters LOG_MAX_SIZE=5000000 NOISY_HOUR_START=9 NOISY_HOUR_STOP=21 #Options (set to true or false) PLEX_ENABLED=true PLEX_CACHE_NEXT_EPISODE=true PLEX_CACHE_SEASON_TILL_END=true VERBOSE=3 #0=Error; 1=Info; 2=More_Info; 3=Debug } ##### No modification below this line ##### CACHE_DISK=$(df -lah "$CACHE_PATH" | awk 'FNR == 2 {print $1}' | tr -d '\n') sys_checks() { # lock if [[ -f /var/lock/smart-cache_plex_transmission ]] then echo "Error: Script already running" exit 1 else touch /var/lock/smart-cache_plex_transmission [[ $VERBOSE -ge 2 ]] && echo "Welcome to $0" fi # check that path are mounted if [[ ! -d $STORAGE_PATH ]] || [[ ! -d $CACHE_PATH ]]; then echo "Error: Paths are not accessibles" rm /var/lock/smart-cache_plex_transmission exit 1 fi # cut log LOG_FILE=$(echo $0 | sed 's|\/script|\/log.txt|') echo "log= $LOG_FILE" LOG_SIZE=$(stat -c %s "$LOG_FILE") [[ $VERBOSE -ge 1 ]] && echo "Info: Log size is $LOG_SIZE" if [[ $LOG_SIZE -ge $LOG_MAX_SIZE ]] then [[ $VERBOSE -ge 1 ]] && echo "Info: Emptying log file" echo "" > "$LOG_FILE" fi [[ $VERBOSE -ge 1 ]] && echo "" } ####################### # Transfers functions # ####################### noisy_hours() { # return 0 if time in noisy hour range if [[ $(date '+%-H') -ge $NOISY_HOUR_START ]] && [[ $(date +%-H) -le $NOISY_HOUR_STOP ]] then return 0 else return 1 fi } rsync_transfer() { # get files and path SOURCE_FILE=$1 DEST_FILE=$2 SOURCE_PATH=$3 DEST_PATH=$4 RS_OPTIONS=$5 [[ $VERBOSE -ge 3 ]] && echo " --- Debug:Rsync_transfer function parameters:" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Source file: $SOURCE_FILE" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Dest. file: $DEST_FILE" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Source path: $SOURCE_PATH" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Dest. path: $DEST_PATH" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Options : $RS_OPTIONS" # check if original file exist if [[ ! -f "${SOURCE_FILE}" ]] && [[ ! -f "${DEST_FILE}" ]] then echo " --- Error: Files:" echo " ${SOURCE_FILE}" echo " ${DEST_FILE}" echo " does not exist" return 1 elif [[ "${DEST_FILE}" = "${DEST_PATH}" ]] || [[ "${SOURCE_FILE}" = "${SOURCE_PATH}" ]] then echo " --- Error: Cannot sync root path!" return 1 elif [[ ! -f "${SOURCE_FILE}" ]] && [[ "${DEST_PATH}" = "${CACHE_PATH}" ]] && [[ -f "${DEST_FILE}" ]] then if noisy_hours then [[ $VERBOSE -ge 2 ]] && echo " -- Info: File is on cache only. Inside noisy hours, sending to storage" rsync_transfer "${DEST_FILE}" "${SOURCE_FILE}" "${DEST_PATH}" "${SOURCE_PATH}" else [[ $VERBOSE -ge 2 ]] && echo " --- Warning: File is on cache only. Outside of noisy hours, doing nothing" fi return elif [[ -f "${DEST_FILE}" ]] && [[ "${DEST_PATH}" = "${CACHE_PATH}" ]] then [[ $VERBOSE -ge 2 ]] && echo " --- Info: File already cached" return fi # get dir SOURCE_DIR=$(dirname "${SOURCE_FILE}") DEST_DIR=$(dirname "${DEST_FILE}") # sync file mkdir -p "${DEST_DIR}" [[ $VERBOSE -ge 1 ]] && echo " --- Info: Syncing ${SOURCE_FILE}" [[ $VERBOSE -ge 2 ]] && echo " --- Info: Syncing ${SOURCE_FILE} to ${DEST_FILE}" rsync -aHq "${SOURCE_FILE}" "${DEST_FILE}" if [[ ! $? -eq 0 ]] then echo " --- Error: cannot rsync ${SOURCE_FILE}" echo " to ${DEST_FILE}" return 1 fi # sync hardlinks hardlink_transfer "${SOURCE_FILE}" "${DEST_FILE}" "${SOURCE_PATH}" "${DEST_PATH}" "${RS_OPTIONS}" # remove original file if requested if [[ "${RS_OPTIONS}" = "--remove-source-files" ]] && $RSYNC_RESULT then [[ $VERBOSE -ge 2 ]] && echo " --- Info: Delete ${SOURCE_FILE}" rm "${SOURCE_FILE}" fi } hardlink_transfer() { # get files and path SOURCE_FILE=$1 DEST_FILE=$2 SOURCE_PATH=$3 DEST_PATH=$4 RS_OPTIONS=$5 [[ $VERBOSE -ge 3 ]] && echo " -- Debug: Hardlink_transfer function parameters:" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Source file: $SOURCE_FILE" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Dest. file: $DEST_FILE" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Source path: $SOURCE_PATH" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Dest. path: $DEST_PATH" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Options : $RS_OPTIONS" # get hardlinks from source find "${SOURCE_PATH}" -samefile "${SOURCE_FILE}" | while read SOURCE_LINK do if [[ ! "${SOURCE_FILE}" = "${SOURCE_LINK}" ]]; then DEST_LINK=$(echo ${SOURCE_LINK} | sed "s|`echo ${SOURCE_PATH}`|`echo ${DEST_PATH}`|") # if DEST is not a hardlink if [[ -f "${DEST_LINK}" ]] && [[ ! `stat -c %h "${DEST_LINK}"` -gt 1 ]] then [[ $VERBOSE -ge 2 ]] && echo " --- Info: Creating hardlink: ${DEST_LINK}" # rm destination file if exist [[ -f "${DEST_LINK}" ]] && rm "${DEST_LINK}" DEST_LINK_DIR=$(dirname "${DEST_LINK}") # and create hardlinks on destination mkdir -p "${DEST_LINK_DIR}" ln "${DEST_FILE}" "${DEST_LINK}" if [[ ! $? -eq 0 ]] then echo " --- Error: cannot hardlink ${DEST_FILE}" echo " to ${DEST_LINK}" return 1 fi else [[ $VERBOSE -ge 2 ]] && echo " --- Info: Hardlink exists" fi # remove hardlinks from source if [[ "${RS_OPTIONS}" = "--remove-source-files" ]] then [[ $VERBOSE -ge 2 ]] && echo " --- Info: Delete hardlink ${SOURCE_LINK}" rm "${SOURCE_LINK}" fi fi done } ######## # Plex # ######## plex_cache() { # get Plex sessions STATUS_SESSIONS=$(curl --silent http://${PLEX_HOST}/status/sessions -H "X-Plex-Token: $PLEX_TOKEN") if [[ -z $STATUS_SESSIONS ]]; then echo "Error: Cannot connect to plex" return 1 fi NB_SESSIONS=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/@size)' -) echo "----------------------------" echo "$NB_SESSIONS active(s) plex session(s):" echo "----------------------------" # for each session if [[ $NB_SESSIONS -gt $PLEX_MAX_CACHED_SESSIONS ]] then NB_SESSIONS=$PLEX_MAX_CACHED_SESSIONS echo "Warning: Caching is limited to $PLEX_MAX_CACHED_SESSIONS plex sessions (user setting)" fi for i in `seq $NB_SESSIONS` do # get title ID=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@ratingKey)' -) TYPE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@type)' -) # eventually get serie info if [[ $TYPE = "episode" ]] then TYPE="Serie" GRANDPARENTTITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@grandparentTitle)' -) SEASON=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@parentIndex)' -) TITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@title)' -) EPISODE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@index)' -) PARENT_ID=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@parentRatingKey)' -) PARENT_NB_EPISODES=$(curl --silent http://${PLEX_HOST}/library/metadata/$PARENT_ID | xmllint --xpath 'string(//MediaContainer/Directory/@leafCount)' -) PARENT_START_EPISODE=$(curl --silent http://${PLEX_HOST}/library/metadata/$PARENT_ID/children | xmllint --xpath 'string(//MediaContainer/Video[1]/@index)' -) PARENT_NB_EPISODES=$(( $PARENT_NB_EPISODES + $PARENT_START_EPISODE - 1 )) TITLE="$TYPE: ${GRANDPARENTTITLE} Season ${SEASON} - Episode ${EPISODE}/${PARENT_NB_EPISODES}: $TITLE" # update nb file to cache START_FILE=$EPISODE $PLEX_CACHE_NEXT_EPISODE && NB_FILES=$(( $EPISODE + 1)) $PLEX_CACHE_SEASON_TILL_END && NB_FILES=$(( $PARENT_NB_EPISODES )) elif [[ $TYPE = "movie" ]] then TYPE="Movie" TITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@title)' -) TITLE="$TYPE: $TITLE" START_FILE=1 NB_FILES=1 else # GRANDPARENTTITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Track['$i']/@grandparentTitle)' -) # PARENTTITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Track['$i']/@parentTitle)' -) # TITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Track['$i']/@title)' -) # TITLE="$TYPE: ${GRANDPARENTTITLE}: ${PARENTTITLE} - ${TITLE}" TYPE="Audio" TITLE="track caching not implemented" TITLE="$TYPE: $TITLE" START_FILE=0 NB_FILES=0 fi echo " - $i/$NB_SESSIONS: $TITLE" # if [[ $NB_FILES -gt $PLEX_MAX_CACHED_SESSIONS ]] # then # echo "Caching is limited to $PLEX_MAX_CACHED_SESSIONS files (user setting)" # NB_FILES=$(( $START_FILE + $PLEX_MAX_CACHED_SESSIONS )) # fi for j in `seq $START_FILE $NB_FILES` do # get file path if [[ $TYPE = "Audio" ]] then [[ $VERBOSE -ge 2 ]] && echo " -- Info: Skipping" else if [[ $TYPE = "Serie" ]] then PLEX_FILE=$(curl --silent http://${PLEX_HOST}/library/metadata/$PARENT_ID/children | xmllint --xpath 'string(//MediaContainer/Video['$(($j - $PARENT_START_EPISODE + 1))']/Media/Part/@file)' -) else PLEX_FILE=$(curl --silent http://${PLEX_HOST}/library/metadata/$ID | xmllint --xpath 'string(//MediaContainer/Video/Media/Part/@file)' -) fi FILE_TO_CACHE=$(sed 's|\"\"|\/|g' <<<${PLEX_FILE}| sed 's|\"||g' | sed 's|\/'${PLEX_CONTAINER_PATH}'\/||') [[ $VERBOSE -ge 2 ]] && echo " -- Info: File $j/$NB_FILES: $FILE_TO_CACHE" STORAGE_FILE="${STORAGE_PATH}${FILE_TO_CACHE}" CACHE_FILE="${CACHE_PATH}${FILE_TO_CACHE}" # and send to rsync rsync_transfer "${STORAGE_FILE}" "${CACHE_FILE}" "${STORAGE_PATH}" "${CACHE_PATH}" ID=$(( $ID + 1 )) fi done done # [[ $NB_SESSIONS != 0 ]] && echo "" [[ $VERBOSE -ge 1 ]] && echo "" } #################### # Delete old files # #################### cleanup() { # get free space a=$(df -h | grep $CACHE_DISK | awk '{ printf "%d", $5 }') b=$CACHE_MIN_FREE_SPACE_PCT echo "---------------------" echo "Cache disk usage: ${a}%" echo "---------------------" if [[ "$a" -ge "$b" ]]; then echo "$a% space used, quota is $b%, cleaning" [[ $VERBOSE -ge 1 ]] && echo "Info: Scanning files..." # get oldest accessed files find "${CACHE_PATH}" -type f -printf "%C@ %p\n" | sort -n | sed "s|`echo ${CACHE_PATH}`|%|g" | cut -d'%' -f2 | while read FILE_TO_CLEAN do # loop start: get free space again a=$(df -h | grep $CACHE_DISK | awk '{ printf "%d", $5 }') b=$CACHE_MAX_FREE_SPACE_PCT # if free space not enough if [[ "$a" -ge "$b" ]]; then [[ $VERBOSE -ge 1 ]] && echo " - Info: $a% space used, target $b%, uncaching $FILE_TO_CLEAN" STORAGE_FILE="${STORAGE_PATH}${FILE_TO_CLEAN}" CACHE_FILE="${CACHE_PATH}${FILE_TO_CLEAN}" # sync back cache to storage rsync_transfer "${CACHE_FILE}" "${STORAGE_FILE}" "${CACHE_PATH}" "${STORAGE_PATH}" "--remove-source-files" fi # loop done fi a=$(df -h | grep $CACHE_DISK | awk '{ printf "%d", $5 }') b=$CACHE_MIN_FREE_SPACE_PCT [[ $VERBOSE -ge 1 ]] && echo " - Info: $a% space used, quota is $b%, nothing to do" # prune empty directories from source dir [[ $VERBOSE -ge 2 ]] && echo " -- Info: Cleaning empty directories..." find "${CACHE_PATH}" -type d -not -path '*/\.*' -empty -prune -exec rmdir --ignore-fail-on-non-empty {} \; [[ $VERBOSE -ge 1 ]] && echo "" } sys_checks $PLEX_ENABLED && plex_cache cleanup echo "" rm /var/lock/smart-cache_plex_transmission exit 0 Edited July 27, 2022 by doobyns Quote
kizer Posted July 25, 2022 Posted July 25, 2022 Ok with your Script and some changes I got it to work. Thank you so much for your efforts and your patience on this. #Plex PLEX_TOKEN="xxxxxxxxxxxxxxxx" PLEX_HOST="192.168.xxx.xxx:32400" PLEX_MAX_CACHED_SESSIONS=10 PLEX_CONTAINER_PATH="TV" #Rsync path STORAGE_PATH="/mnt/user0/TV/Shows/" CACHE_PATH="/mnt/cache/TV/Shows/" CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" I'm running Mover Tuning Script to limit my Mover, and now this script. So I have to come up with a Happy Medium on the Settings to make sure its working smoothly. Can you Please explain a little bit of these. They almost sound like they are backwards in the Amounts and the Naming. CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" I'm trying to understand the clean up. I've adjusted the above numbers up and down and haven't seen any files delete. Quote
doobyns Posted July 25, 2022 Posted July 25, 2022 11 hours ago, kizer said: CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" That i understand from these settings : CACHE_MIN_FREE_SPACE_PCT (90%) is the limit used by the script, above this point the script will free space in the cache by deleting oldest files until CACHE_MAX_FREE_SPACE_PCT is achieved, here 85% So if you use the cache only for plex and only with this script (don't forget to set schedule on this to "custom" and "* * * * *") the mover will be useless for this cache pool. Quote
kizer Posted July 25, 2022 Posted July 25, 2022 Ok, I kinda get it now. One is a Hard Ceiling and the other is a Soft Ceiling. Meaning No matter what this script will not allow files to go higher than 90% and if so will clear until files are below 85%. I did notice a time or two when I manually click the script it said currently already running. Nothing i could tell was running and no matter how long I waited when I clicked the script it said the same thing. I did a reboot and tried again and it appeared to be running normal. Waited a few hours and did the same thing and got the same message. I'm not scared of background processes, but not knowing what something is doing and not knowing when it will end sometimes is a little unnerving. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.