doobyns

Members
  • Posts

    62
  • Joined

  • Last visited

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

doobyns's Achievements

Rookie

Rookie (2/14)

8

Reputation

  1. thank you Braulio, i use Luckybackup now, but i will keep the info when'll try your script..
  2. There is only a 2.5 SSD in the DS918+. But the node 804 wasn't really idling there was two transcode on the Unraid server, you can count roughly 15W for 2 stream on the Quadro P2200, and my p2p drive (seagate barracuda 8TB) was spinning, and the two nvme ssd and sata ssd were always active... (like all the ssd). I will make better measurements when I will remove the two Synology and use the PSU only for the Unraid server.
  3. Can't give a precise number at this moment, i'm using an UPS eaton Ellipse 1200 (1200 VA) via network, the ups is connected via usb on my DS918+ but there are three devices idling on the same UPS my server, a Synology DS918+, and a Synology DX513 (with 5 Seagate exos x16 16TB), i can give you a recent pic all hard drives off (files are served y cache) and just 2 transcoding on plex, plus the two Synology idling on the same ups :
  4. if you plan to really use the script on schedule please copy and paste the code again, i've updated the code : now delete files is working correctly
  5. Thanks for the information, i read this in the doc : I understand that i just have to make a top level folder on any pool and it will be considered by the filesystem as the same as it would be selected as "cache" in the gui. My question is when the same data is stored on two same top level folders, one on a array (on a hard drive) and another on a pool (eg on a ssd), when accessing to the array via /mnt/user/toplevelfolder/mydata, how to tell fuse file system to read the file on the ssd (pool) rather than the hard drive (array) ? It will pick automatically the fastest device ?
  6. It's the normal behavior of the script if you launch it manually, i use it scheduled (* * * * *) and on a cache pool dedicated to plex, i think it's safer to have a pool only used for plex cache, and never use the mover on this cache pool, just the "smart caching script"...
  7. How do you do to change the mapping in the unraid GUI?
  8. i need to have cache pool on this share for others operations, but when i copy a large amount of files with SMB i need to avoid filling the cache, and i don't use the cache normally, i use it only for the plex smart caching script here : https://forums.unraid.net/topic/92126-smart-caching-script/ I use my cache as "read only", some data only copied on the cache but all my files stay safe on the array, it would be simpler to have a button in the SMB SECURITY SETTINGS of the share to select if i want use the cache or bypass it, but it's not implemented yet...
  9. yes it's exactly that, where is the setting in shares to avoid using the cache for smb ?
  10. This script allow to bypass UNRAID caching for a specified SMB Share. The Share must be already activated in the UNRAID GUI (Export set to 'Yes'). The name of the share must be filled in the SHARE_NAME parameter before running the script. SMB_PREVENT_CLIENTS_SPINUP add optional settings to the share in order to prevent creation of Thumbs.db and .DS_Store files by clients Notes: Changes made by this script aren't persistent and won't survive after a reboot, if you decide to stop using this script without reboot you can revert the cache bypass by filling SHARE_NAME with a value not corresponding to any share in use, and run the script manually, this will revert all the changes made by this script. The script will check validity and no modifications will be permanent after a reboot (you must set schedule as specified to allow script running after each reboot). If you change the SMB Security Settings oy the share (eg: you set export to "No") then you must to run the script again to reflect the settings in the script (press "Run Script" button in CA User Scripts). Installation: Install the "User Scripts" plugin from "Apps" Add a new script and paste the following script Set the schedule to "At First Array Start" Enjoy 😃 : #!/bin/bash # #This script allow to access a SMB share directly without using the cache # # By Quinto - 27 july 2022 # v.0.9.5 # settings { SHARE_NAME="share_name_without_slashes" #Options SMB_PREVENT_CLIENTS_SPINUP=true #Prevent creation of Thumbs.db and .DS_Store files by clients,set to true or false VERBOSE=0 #0=Error; 1=Info; 2=Debug } ##### No modification below this line ##### SMB_MOUNT=$(cat /etc/samba/smb-shares.conf | awk '/\['${SHARE_NAME}'\]/{flag=1;next}/\[*\]/{flag=0}flag' | sed 's/\/mnt\/user/\/mnt\/user0/g') if [[ ! $SMB_MOUNT ]] then echo -e "Auto-configured share ${SHARE_NAME} not found !\nCheck this script configuration then run it again..." if test -f /etc/samba/smb-override.conf then echo -e "Warning : Script probably running with bad configuration !\nShare ${SHARE_NAME} was probably renamed or deleted, check this script configuration then run it again..." [[ $VERBOSE -ge 1 ]] && echo "Restoring normal state: Deleting unvalid configuration" rm /etc/samba/smb-override.conf sed -i '/include = \/etc\/samba\/smb-override.conf/d' /etc/samba/smb.conf sed -i '/#smb-override script launched, deleted after a reboot/d' /etc/samba/smb.conf [[ $VERBOSE -ge 1 ]] && echo "Updating SMB configuration" smbcontrol all reload-config fi else [[ $VERBOSE -ge 1 ]] && echo "Analysing /etc/samba/smb.conf..." TOTAL_CONFIG="$SMB_MOUNT" if grep -q "include = /etc/samba/smb-override.conf" /etc/samba/smb.conf; then echo -e "Auto-configured share ${SHARE_NAME} exist...\nValues are updated"; fi; [[ $VERBOSE -ge 1 ]] && echo "Installation of smb-override in /etc/samba/smb.conf..." echo -e "\n #smb-override script launched, deleted after a reboot\n include = /etc/samba/smb-override.conf" >> /etc/samba/smb.conf if [[ $SMB_PREVENT_CLIENTS_SPINUP ]] then [[ $VERBOSE -ge 1 ]] && echo "Option to prevent unwanted spinning by Windows & Mac OS clients enabled" SMB_OPTIONS=" veto files = /.smbdelete*/.DS_Store/Thumbs.db/\n delete veto files = yes\n" else SMB_OPTIONS="" fi [[ $VERBOSE -ge 1 ]] && echo -e "UNRAID caching feature is now bypassed when using SMB share : ${SHARE_NAME}" [[ $VERBOSE -ge 2 ]] && echo -e "nNew configuration is:\n${SMB_MOUNT}\n${SMB_OPTIONS}" echo -e "#smb-override script launched, deleted after a reboot\n\n[${SHARE_NAME}]\n${SMB_MOUNT}\n${SMB_OPTIONS}" > /etc/samba/smb-override.conf [[ $VERBOSE -ge 1 ]] && echo "Updating SMB configuration" smbcontrol all reload-config echo "Done, script was correctly executed !" fi echo "" exit 0
  11. That i understand from these settings : CACHE_MIN_FREE_SPACE_PCT (90%) is the limit used by the script, above this point the script will free space in the cache by deleting oldest files until CACHE_MAX_FREE_SPACE_PCT is achieved, here 85% So if you use the cache only for plex and only with this script (don't forget to set schedule on this to "custom" and "* * * * *") the mover will be useless for this cache pool.
  12. it's weird your source path should be : "/mnt/user0/TV/", and the Dest path : "/mnt/cache/TV/". can you try with a modified version of the script ? (the same i use, a slightly modified version) i've filled some settings, at first try with STORAGE_PATH="/mnt/user0/TV/" and if it's not working try with STORAGE_PATH="/mnt/user0/TV/Shows/". In this version you must specify the container path of your data in plex, for you it's: "TV" (without slash it's important), and for CACHE_DISK it's automatically filled now... Here it is : #!/bin/bash # This script gets actives torrents list from Transmission, playing sessions from Plex, and tells rsync to copy them from array to cache drive. # It also cleans oldest modified files by rsyncing them back to array (in case of modification). # Hardlink are preserved # # By Reynald - 06 may 2020 - mailto:reynald.rollet@gmail.com # modified by quinto # v.0.5.15 # settings { #Plex PLEX_TOKEN="redacted" PLEX_HOST="192.168.0.xxx:32400" PLEX_MAX_CACHED_SESSIONS=10 PLEX_CONTAINER_PATH="TV" #Rsync path STORAGE_PATH="/mnt/user0/TV/" CACHE_PATH="/mnt/cache/TV/" CACHE_MIN_FREE_SPACE_PCT="90" CACHE_MAX_FREE_SPACE_PCT="85" #Parameters LOG_MAX_SIZE=5000000 NOISY_HOUR_START=9 NOISY_HOUR_STOP=21 #Options (set to true or false) PLEX_ENABLED=true PLEX_CACHE_NEXT_EPISODE=true PLEX_CACHE_SEASON_TILL_END=true VERBOSE=3 #0=Error; 1=Info; 2=More_Info; 3=Debug } ##### No modification below this line ##### CACHE_DISK=$(df -lah "$CACHE_PATH" | awk 'FNR == 2 {print $1}' | tr -d '\n') sys_checks() { # lock if [[ -f /var/lock/smart-cache_plex_transmission ]] then echo "Error: Script already running" exit 1 else touch /var/lock/smart-cache_plex_transmission [[ $VERBOSE -ge 2 ]] && echo "Welcome to $0" fi # check that path are mounted if [[ ! -d $STORAGE_PATH ]] || [[ ! -d $CACHE_PATH ]]; then echo "Error: Paths are not accessibles" rm /var/lock/smart-cache_plex_transmission exit 1 fi # cut log LOG_FILE=$(echo $0 | sed 's|\/script|\/log.txt|') echo "log= $LOG_FILE" LOG_SIZE=$(stat -c %s "$LOG_FILE") [[ $VERBOSE -ge 1 ]] && echo "Info: Log size is $LOG_SIZE" if [[ $LOG_SIZE -ge $LOG_MAX_SIZE ]] then [[ $VERBOSE -ge 1 ]] && echo "Info: Emptying log file" echo "" > "$LOG_FILE" fi [[ $VERBOSE -ge 1 ]] && echo "" } ####################### # Transfers functions # ####################### noisy_hours() { # return 0 if time in noisy hour range if [[ $(date '+%-H') -ge $NOISY_HOUR_START ]] && [[ $(date +%-H) -le $NOISY_HOUR_STOP ]] then return 0 else return 1 fi } rsync_transfer() { # get files and path SOURCE_FILE=$1 DEST_FILE=$2 SOURCE_PATH=$3 DEST_PATH=$4 RS_OPTIONS=$5 [[ $VERBOSE -ge 3 ]] && echo " --- Debug:Rsync_transfer function parameters:" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Source file: $SOURCE_FILE" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Dest. file: $DEST_FILE" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Source path: $SOURCE_PATH" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Dest. path: $DEST_PATH" [[ $VERBOSE -ge 3 ]] && echo " ---- Debug: Options : $RS_OPTIONS" # check if original file exist if [[ ! -f "${SOURCE_FILE}" ]] && [[ ! -f "${DEST_FILE}" ]] then echo " --- Error: Files:" echo " ${SOURCE_FILE}" echo " ${DEST_FILE}" echo " does not exist" return 1 elif [[ "${DEST_FILE}" = "${DEST_PATH}" ]] || [[ "${SOURCE_FILE}" = "${SOURCE_PATH}" ]] then echo " --- Error: Cannot sync root path!" return 1 elif [[ ! -f "${SOURCE_FILE}" ]] && [[ "${DEST_PATH}" = "${CACHE_PATH}" ]] && [[ -f "${DEST_FILE}" ]] then if noisy_hours then [[ $VERBOSE -ge 2 ]] && echo " -- Info: File is on cache only. Inside noisy hours, sending to storage" rsync_transfer "${DEST_FILE}" "${SOURCE_FILE}" "${DEST_PATH}" "${SOURCE_PATH}" else [[ $VERBOSE -ge 2 ]] && echo " --- Warning: File is on cache only. Outside of noisy hours, doing nothing" fi return elif [[ -f "${DEST_FILE}" ]] && [[ "${DEST_PATH}" = "${CACHE_PATH}" ]] then [[ $VERBOSE -ge 2 ]] && echo " --- Info: File already cached" return fi # get dir SOURCE_DIR=$(dirname "${SOURCE_FILE}") DEST_DIR=$(dirname "${DEST_FILE}") # sync file mkdir -p "${DEST_DIR}" [[ $VERBOSE -ge 1 ]] && echo " --- Info: Syncing ${SOURCE_FILE}" [[ $VERBOSE -ge 2 ]] && echo " --- Info: Syncing ${SOURCE_FILE} to ${DEST_FILE}" rsync -aHq "${SOURCE_FILE}" "${DEST_FILE}" if [[ ! $? -eq 0 ]] then echo " --- Error: cannot rsync ${SOURCE_FILE}" echo " to ${DEST_FILE}" return 1 fi # sync hardlinks hardlink_transfer "${SOURCE_FILE}" "${DEST_FILE}" "${SOURCE_PATH}" "${DEST_PATH}" "${RS_OPTIONS}" # remove original file if requested if [[ "${RS_OPTIONS}" = "--remove-source-files" ]] && $RSYNC_RESULT then [[ $VERBOSE -ge 2 ]] && echo " --- Info: Delete ${SOURCE_FILE}" rm "${SOURCE_FILE}" fi } hardlink_transfer() { # get files and path SOURCE_FILE=$1 DEST_FILE=$2 SOURCE_PATH=$3 DEST_PATH=$4 RS_OPTIONS=$5 [[ $VERBOSE -ge 3 ]] && echo " -- Debug: Hardlink_transfer function parameters:" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Source file: $SOURCE_FILE" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Dest. file: $DEST_FILE" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Source path: $SOURCE_PATH" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Dest. path: $DEST_PATH" [[ $VERBOSE -ge 3 ]] && echo " --- Debug: Options : $RS_OPTIONS" # get hardlinks from source find "${SOURCE_PATH}" -samefile "${SOURCE_FILE}" | while read SOURCE_LINK do if [[ ! "${SOURCE_FILE}" = "${SOURCE_LINK}" ]]; then DEST_LINK=$(echo ${SOURCE_LINK} | sed "s|`echo ${SOURCE_PATH}`|`echo ${DEST_PATH}`|") # if DEST is not a hardlink if [[ -f "${DEST_LINK}" ]] && [[ ! `stat -c %h "${DEST_LINK}"` -gt 1 ]] then [[ $VERBOSE -ge 2 ]] && echo " --- Info: Creating hardlink: ${DEST_LINK}" # rm destination file if exist [[ -f "${DEST_LINK}" ]] && rm "${DEST_LINK}" DEST_LINK_DIR=$(dirname "${DEST_LINK}") # and create hardlinks on destination mkdir -p "${DEST_LINK_DIR}" ln "${DEST_FILE}" "${DEST_LINK}" if [[ ! $? -eq 0 ]] then echo " --- Error: cannot hardlink ${DEST_FILE}" echo " to ${DEST_LINK}" return 1 fi else [[ $VERBOSE -ge 2 ]] && echo " --- Info: Hardlink exists" fi # remove hardlinks from source if [[ "${RS_OPTIONS}" = "--remove-source-files" ]] then [[ $VERBOSE -ge 2 ]] && echo " --- Info: Delete hardlink ${SOURCE_LINK}" rm "${SOURCE_LINK}" fi fi done } ######## # Plex # ######## plex_cache() { # get Plex sessions STATUS_SESSIONS=$(curl --silent http://${PLEX_HOST}/status/sessions -H "X-Plex-Token: $PLEX_TOKEN") if [[ -z $STATUS_SESSIONS ]]; then echo "Error: Cannot connect to plex" return 1 fi NB_SESSIONS=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/@size)' -) echo "----------------------------" echo "$NB_SESSIONS active(s) plex session(s):" echo "----------------------------" # for each session if [[ $NB_SESSIONS -gt $PLEX_MAX_CACHED_SESSIONS ]] then NB_SESSIONS=$PLEX_MAX_CACHED_SESSIONS echo "Warning: Caching is limited to $PLEX_MAX_CACHED_SESSIONS plex sessions (user setting)" fi for i in `seq $NB_SESSIONS` do # get title ID=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@ratingKey)' -) TYPE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@type)' -) # eventually get serie info if [[ $TYPE = "episode" ]] then TYPE="Serie" GRANDPARENTTITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@grandparentTitle)' -) SEASON=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@parentIndex)' -) TITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@title)' -) EPISODE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@index)' -) PARENT_ID=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@parentRatingKey)' -) PARENT_NB_EPISODES=$(curl --silent http://${PLEX_HOST}/library/metadata/$PARENT_ID | xmllint --xpath 'string(//MediaContainer/Directory/@leafCount)' -) PARENT_START_EPISODE=$(curl --silent http://${PLEX_HOST}/library/metadata/$PARENT_ID/children | xmllint --xpath 'string(//MediaContainer/Video[1]/@index)' -) PARENT_NB_EPISODES=$(( $PARENT_NB_EPISODES + $PARENT_START_EPISODE - 1 )) TITLE="$TYPE: ${GRANDPARENTTITLE} Season ${SEASON} - Episode ${EPISODE}/${PARENT_NB_EPISODES}: $TITLE" # update nb file to cache START_FILE=$EPISODE $PLEX_CACHE_NEXT_EPISODE && NB_FILES=$(( $EPISODE + 1)) $PLEX_CACHE_SEASON_TILL_END && NB_FILES=$(( $PARENT_NB_EPISODES )) elif [[ $TYPE = "movie" ]] then TYPE="Movie" TITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Video['$i']/@title)' -) TITLE="$TYPE: $TITLE" START_FILE=1 NB_FILES=1 else # GRANDPARENTTITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Track['$i']/@grandparentTitle)' -) # PARENTTITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Track['$i']/@parentTitle)' -) # TITLE=$(echo $STATUS_SESSIONS | xmllint --xpath 'string(//MediaContainer/Track['$i']/@title)' -) # TITLE="$TYPE: ${GRANDPARENTTITLE}: ${PARENTTITLE} - ${TITLE}" TYPE="Audio" TITLE="track caching not implemented" TITLE="$TYPE: $TITLE" START_FILE=0 NB_FILES=0 fi echo " - $i/$NB_SESSIONS: $TITLE" # if [[ $NB_FILES -gt $PLEX_MAX_CACHED_SESSIONS ]] # then # echo "Caching is limited to $PLEX_MAX_CACHED_SESSIONS files (user setting)" # NB_FILES=$(( $START_FILE + $PLEX_MAX_CACHED_SESSIONS )) # fi for j in `seq $START_FILE $NB_FILES` do # get file path if [[ $TYPE = "Audio" ]] then [[ $VERBOSE -ge 2 ]] && echo " -- Info: Skipping" else if [[ $TYPE = "Serie" ]] then PLEX_FILE=$(curl --silent http://${PLEX_HOST}/library/metadata/$PARENT_ID/children | xmllint --xpath 'string(//MediaContainer/Video['$(($j - $PARENT_START_EPISODE + 1))']/Media/Part/@file)' -) else PLEX_FILE=$(curl --silent http://${PLEX_HOST}/library/metadata/$ID | xmllint --xpath 'string(//MediaContainer/Video/Media/Part/@file)' -) fi FILE_TO_CACHE=$(sed 's|\"\"|\/|g' <<<${PLEX_FILE}| sed 's|\"||g' | sed 's|\/'${PLEX_CONTAINER_PATH}'\/||') [[ $VERBOSE -ge 2 ]] && echo " -- Info: File $j/$NB_FILES: $FILE_TO_CACHE" STORAGE_FILE="${STORAGE_PATH}${FILE_TO_CACHE}" CACHE_FILE="${CACHE_PATH}${FILE_TO_CACHE}" # and send to rsync rsync_transfer "${STORAGE_FILE}" "${CACHE_FILE}" "${STORAGE_PATH}" "${CACHE_PATH}" ID=$(( $ID + 1 )) fi done done # [[ $NB_SESSIONS != 0 ]] && echo "" [[ $VERBOSE -ge 1 ]] && echo "" } #################### # Delete old files # #################### cleanup() { # get free space a=$(df -h | grep $CACHE_DISK | awk '{ printf "%d", $5 }') b=$CACHE_MIN_FREE_SPACE_PCT echo "---------------------" echo "Cache disk usage: ${a}%" echo "---------------------" if [[ "$a" -ge "$b" ]]; then echo "$a% space used, quota is $b%, cleaning" [[ $VERBOSE -ge 1 ]] && echo "Info: Scanning files..." # get oldest accessed files find "${CACHE_PATH}" -type f -printf "%C@ %p\n" | sort -n | sed "s|`echo ${CACHE_PATH}`|%|g" | cut -d'%' -f2 | while read FILE_TO_CLEAN do # loop start: get free space again a=$(df -h | grep $CACHE_DISK | awk '{ printf "%d", $5 }') b=$CACHE_MAX_FREE_SPACE_PCT # if free space not enough if [[ "$a" -ge "$b" ]]; then [[ $VERBOSE -ge 1 ]] && echo " - Info: $a% space used, target $b%, uncaching $FILE_TO_CLEAN" STORAGE_FILE="${STORAGE_PATH}${FILE_TO_CLEAN}" CACHE_FILE="${CACHE_PATH}${FILE_TO_CLEAN}" # sync back cache to storage rsync_transfer "${CACHE_FILE}" "${STORAGE_FILE}" "${CACHE_PATH}" "${STORAGE_PATH}" "--remove-source-files" fi # loop done fi a=$(df -h | grep $CACHE_DISK | awk '{ printf "%d", $5 }') b=$CACHE_MIN_FREE_SPACE_PCT [[ $VERBOSE -ge 1 ]] && echo " - Info: $a% space used, quota is $b%, nothing to do" # prune empty directories from source dir [[ $VERBOSE -ge 2 ]] && echo " -- Info: Cleaning empty directories..." find "${CACHE_PATH}" -type d -not -path '*/\.*' -empty -prune -exec rmdir --ignore-fail-on-non-empty {} \; [[ $VERBOSE -ge 1 ]] && echo "" } sys_checks $PLEX_ENABLED && plex_cache cleanup echo "" rm /var/lock/smart-cache_plex_transmission exit 0
  13. It's not a joke, I managed to make it work; it's also depending of the settings of your plex docker, what is the Container Path linking to your data in Plex ?
  14. Great script, but i've some issues with the cache, eg when the script add a directory on the top level of the cache, a share is automatically created by unraid, when the script add : /data/tv_shows/a_tv_show/tv_show 01x01 Ep1.mkv on the cache, unraid create the share "tv_shows" automatically, my question is : can we add the hardlink in a subdirectory of the cache ? it will be simpler to organise...
  15. J'utilise lucky backup pour faire une copie de certaines shares de mon syno vers mon serveur Unraid, y'a quelques petites manips à faire pour que la connexion ssh se fasse automatiquement sur le serveur syno et il faut un peu configurer lucky backup mais ça fonctionne.