ronia

Sorry to necro this thread, but it was the first topic that came up when I googled "what happens if unraid has two copies of the same file in cache and array". In response to JonathanM, the behavior is like this: 1. Create a test.txt in /mnt/cache with contents "this is on cache" 2. Create a test.txt in /mnt/disk1 with contents "this is on disk1" 3. Create a test.txt in /mnt/disk2 with contents "this is on disk2" Since I was curious about what would happen (and also a little scared about violating cache coherency), this was the result: 1. cat /mnt/user/test.txt -> "this is on cache" 2. delete /mnt/cache/test.txt 3. cat /mnt/user/test.txt -> "this is on disk1" 4. delete /mnt/disk1/test.txt 5. cat /mnt/user/test.txt -> this is on disk2" So it seems like the cache copy is what will be visible to the /mnt/user/ and then each disk's copy is ranked in order of priority based on some kind of inherent priority. Also, I created the cache copy first. If the priority was based on creation/modification date, then it would have displayed disk2's copy as it was technically "newest". This was not the case, so I think you can rule out atime/ctime/mtime. I think this behavior is extremely useful if you're trying to use the cache pool as a real cache and implementing background scripts to intelligently move files between cache and array. Context: I have stumbled across this post while trying to create my own script that rsyncs the contents of an entire tvshow/season from cache to array if it is currently being watched. Odds are, if someone watches episode 1, there is a good chance they will watch episode 2 and so on. My original implementation was to rsync the array copy to the cache, touch it and then delete the array copy (cache coherency) and let the mover move it back after it becomes old again. But actually, now I think I will simply rsync the array copy to cache and leave the array copy. I will write ANOTHER script to handle "old cache copies" when the cache becomes full. The benefits of this are: 1. The array disk is only written once. 2. If a user watches episodes 1 and 2 and then watches episodes 3 and 4 the next day, the array disk is spun up twice. If all episodes are copied to the cache, the disk is spun up once to read the entire season once, but then spun down and left untouched regardless of rewatching episodes, watching only partial episodes at a time, etc.

Hi Everyone, This is an expansion/update on these two extremely helpful guides: Unfortunately, if you're like me, you didn't read either of those guides and purchased 2 years worth of NordVPN and realized that NordVPN does not provide an easy configuration file to setup your WG tunnel. After much experimenting and failure, I believe I have found the solution to manually create the wireguard tunnel. First off, you want to follow this guide for getting your nordvpn private/public keys manually. Note: you will need a Linux OS (an unraid VM works too. I personally spun up a quick ubuntu instance to do just this). Original guide here: https://gist.github.com/bluewalk/7b3db071c488c82c604baf76a42eaad3 But for the most part, in the comments, darki73 created a script for pulling out the relevant information. Re-posted below for convenience: #!/usr/env/bin bash required_packages=() check_if_connected() { if [ -n "$(nordvpn status | grep "Status: Connected")" ]; then return 0 else return 1 fi } # Check whether jq package is installed if ! command -v jq &> /dev/null; then required_packages+=("jq") fi # Check whether wireguard package is installed if ! command -v wg &> /dev/null; then required_packages+=("wireguard") fi # Check if curl package is installed if ! command -v curl &> /dev/null; then required_packages+=("curl") fi # Check if nordvpn package is installed if ! command -v nordvpn &> /dev/null; then required_packages+=("nordvpn") fi # Install missing packages required to generate the configuration file if [ ${#required_packages[@]} -gt 0 ]; then sudo apt install -y "${required_packages[@]}" fi if ! check_if_connected; then nordvpn connect fi interface_name=$(sudo wg show | grep interface | cut -d " " -f 2) private_key=$(sudo wg show $interface_name private-key | cut -d " " -f 2) my_address=$(ip -f inet addr show $interface_name | grep inet | awk '{print $2}' | cut -d "/" -f 1) api_response=$(curl -s "https://api.nordvpn.com/v1/servers/recommendations?&filters\[servers_technologies\]\[identifier\]=wireguard_udp&limit=1") host=$(jq -r '.[]|.hostname' <<< $api_response) ip=$(jq -r '.[]|.station' <<< $api_response) city=$(jq -r '.[]|(.locations|.[]|.country|.city.name)' <<< $api_response) country=$(jq -r '.[]|(.locations|.[]|.country|.name)' <<< $api_response) server_public_key=$(jq -r '.[]|(.technologies|.[].metadata|.[].value)' <<< $api_response) server_identifier=$(echo $host | cut -d "." -f 1) configuration_file="nordvpn-$server_identifier.conf" { echo "# Configuration for $host ($ip) in $city, $country" echo "[Interface]" echo "Address = $my_address" echo "PrivateKey = $private_key" echo "" echo "[Peer]" echo "PublicKey = $server_public_key" echo "AllowedIPs = 0.0.0.0/0" echo "Endpoint = $host:51820" } > "$configuration_file" if check_if_connected; then nordvpn disconnect fi You'll need to sudo apt-get all the relevant tools above. To be clear: The above script/manual steps pull out the relevant information. It does not create a useable configuration file. That would clearly be too easy. Specifically from this step you want: - Your nordvpn private key (henceforth: private key) - Your nordvpn public key (henceforth: public key) - Nordvpn's public key (henceforth: public key 2) - Nordvpn's server address (henceforth: nord server) - Your LynxVPN IP address (henceforth: lynx IP) Now you're ready to setup the tunnel. 1. Head over to Settings> Network Services>VPN Manager in your Unraid server. 2. Add tunnel. (Mine became wg1). 3. Setup the interface to look something like this: 4a. Setup the peer to look something like this (for system): 4b. Setup the peer to look something like this (for dockers): 5. If the tunnel works you should see the handshake like above. Also, you can sanity test by downloading the firefox docker and assigning the network type to "Custom: wg2" (or whatever wg tunnel instance). Then you can navigate to what's my IP or other website to verify the VPN address. Hope this helps!

I just wanted to comment and say that this script is brilliant and it taught me a lot about how caching works in general. I was originally very confused as I was reading through the script since nothing really seems to be "writing the file to cache". I thought that there would be some cache handle or an API, but all I found was: seconds=$( { time head -c "$preload_head_size" "$file" >/dev/null; } 2>&1 ) Surely, I thought, this can't be it since it's reading the first $preload_head_size into /dev/null. Eventually, I realized that the act of reading the header at all causes the operating system to immediately read it into cache on it's own. Thus @onesbug, I believe the answer to your question is that it already works for Jellyfin. In fact, if I've understood this correctly, this is platform agnostic. The OP is reading all the newest files N bytes into the void, of which the act itself is causing the operating system to 'store' it in memory. The operating system doesn't know we're trying to preload video files. All it knows is that you've read "something" and it's keeping a copy of it in memory just in case you want to read that "something" again. This is also probably why it isn't "reserving memory". The contents are "cached" for as long as nothing else is read and requires that space. You simply need to omit the entire section: # check if paths are used in docker containers I actually don't know why you need to know about the docker container. It seems to be largely a sanity check as none of the information from 'docker container inspect' is used later in the script. In fact, if you want to get this to work with cache pools using SSDs (I suspect that OP originally wrote this script before cache pools were introduced), you can change the first part of the script like so: video_paths=( "/mnt/user0/Movie" "/mnt/user0/TV" ) and modify the sanity check as: # check if paths are used in docker containers if docker info > /dev/null 2>&1; then # get docker mounts of all running containers # shellcheck disable=SC2016 docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$") # for path in "${video_paths[@]}"; do # if [[ $docker_mounts != *"$path"* ]]; then # /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!" # exit 1 # fi # done fi Since the script will complain that /mnt/user0 (the non-cache pool) is not part of any docker container (which makes sense as it wouldn't). Hope this helps anyone else.

Hey thanks for this! This was pretty much exactly what I needed the mover to do with some slight modifications. I've attached my version below in case this is useful for anyone else: #!/bin/bash START_TIME=`date +%s` DATE=`date` # Define variables TARGET_DIRS=("/mnt/cache/plex_lib") OUTPUT_DIR="/dev/shm/cache_mover" OUTPUT_FILE="$OUTPUT_DIR/ignore.txt" MOVE_FILE="$OUTPUT_DIR/moved.log" LOG_FILE="$OUTPUT_DIR/verbose.log" MAX_SIZE="500000000000" # 500 gigabytes in bytes #EXTENSIONS=("mkv" "srt" "mp4" "avi" "rar") VERBOSE=false # Ensure the output directory exists mkdir -p "$OUTPUT_DIR" # Ensure the moved log exists touch $MOVE_FILE # Cleanup previous temporary files rm -f "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_all_files.txt" rm -f $OUTPUT_FILE # MOVE_FILE and LOG_FILE intentionally kept persistent for target_dir in "${TARGET_DIRS[@]}"; do # Step 1: Change directory to the target directory cd "$target_dir" || exit # Step 2: Find files with specified extensions and obtain metadata (loop through extensions) #for ext in "${EXTENSIONS[@]}"; do # find "$(pwd)" -type f -iname "*.$ext" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt" #done # Step 2(alt): Find all files. No filter. find "$(pwd)" -type f -iname "*" -exec stat --printf="%i %Z %n\0" {} + >> "$OUTPUT_DIR/temp_metadata.txt" done # Step 3: Sort metadata by ctime (second column) in descending order sort -z -k 2,2nr -o "$OUTPUT_DIR/temp_metadata.txt" "$OUTPUT_DIR/temp_metadata.txt" # Step 4: Get the newest files up to the specified size limit total_size=0 move_size=0 processed_inodes=() while IFS= read -r -d $'\0' line; do read -r inode ctime path <<< "$line" # Keep track of all files echo $path >> $OUTPUT_DIR/temp_all_files.txt # Skip if the inode has already been processed if [[ "${processed_inodes[*]}" =~ $inode ]]; then continue fi size=$(stat --printf="%s" "$path") if ((total_size + size <= MAX_SIZE)); then if $VERBOSE; then echo "$DATE: Processing file: $total_size $path" >> $LOG_FILE # Debug information to log fi #echo "$path" >> "$OUTPUT_FILE" # Appending only path and filename to the file total_size=$((total_size + size)) # Mark the current inode as processed processed_inodes+=("$inode") # Step 4a: List hardlinks for the current file #hard_links=$(find "$TARGET_DIR" -type f -samefile "$path") #if [ -n "$hard_links" ]; then # echo "$hard_links" >> "$OUTPUT_FILE" #else # echo $path >> $OUTPUT_FILE #fi # Step 4a(alt): Script does not support hardlinks, but is significantly faster and supports multiple TARGET_DIR echo $path >> $OUTPUT_FILE else if $VERBOSE; then echo "$DATE: Moving file: $move_size $path" >> $LOG_FILE # Debug information to log fi move_size=$((move_size + size)) # Do not add to the move file log if previously added if ! grep -q "$path" $MOVE_FILE ; then echo "$DATE: $path" >> $MOVE_FILE fi continue #break fi done < "$OUTPUT_DIR/temp_metadata.txt" # Step 5: Cleanup temporary files rm "$OUTPUT_DIR/temp_metadata.txt" END_TIME=`date +%s` if $VERBOSE; then echo "$DATE: File list generated and saved to: $OUTPUT_FILE" >> $LOG_FILE echo "$DATE: Execution time: $(($END_TIME - $START_TIME)) seconds." >> $LOG_FILE fi Summary: - Removed hardlinks from step 4a. I will not have hardlinks in my share, and it significantly improves execution time - Removed looping over an extension list. I didn't feel like there was anything in my share that couldn't be moved between cache and array - Added execution time logging - Added a 'temp_all_files.txt" file. This was useful in convincing myself that the ignore.txt was capturing everything on the share and nothing was missed. You can just diff the two files together when the cache is below MAX_SIZE and there should be identical. - Added a verbose logging mode - Added a 'moved.log' for tracking what has been moved between cache/array - Changed TARGET_DIR to TARGET_DIRS to loop over all potential shares. At the moment this is just my plex library, but I have some ideas where else this could be used. - Moved the output directory to /dev/shm. My mover is setup to run on the hour, so this reduces the number of read/writes on my cache drive