Samsonight

Just since this surely overlooked by many, especially considering how central this is in Unraid and that there is nothing at all notifying you about it: Run this and tell me you have all zeros: for m in $(findmnt -t btrfs -n -o TARGET); do echo "--- $m ---"; btrfs device stats "$m"; done (it will loop through all you btrfs mounts and output error statistics) You can see the same info in Main > <each> Disk Settings > Pool Device Status but unless you looking for it, and understand what it means and what you should do... And as yet another iterating reminder: Unraid Array protects you from disk loss, not data corruption. BtrFS detects data corruption but can't do anything about it. ZFS does both, but can only handle even size disks (don't drift to debate over the specifics, the general gist eli5 is that data corruption happens under our noses) Only persisting reason I noticed this was that https://github.com/netdata/netdata keeps notifying about it until reset.

You really need to mention all circumstances to actually get help, like which containers, otherwise it is like asking why is there no beer... but... https://github.com/qdm12/gluetun/issues/1407#issuecomment-2870017373

Have to? (let the container read the host resources, keeps host clean from modifications, for example Unraid) docker run -d \ --name wazuh-agent \ --privileged \ --network host \ -v /var/log:/host/log:ro \ -v /var/ossec:/var/ossec \ -v /proc:/host/proc:ro \ -v /sys:/host/sys:ro \ wazuh/wazuh-agent https://dev.to/dutdavid/wazuh-agent-as-a-docker-image-1b5n

Surprisingly the container Gluetun runs by default in Userspace 2025-01-17T17:03:16+09:00 INFO [wireguard] Using userspace implementation since Kernel support does not exist by adding the Extra parameter --device /dev/tun the status changes in Log to 2025-01-17T17:03:21+02:00 INFO [wireguard] Using available kernelspace implementation the overhead (slower) is staggering for usermode: https://nordvpn.com/blog/wireguard-kernel-module-vs-user-space/ https://www.netmaker.io/resources/kernel-module-vs-user-space-wireguard

In the spirit this thread I had our friend ChatGPT write us this: You schedule it with Cron to run every 10 minutes. */10 * * * * https://unraid/Settings/Userscripts And add /mnt/user/appdata/cache-filler/cache-filler-exclusion.txt the Mover settings of files to ignore https://unraid/Settings/Scheduler The same file can be used by other scripts to move detected content to the cache. #!/bin/bash # Script to build a persistent exclusion file for Unraid mover script # Monitors /mnt/user/Media for most accessed files using inotify # Outputs filenames to /mnt/user/appdata/cache-filler/cache-filler-exclusion.txt # Only includes files up to 80% of disk space of /mnt/turbo # Exits after 540 seconds to ensure commits # Provides real-time status updates and detailed statistics # Persists all data between runs # Includes concurrent execution protection with PID check ######################################## # Configuration Variables ######################################## WATCH_DIR="/mnt/user/Media" EXCLUSION_FILE="/mnt/user/appdata/cache-filler/cache-filler-exclusion.txt" PERSISTENT_DATA_FILE="/mnt/user/appdata/cache-filler/cache-filler-data.txt" DISK_LIMIT_PATH="/mnt/turbo" SCRIPT_RUNTIME=540 # Script runs for 540 seconds, 9 minutes UPDATE_INTERVAL=120 # Update exclusion file every X seconds DISK_USAGE_THRESHOLD=80 # Percentage of disk usage allowed (80%) DISK_LIMIT_PATH LOCKFILE="/tmp/cache-filler.lock" ######################################## # Initialization ######################################## # Initialize associative arrays to store file data declare -A FILE_ACCESS_COUNT # Stores access count for files declare -A FILE_SIZES # Stores sizes for files declare -A FILE_LAST_ACCESSED # Stores last accessed times # Ensure the exclusion file's directory exists EXCLUSION_DIR=$(dirname "$EXCLUSION_FILE") mkdir -p "$EXCLUSION_DIR" # Ensure the persistent data file's directory exists PERSISTENT_DIR=$(dirname "$PERSISTENT_DATA_FILE") mkdir -p "$PERSISTENT_DIR" ######################################## # Concurrent Execution Protection ######################################## # Function to check if a process is running is_running() { pid=$1 if [ -z "$pid" ]; then return 1 fi if [ -d "/proc/$pid" ]; then return 0 else return 1 fi } # Check if the lock file exists if [ -e "$LOCKFILE" ]; then existing_pid=$(cat "$LOCKFILE") if is_running "$existing_pid"; then echo "Another instance of the script is already running with PID $existing_pid." exit 1 else echo "Found stale lock file with PID $existing_pid. Removing it." rm -f "$LOCKFILE" fi fi # Create a lock file with the current PID echo $$ > "$LOCKFILE" # Set trap to remove lock file on exit trap 'echo "Script terminating..."; rm -f "$LOCKFILE"; update_exclusion_file; save_persistent_data; exit' EXIT SIGINT SIGTERM ######################################## # Function Definitions ######################################## # Function to load persisted data load_persistent_data() { if [ -f "$PERSISTENT_DATA_FILE" ]; then while IFS='|' read -r filepath access_count size last_accessed; do if [[ -n "$filepath" && -f "$filepath" ]]; then normalized_path=$(realpath "$filepath" 2>/dev/null) if [[ -n "$normalized_path" ]]; then FILE_ACCESS_COUNT["$normalized_path"]="$access_count" FILE_SIZES["$normalized_path"]="$size" FILE_LAST_ACCESSED["$normalized_path"]="$last_accessed" fi fi done < "$PERSISTENT_DATA_FILE" echo "Loaded persisted data with ${#FILE_ACCESS_COUNT[@]} files." else echo "No persisted data found. Starting fresh." fi } # Function to save persisted data save_persistent_data() { echo "Saving persisted data..." > "$PERSISTENT_DATA_FILE" # Clear the file for file in "${!FILE_ACCESS_COUNT[@]}"; do access_count="${FILE_ACCESS_COUNT["$file"]}" size="${FILE_SIZES["$file"]}" last_accessed="${FILE_LAST_ACCESSED["$file"]}" echo "$file|$access_count|$size|$last_accessed" >> "$PERSISTENT_DATA_FILE" done echo "Persisted data saved with ${#FILE_ACCESS_COUNT[@]} files." } # Function to calculate total size of tracked files calculate_total_size() { total_size=0 for file in "${!FILE_SIZES[@]}"; do total_size=$((total_size + FILE_SIZES["$file"])) done } # Function to update the exclusion file update_exclusion_file() { echo "Updating exclusion file..." > "$EXCLUSION_FILE" # Clear the file for file in "${!FILE_ACCESS_COUNT[@]}"; do echo "$file" >> "$EXCLUSION_FILE" done echo "Exclusion file updated with ${#FILE_ACCESS_COUNT[@]} files." } # Function to provide real-time status updates status_update() { elapsed_time=$((SECONDS - start_time)) remaining_time=$((SCRIPT_RUNTIME - elapsed_time)) echo "----------------------------------------" echo "Status Update at $(date +'%Y-%m-%d %H:%M:%S'):" echo "Time Elapsed: ${elapsed_time}s, Remaining: ${remaining_time}s" disk_usage=$(df -h "$DISK_LIMIT_PATH" | awk 'NR==2 {print $5}') echo "Disk Usage of $DISK_LIMIT_PATH: $disk_usage" echo "Total Disk Size: $(df -h "$DISK_LIMIT_PATH" | awk 'NR==2 {print $2}')" echo "Available Disk Space: $(df -h "$DISK_LIMIT_PATH" | awk 'NR==2 {print $4}')" echo "Files Tracked: ${#FILE_ACCESS_COUNT[@]}" echo "Total Size of Tracked Files: $(numfmt --to=iec "$total_size")" # Display top 5 most accessed files echo "Top 5 Most Accessed Files:" printf "%-10s %-50s %-10s\n" "Count" "Filename" "Size" # Generate and process data using awk for f in "${!FILE_ACCESS_COUNT[@]}"; do echo "${FILE_ACCESS_COUNT["$f"]}|$f|${FILE_SIZES["$f"]}" done | sort -t'|' -k1,1nr 2>/dev/null | awk -F'|' 'NR<=5 { count = $1 file = $2 size = $3 n = split(file, a, "/") filename = a[n] # Convert size to human-readable format cmd = "numfmt --to=iec " size cmd | getline size_human close(cmd) printf "%-10s %-50s %-10s\n", count, filename, size_human }' echo "----------------------------------------" } ######################################## # Main Script Execution ######################################## # Load persisted data at script start load_persistent_data echo "Starting monitoring of $WATCH_DIR for ACCESS events." # Record script start time start_time=$SECONDS # Initialize last update time last_update_time=$SECONDS # Start inotifywait and process events using process substitution while read -r file; do # Check if script runtime has been exceeded elapsed_time=$((SECONDS - start_time)) if (( elapsed_time >= SCRIPT_RUNTIME )); then echo "Script runtime of $SCRIPT_RUNTIME seconds reached." break # Exit the loop gracefully fi # Normalize the file path real_file=$(realpath "$file" 2>/dev/null) if [[ -z "$real_file" || ! -f "$real_file" ]]; then continue fi # Get the filename without the path for user output file_name=$(basename "$real_file") # Update access count and file size without triggering ACCESS events if [[ -z "${FILE_ACCESS_COUNT["$real_file"]}" ]]; then FILE_ACCESS_COUNT["$real_file"]=1 FILE_SIZES["$real_file"]=$(stat -c%s "$real_file" 2>/dev/null) FILE_LAST_ACCESSED["$real_file"]=$(date +"%Y-%m-%d %H:%M:%S") echo "New file accessed: $file_name (Access Count: ${FILE_ACCESS_COUNT["$real_file"]})" else FILE_ACCESS_COUNT["$real_file"]=$((FILE_ACCESS_COUNT["$real_file"] + 1)) FILE_LAST_ACCESSED["$real_file"]=$(date +"%Y-%m-%d %H:%M:%S") # echo "File accessed: $file_name (Access Count: ${FILE_ACCESS_COUNT["$real_file"]})" fi # Update exclusion file and status every UPDATE_INTERVAL seconds if (( SECONDS - last_update_time >= UPDATE_INTERVAL )); then # Calculate total size of tracked files calculate_total_size # Get total disk size and calculate maximum allowed size total_disk_size=$(df --output=size -k "$DISK_LIMIT_PATH" | awk 'NR==2 {print $1 * 1024}') max_allowed_size=$((total_disk_size * DISK_USAGE_THRESHOLD / 100)) # Remove least accessed files if total size exceeds limit while [ "$total_size" -gt "$max_allowed_size" ]; do # Find the least accessed file least_accessed_file=$(printf "%s\n" "${!FILE_ACCESS_COUNT[@]}" | \ while read -r f; do echo "${FILE_ACCESS_COUNT["$f"]}|$f" done | sort -n | head -n1 | awk -F'|' '{print $2}') # Get the filename without the path for user output least_accessed_file_name=$(basename "$least_accessed_file") echo "Removing file due to size limit: $least_accessed_file_name" # Remove file from tracking unset "FILE_ACCESS_COUNT[$least_accessed_file]" unset "FILE_SIZES[$least_accessed_file]" unset "FILE_LAST_ACCESSED[$least_accessed_file]" # Recalculate total size calculate_total_size done # Update exclusion file and status update_exclusion_file status_update # Update last update time last_update_time=$SECONDS fi done < <(inotifywait -m -r -e access --format '%w%f' "$WATCH_DIR" 2>/dev/null) # Save persisted data before exiting (in case trap did not catch) save_persistent_data echo "Script execution completed." comes with a "Ai" logo too

Afaik. If a share has its primary storage on a Pool (cache) and you write to /mnt/user/ShareX it goes to the Pool (instead of the Array). If you write to the /mnt/user0 it goes to the Array (disk). If /mnt/user0 isn't the (array) disk-only path and /mnt/user is the top layer then what does Mover do when moving to / from array / pool. However it seems Lime is depricating user0, but equally they have not clearly stated before, nor stating now where this functionality would occur. Basically causing threads and confusions like this to fill the forums. Documentation search for user0 https://docs.unraid.net/search/?q=user0 returns only https://docs.unraid.net/unraid-os/manual/shares/#user-shares while the actual reference is https://docs.unraid.net/unraid-os/manual/shares/user-shares/ Please, since you imply factual knowledge of the absolute state of the FUSE layering, could you explain in detail why/how "That would almost certainly result in lost data". Preferrably citing documentation. Just to avoid obscure fud and misunderstandings like this... and how you would suggest a better result achieved. ❤️

Have we considered using (60s just a sample timer) inotifywatch -v -e access -e modify -t 60 -r /mnt/user0 which basically collects statistics of all non-cached (/mnt/user0) files and folders. And then using a script to move the most accessed files to the (unraid) cache (/mnt/user). This would effectively create a "read cache" which could be iterated over and over, for X amount of time, untill the cache is X % full. Please note this records literal disk reads and memory cached content and application cached content is not included. Random example excerpt from the depths of the interwebz... % inotifywatch -v -e access -e modify -t 60 -r ~/.beagle Establishing watches... Setting up watch(es) on /home/rohan/.beagle OK, /home/rohan/.beagle is now being watched. Total of 302 watches. Finished establishing watches, now collecting statistics. Will listen for events for 60 seconds. total access modify filename 1436 1074 362 /home/rohan/.beagle/Indexes/FileSystemIndex/PrimaryIndex/ 1323 1053 270 /home/rohan/.beagle/Indexes/FileSystemIndex/SecondaryIndex/ 303 116 187 /home/rohan/.beagle/Indexes/KMailIndex/PrimaryIndex/ 261 74 187 /home/rohan/.beagle/TextCache/ 206 0 206 /home/rohan/.beagle/Log/ 42 0 42 /home/rohan/.beagle/Indexes/FileSystemIndex/Locks/ 18 6 12 /home/rohan/.beagle/Indexes/FileSystemIndex/ 12 0 12 /home/rohan/.beagle/Indexes/KMailIndex/Locks/ 3 0 3 /home/rohan/.beagle/TextCache/54/ 3 0 3 /home/rohan/.beagle/TextCache/bc/ 3 0 3 /home/rohan/.beagle/TextCache/20/ 3 0 3 /home/rohan/.beagle/TextCache/62/ 2 2 0 /home/rohan/.beagle/Indexes/KMailIndex/SecondaryIndex/