I want to consolidate some files on a single disk, these files are currently spread across a huge share and I want to avoid creating another share and pin it to a disk. 

 

/mnt/scratch/reseeds/ {lots of symlinks here to /mnt/user/movies/<folder>/<file>}

 

I would like to take all of the folders that have symlinked files and move them all to a single disk, ensuring there is enough space:

/mnt/user/movies/Folder_1/file_1

/mnt/user/movies/Folder_2/file_2

/mnt/user/movies/Folder_8/file_8

->

/mnt/disk1/movies/<here>

 

I'm aware of the re-balance plugin and have used before but I'd like to automate this a bit.

 

My other thought was keep a copy of the file on the array and copy the files across to a ssd or m.2 drive, but that would end up being TBs of data duplicated.  I also seed and unseed quite a bit.

 

I'm happy to tailor a script a little, I just figured I'd check to see if there's something out there before writing something.

Edited December 24, 20241 yr by scs3jb

ok. lets review:

 

Plan

Identify Symlinks: Traverse the source directory (/mnt/scratch/reseeds/) to find symlinks pointing to actual files.

Resolve Real Paths: Get the actual file paths of the symlink targets.

Move Files to the Target Disk: Check the destination disk (/mnt/disk1) for sufficient space and move the files.

Preserve Folder Structure: Maintain the original directory structure under /mnt/disk1/movies/.

 

Here's a script to achieve this:

#!/bin/bash

# Variables
SOURCE_DIR="/mnt/scratch/reseeds"
DESTINATION_DISK="/mnt/disk1/movies"
THRESHOLD_MB=5000  # Minimum free space in MB to ensure on the destination disk

# Check available space on destination disk
available_space=$(df --output=avail -m "$DESTINATION_DISK" | tail -n 1)

if [ "$available_space" -lt "$THRESHOLD_MB" ]; then
  echo "Insufficient space on $DESTINATION_DISK. Available: ${available_space}MB. Required: ${THRESHOLD_MB}MB."
  exit 1
fi

# Traverse the source directory and process symlinks
find "$SOURCE_DIR" -type l | while read -r symlink; do
  # Resolve the real path of the symlink
  real_path=$(readlink -f "$symlink")
  
  # Check if the file exists
  if [ ! -f "$real_path" ]; then
    echo "Skipping: Target file for symlink $symlink not found."
    continue
  fi

  # Construct the destination path
  relative_path=$(realpath --relative-to="/mnt/user/movies" "$real_path")
  dest_path="$DESTINATION_DISK/$relative_path"

  # Create the destination folder if it doesn't exist
  dest_dir=$(dirname "$dest_path")
  mkdir -p "$dest_dir"

  # Move the file
  echo "Moving $real_path to $dest_path"
  mv "$real_path" "$dest_path" || { echo "Failed to move $real_path"; continue; }

  # Remove the symlink
  rm "$symlink" || echo "Failed to remove symlink $symlink"
done

echo "All files processed."

 

How It Works

Check Disk Space: Ensures the destination disk has at least THRESHOLD_MB free space.

Find Symlinks: Uses find to locate all symlinks in the source directory.

Resolve Paths: Resolves the actual file path for each symlink using readlink -f.

Move Files: Moves files to the destination disk, preserving their relative path under /mnt/user/movies/.

Remove Symlinks: Deletes the original symlink after moving the file.

 

...
user secript plugin. run in backgorund it will take some time to complete.

 

#######

Customizations

Disk Selection: Change DESTINATION_DISK to the disk you want to consolidate to.

Free Space Threshold: Adjust THRESHOLD_MB based on your safety margin.

Logging: Redirect output to a log file for tracking

example:

./move_symlinks.sh >> /path/to/logfile.log 2>&1

Warnings:

Testing: Test the script on a small subset of symlinks before running it on the entire directory.

Unraid Disk Behavior: Avoid writing directly to /mnt/user in this operation, as it could invoke the FUSE layer. Use /mnt/disk1 for direct disk operations.

Duplicate Data: The script moves the files rather than copying them, avoiding duplication.

 

You could use MergerFS, I believe it does all this including symlink reworks.

These are great, thanks so much!  I'll take the script approach!

Edited January 4, 20251 yr by scs3jb

hmmm seems the source is a bit wrong:

mv '/mnt/user/movies/The Knocking (2022) {imdb-tt12851130}/The.Knocking.2022.1080p.BluRay.Remux.AVC.DTS-HD.MA.5.1-NOGROUP.mkv' '/mnt/disk8/movies/The Knocking (2022) {imdb-tt12851130}/The.Knocking.2022.1080p.BluRay.Remux.AVC.DTS-HD.MA.5.1-NOGROUP.mkv'

Should be:

mv '/mnt/disk13/movies/The Knocking (2022) {imdb-tt12851130}/The.Knocking.2022.1080p.BluRay.Remux.AVC.DTS-HD.MA.5.1-NOGROUP.mkv' '/mnt/disk8/movies/The Knocking (2022) {imdb-tt12851130}/The.Knocking.2022.1080p.BluRay.Remux.AVC.DTS-HD.MA.5.1-NOGROUP.mkv'

Or the copy will fail, as disk8 is in the share for /mnt/user/movies.

 

Is there a quick helper to get from /mnt/user/movies/ -> /mnt/disk13/movies or is the only option testing for the presence of the file on every disk in a loop?

 

I think I'll also want to move the folder so there aren't blank folders left around on the disks.

Edited January 5, 20251 yr by scs3jb

make sure in shares that you don't have mutiple disk setup and that moveer is not tocuhing the files.

 

otherwsie:

Lets address this issue, you can modify the script to specify the source disk at the beginning and handle moving entire folders while avoiding leaving empty directories behind. Here's the updated script:

 

#!/bin/bash

# Prompt user to specify the source disk
read -p "Enter the source disk (e.g., /mnt/disk13): " SOURCE_DISK

# Variables
SOURCE_DIR="$SOURCE_DISK/movies"
DESTINATION_DISK="/mnt/disk1/movies"
THRESHOLD_MB=5000  # Minimum free space in MB to ensure on the destination disk

# Check available space on destination disk
available_space=$(df --output=avail -m "$DESTINATION_DISK" | tail -n 1)

if [ "$available_space" -lt "$THRESHOLD_MB" ]; then
  echo "Insufficient space on $DESTINATION_DISK. Available: ${available_space}MB. Required: ${THRESHOLD_MB}MB."
  exit 1
fi

# Traverse the source directory and process symlinks
find "$SOURCE_DIR" -type l | while read -r symlink; do
  # Resolve the real path of the symlink
  real_path=$(readlink -f "$symlink")
  
  # Check if the file exists
  if [ ! -f "$real_path" ]; then
    echo "Skipping: Target file for symlink $symlink not found."
    continue
  fi

  # Construct the destination path
  relative_path=$(realpath --relative-to="$SOURCE_DISK/movies" "$real_path")
  dest_path="$DESTINATION_DISK/$relative_path"

  # Create the destination folder if it doesn't exist
  dest_dir=$(dirname "$dest_path")
  mkdir -p "$dest_dir"

  # Move the file
  echo "Moving $real_path to $dest_path"
  mv "$real_path" "$dest_path" || { echo "Failed to move $real_path"; continue; }

  # Remove the symlink
  rm "$symlink" || echo "Failed to remove symlink $symlink"
done

# Clean up: Remove empty folders from the source directory
find "$SOURCE_DIR" -type d -empty -delete

echo "All files processed, and empty folders removed."

 

Key Changes:

User-Specified Disk:

Prompt the user to specify the source disk (e.g., /mnt/disk13) at the beginning of the script.

Dynamic Source Directory:

The SOURCE_DIR is dynamically updated based on the user input.

Folder Cleanup:

Added a command to remove empty directories after moving files using find "$SOURCE_DIR" -type d -empty -delete.

Simplified Path Resolution:

Adjusted realpath to construct the correct relative paths based on the specified disk.

 

How It Works:

The user specifies the disk containing the original files (e.g., /mnt/disk13).

The script processes files and folders under the specified disk.

Moves files while preserving relative paths.

Cleans up any empty directories after processing.

if we don't want the prompt. lets make it a varable

 

#!/bin/bash

# Variables
SOURCE_DISK="/mnt/disk13"  # Set the source disk here
SOURCE_DIR="$SOURCE_DISK/movies"
DESTINATION_DISK="/mnt/disk1/movies"
THRESHOLD_MB=5000  # Minimum free space in MB to ensure on the destination disk

# Check available space on destination disk
available_space=$(df --output=avail -m "$DESTINATION_DISK" | tail -n 1)

if [ "$available_space" -lt "$THRESHOLD_MB" ]; then
  echo "Insufficient space on $DESTINATION_DISK. Available: ${available_space}MB. Required: ${THRESHOLD_MB}MB."
  exit 1
fi

# Traverse the source directory and process symlinks
find "$SOURCE_DIR" -type l | while read -r symlink; do
  # Resolve the real path of the symlink
  real_path=$(readlink -f "$symlink")
  
  # Check if the file exists
  if [ ! -f "$real_path" ]; then
    echo "Skipping: Target file for symlink $symlink not found."
    continue
  fi

  # Construct the destination path
  relative_path=$(realpath --relative-to="$SOURCE_DISK/movies" "$real_path")
  dest_path="$DESTINATION_DISK/$relative_path"

  # Create the destination folder if it doesn't exist
  dest_dir=$(dirname "$dest_path")
  mkdir -p "$dest_dir"

  # Move the file
  echo "Moving $real_path to $dest_path"
  mv "$real_path" "$dest_path" || { echo "Failed to move $real_path"; continue; }

  # Remove the symlink
  rm "$symlink" || echo "Failed to remove symlink $symlink"
done

# Clean up: Remove empty folders from the source directory
find "$SOURCE_DIR" -type d -empty -delete

echo "All files processed, and empty folders removed."

 

Key Changes:

Predefined Variable:

Replaced the user prompt with a predefined variable SOURCE_DISK for specifying the source disk.

Ease of Configuration:

To change the source disk, simply update the SOURCE_DISK variable at the top of the script.

 

Benefits:

Automation: No user input is required, making the script more suitable for automated workflows.

Flexibility: You can easily change the source disk by modifying a single variable.

This version maintains the functionality and avoids blank folders on the source disks after moving files.

 

This will require some investigation on your end to specfi the source disk. the data your wanign to copy data off of

#!/bin/bash

# Variables
#SOURCE_DIR="/mnt/scratch/completes/transmission/reseeds/ptp"
SOURCE_DIR="/mnt/scratch/completes/transmission/reseeds/cg"
DESTINATION_DISK="/mnt/disk8/movies"
THRESHOLD_MB=20000  # Minimum free space in MB to ensure on the destination disk

# Check available space on destination disk
available_space=$(df --output=avail -m "$DESTINATION_DISK" | tail -n 1)

if [ "$available_space" -lt "$THRESHOLD_MB" ]; then
  echo "Insufficient space on $DESTINATION_DISK. Available: ${available_space}MB. Required: ${THRESHOLD_MB}MB."
  exit 1
fi

# Traverse the source directory and process symlinks
find "$SOURCE_DIR" -type l | while read -r symlink; do
  # Resolve the real path of the symlink
  real_path=$(readlink -f "$symlink")

  # Check if the file exists
  if [ ! -f "$real_path" ]; then
    echo "Skipping: Target file for symlink $symlink not found."
    continue
  fi

  # Construct the destination path
  relative_path=$(realpath --relative-to="/mnt/user/movies" "$real_path")
  dest_path="$DESTINATION_DISK/$relative_path"

  # Locate the disk the file is on and set it to src_path
  for file in /mnt/disk{1..30}/movies/"$relative_path"
  do
    [[ -f "$file" ]] && src_path="$file" && break
  done

  # If the src_path contains the destinition disk there's nothing to do, continue
  ALREADY_THERE=`echo "$src_path" | grep "$DESTINATION_DISK"`
  if [ ! -z "$ALREADY_THERE" ]; then
    echo "Skipping: Target $src_path is on $DESTINATION_DISK"
    continue
  fi


  # Make sure we are copying from somewhere
  if [[ -f $src_path ]]; then
    # Create the destination folder if it doesn't exist
    dest_dir=$(dirname "$dest_path")
    mkdir -p "$dest_dir"

    # Move the file
    echo "Moving $src_path to $dest_path"
    mv "$src_path" "$dest_path" || { echo "Failed to move $src_path"; continue; }
  else
    # We shouldn't get here, in case there's something weird with greps, etc.
    echo "Skipping: Something is messed up with $real_path vs disks"
    continue
  fi

done

echo "All files processed."

 

Was what I used in the end and it worked great.  Unfortunately $real_path resolves to /mnt/user/movies (share) so the move will fail, so in the above, it looks for the actual file on the physical disk and sets to $src_path.

 

issues:

variable names suck, src_path and real_path for example should probably be flipped.

/mnt/disk{1..30}/movies/ is hardcoded and would only work for my setup.

folders are left behind.

 

Anyway, now i only have one disk spinning for reseeds, so i'm happy

Edited January 6, 20251 yr by scs3jb