I'm looking for recommendations on what plugins or setups to use for automatic transfer of files from an remote server(SFTP) to an unraid local server and then deleting the files on the remote server after the transfer completes.

Current Setup:

• Remote Seedbox with SFTP access
  /home/remoteserver/media/folder1
  /home/remoteserver/downloads/folder2
  

• Unraid local host
  /data1/folder1
  /data1/documents/folder2
  

  Goal: to download all files from seedbox.folder1 and subfolders -> local.folder1 then clear contents of remote.folder1 Then repeat the process on an hourly basis.

I assume you have ssh access too and not just sftp? Just ssh into the box and use rsync. Set up a script with the user script plugin to rsync all the data over then delete on the remote server. Run the script on whatever schedule you like. The script can maybe be something like this:

#!/bin/bash

# Remote and local directories
REMOTE_SERVER="user@remoteserver" # replace with your remote server SSH user and address
REMOTE_FOLDER1="/home/remoteserver/media/folder1/"
REMOTE_FOLDER2="/home/remoteserver/downloads/folder2/"
LOCAL_FOLDER1="/data1/folder1/"
LOCAL_FOLDER2="/data1/documents/folder2/"

# Path to the SSH key (if it's not the default)
SSH_KEY_PATH="/path/to/your/private/key" # Replace with your private key path, if different from default (~/.ssh/id_rsa)

# Make sure the SSH key has proper permissions
chmod 600 "$SSH_KEY_PATH"

# Function to rsync and clear remote folder
sync_and_clear() {
    local remote_folder=$1
    local local_folder=$2

    echo "Syncing from $remote_folder to $local_folder..."
    rsync -avz -e "ssh -i $SSH_KEY_PATH" --remove-source-files "$remote_folder" "$local_folder"
    
    if [ $? -eq 0 ]; then
        echo "Sync completed successfully. Clearing remote folder $remote_folder..."
        ssh -i "$SSH_KEY_PATH" "$REMOTE_SERVER" "find '$remote_folder' -type f -delete" # Delete files from the remote folder
        echo "Contents of $remote_folder cleared."
    else
        echo "Error during syncing. Remote folder contents were not cleared."
    fi
}

# Sync and clear folder1
sync_and_clear "$REMOTE_FOLDER1" "$LOCAL_FOLDER1"

# Sync and clear folder2
sync_and_clear "$REMOTE_FOLDER2" "$LOCAL_FOLDER2"

3 hours ago, strike said:

sh into the box and use rsync. Set up a script with t

Thanks, I got it working in terminal specifying rsync --protocol=31 since my seedbox won't let me update rsync beyond that version. That works successfully when I run it via unraid terminal, but for some reason running the same thing from user scripts throws an error
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(232) [Receiver=3.4.1]
Error during syncing.

So i'll have to experiment more.

Edited June 5, 20251 yr by Specks

Comment out this part if your ssh key is in the default ~/.ssh/id_rsa location:

# Path to the SSH key (if it's not the default)
SSH_KEY_PATH="/path/to/your/private/key" # Replace with your private key path, if different from default (~/.ssh/id_rsa)

The --remove-source-files option will only remove files that have successfully been transferred. If the connection is dropped before any files are transferred, that can lead to the error you are seeing. You might try removing that option temporarily to see if the sync goes through, which can help narrow down issues.

If you're still facing issues, you can add the -v option to ssh in your rsync command to get more verbose output, which may help in troubleshooting:

rsync -avz -e "ssh -i $SSH_KEY_PATH -v" "$remote_folder" "$local_folder"

This should provide more context about the connection issue. You can also add -vvv for even more verbosity to see all debug messages.

Also, maybe try this: https://serverfault.com/questions/447699/rsync-error-unexplained-error-code-255-at-io-c

Appreciate all the help - I realized my issue - I had:
rsync --protocol=31 -avz -e "ssh -v" --remove-source-files "$REMOTE_SERVER":"$remote_folder" "$local_folder"

Should have been:
rsync --protocol=31 -avz -e "ssh -v" --remove-source-files "$REMOTE_SERVER:$remote_folder" "$local_folder"

I also added:
chmod -Rv 770 $local_folder

chown -Rv nobody:users $local_folder

to the end of sync_and_clear() so that I could still delete/ modify the files from my user shares. However this re-applies permissions to every file in the $local_folder. Which could be a problem with slowness/compute time when I'm transferring to massive directories. How would I only apply new rights to files that have just been transferred?

To apply permissions and ownership only to the newly transferred files, you can modify your script to specifically target only those files. One way to achieve this is to include the --itemize-changes command and log the files transfered which can help you apply the chmod and chown commands to just the most recently transferred files after your rsync operation.

You can make the following changes to your script wherever you handle the permissions:

1. Use rsync --itemize-changes to identify which files were just transferred.

2. Process only those files for changing permissions and ownership.

Here’s how you can adjust your script:

#!/bin/bash

# Remote and local directories
REMOTE_SERVER="user@remoteserver" # replace with your remote server SSH user and address
REMOTE_FOLDER1="/home/remoteserver/media/folder1/"
REMOTE_FOLDER2="/home/remoteserver/downloads/folder2/"
LOCAL_FOLDER1="/data1/folder1/"
LOCAL_FOLDER2="/data1/documents/folder2/"

# Path to the SSH key (if it's not the default)
SSH_KEY_PATH="/path/to/your/private/key" # Replace with your private key path, if different from default (~/.ssh/id_rsa)

# Function to rsync and clear remote folder
sync_and_clear() {
    local remote_folder=$1
    local local_folder=$2

    echo "Syncing from $remote_folder to $local_folder..."
    
    # Use --itemize-changes with --log-file to capture detailed info about the transfer
   rsync --protocol=31 -avz --itemize-changes -e "ssh -v" --remove-source-files "$REMOTE_SERVER:$remote_folder" "$local_folder" --log-file=/tmp/rsync_log.txt

    if [ $? -eq 0 ]; then
        echo "Sync completed successfully. Clearing remote folder $remote_folder..."
        ssh -i "$SSH_KEY_PATH" "$REMOTE_SERVER" "find '$remote_folder' -type f -delete" # Delete files from the remote folder
        
        # Process the log to apply permissions and ownership only to the newly transferred files
        echo "Changing permissions and ownership for newly transferred files..."
        while read -r line; do
            # Extracting the transferred file path
            if [[ $line =~ ^[<][^ ] ]]; then
                new_file=$(echo $line | awk '{print $2}')
                # Ensure it starts with the local folder path
                if [[ $new_file == $local_folder* ]]; then
                    chmod 770 "$new_file"
                    chown nobody:users "$new_file"
                fi
            fi
        done < /tmp/rsync_log.txt
        
        # Clean up
        rm /tmp/rsync_log.txt

        echo "Contents of $remote_folder cleared and permissions applied."
    else
        echo "Error during syncing. Remote folder contents were not cleared."
    fi
}

# Sync and clear folder1
sync_and_clear "$REMOTE_FOLDER1" "$LOCAL_FOLDER1"

# Sync and clear folder2
sync_and_clear "$REMOTE_FOLDER2" "$LOCAL_FOLDER2"

Key Changes:

• I added the --itemize-changes option to the rsync command, which provides detailed output about changes made during the transfer process.

• The log file specified with --log-file will reflect only the transferred files.

• Processing the Log: After transferring files, the script reads the /tmp/rsync_log.txt file.

  • The while read -r line; do loop processes each line of the log.

  • It checks if the line starts with a <, which indicates a file that was transferred.

  • It extracts the new file path and applies chmod and chown only to that file.

• Delete the Log: After processing the log file, the script removes it with rm /tmp/rsync_log.txt to keep the environment clean.

Important Note:

Make sure to check the paths and adjust based on the actual transfer and any potential patterns that rsync logs. This approach assumes a certain log format and might need tweaking based on the actual output.

This will ensure that you apply permissions and ownership only to files that were just transferred, rather than the entire local folder.

How It Works:

• Files that were newly transferred will begin with a < character in this output when --itemize-changes is combined with rsync.

• The script reads through the log file, checks for newly transferred entries, and applies the desired permission/ownership changes only to those files.

This should help you isolate newly transferred files for permission adjustments in a more effective manner.

Edited June 5, 20251 yr by strike

Thanks really appreciate all the details. I got it working with setting perms for individual files after tossing the first 4 segments of each line on the log file but then realized the transferred folder permissions were still preventing file deletion. So rather than also search the transfer log and modifying folder perms for all folders as well I found I could simply use the: --no-perms --no-owner --no-group --chmod=777

Less secure, but far simpler and works for my purposes
rsync --protocol=31 -avz --no-perms --no-owner --no-group --chmod=777 -e "ssh -v" "$REMOTE_SERVER:$remote_folder" "$local_folder" --remove-source-files

That is indeed far simpler. ☺️ Glad you found something that works.