November 27, 20241 yr I looked through some posts. - https://forums.unraid.net/topic/88128-newbie-question-copying-an-external-usb-drive-to-array/ - https://www.reddit.com/r/unRAID/comments/qb3dze/rsyncscp_to_unraid_but_what_for_the_destination/?rdt=54333 First of all, I want to automatically back up data on a remote server and upload it to unraid using a User-Scripts unraid plugins. After creating a script in User-Scripts, I modified the script as follows. #!/bin/bash ### copy fp # ssh-keyscan 192.168.0.2 > /boot/config/ssh/root/known_hosts ### ssh -i /boot/config/ssh/root/server.pem -p 22 [email protected] 'tar -cf /root/web.tar /var/www/html/' scp -i /boot/config/ssh/root/server.pem -P 22 [email protected]:/root/web.tar /where/is/safe/directory/ ssh -i /boot/config/ssh/root/server.pem -p 22 [email protected] 'rm /root/web.tar' I know that `/mnt/user/user_name/` is safe from parity checks on uploads. Is it really safe? And also, when copying files with scp, the file permissions and owner seem to be incorrect. Is this also safe? Or should I fix it as below? chmod -R user_name:users /where/is/safe/directory/ find /where/is/safe/directory/ -type d -print0 | xargs -0 chmod 0777 find /where/is/safe/directory/ -type f -print0 | xargs -0 chmod 0766
November 27, 20241 yr Community Expert Solution Using rsync or SFTP instead of scp is generally more efficient and flexible for tasks like automated backups. Here’s how to improve the script and address your questions: Why rsync or SFTP? Advantages of rsync: Transfers only the changes in files, making subsequent backups faster. Preserves permissions, ownership, and symbolic links by default. Includes options for incremental backups and detailed logging. Advantages of SFTP: Secure and reliable for transferring files. Ensures encrypted communication. Updated Script with rsync Setup Steps SSH Keys: Ensure SSH keys are set up correctly, as in your current script. Modify Script: #!/bin/bash # Define variables REMOTE_SERVER="192.168.0.2" REMOTE_USER="root" REMOTE_DIR="/var/www/html/" LOCAL_DIR="/mnt/user/user_name/backup/" SSH_KEY="/boot/config/ssh/root/server.pem" # Ensure known_hosts is populated ssh-keyscan -p 22 $REMOTE_SERVER >> /boot/config/ssh/root/known_hosts # Use rsync for efficient transfer rsync -avz -e "ssh -i $SSH_KEY -p 22" --delete $REMOTE_USER@$REMOTE_SERVER:$REMOTE_DIR $LOCAL_DIR # Ensure proper permissions on destination chown -R nobody:users $LOCAL_DIR find $LOCAL_DIR -type d -print0 | xargs -0 chmod 0777 find $LOCAL_DIR -type f -print0 | xargs -0 chmod 0666 How It Works rsync Options: -a: Archive mode to preserve file attributes. -v: Verbose output for debugging. -z: Compresses data during transfer. --delete: Removes files in the local destination that no longer exist on the remote server. Permissions Fix: chown -R nobody:users: Ensures correct ownership. chmod: Sets appropriate permissions for directories (0777) and files (0666). Why /mnt/user/user_name/ is Safe Yes, using /mnt/user/user_name/ is safe for backups: Files copied here are automatically included in the Unraid parity calculations. However, copying large files or many small files can temporarily affect performance during parity syncs. If performance is critical, consider using a dedicated share or cache pool for backups. Additional Recommendations Logging: Add logging to monitor backup success rsync -avz --log-file=/path/to/logfile.log -e "ssh -i $SSH_KEY -p 22" $REMOTE_USER@$REMOTE_SERVER:$REMOTE_DIR $LOCAL_DIR *Edit the rsync line... Use --progress for Real-Time Monitoring: Add --progress to see transfer progress during manual runs. Cron Jobs for Automation: Schedule the script with a cron job or via User Scripts plugin in Unraid for periodic backups. Incremental Backups: Add --link-dest to enable incremental backups by maintaining snapshots of previous states.
November 28, 20241 yr Author The answer to why I use scp is that I learned about scp while using the putty program and I simply used scp. It's time for me to learn about rsync now! Among the script contents you sent, I have a question about permission settings. # ls -alh /mnt/user/ drwxrwxrwx 1 nobody users 14 Nov 28 00:00 ./ drwxr-xr-x 8 root root 160 Nov 28 00:00 ../ drwxrwxrwx 1 nobody users 68 Nov 28 00:00 user_name/ # ls -alh /mnt/user/user_name/ drwxrwxrwx 1 nobody users 20 Nov 28 00:00 ./ drwxrwxrwx 1 nobody users 14 Nov 28 00:00 ../ drwxrwxrwx 1 user_name users 3.2K Nov 28 00:00 backup_directory/ -rwxrw-rw- 1 user_name users 2.4G Nov 28 00:00 web.tar.gz* This is a portion of the files and directories uploaded via smb. The permissions of the directory is 777. The permissions of the file is 766. The owner of main directory(/mnt/user/user_name/) is nobody:users. The owner of sub directory(/mnt/user/user_name/backup_dir) is user_name:users. I think it's okay to use 666 permissions, but I'm worried that it will cause problems when using Unraid later. I've looked around and it looks like there's no official documentation for this. I'll check out other plugins that use rsync and update if I find anything that requires modification to the script!
November 28, 20241 yr Community Expert permission to: user(u) group(g) other(o) /¯¯¯\ /¯¯¯\ /¯¯¯\ octal: 6 6 6 binary: 1 1 0 1 1 0 1 1 0 what to permit: r w x r w x r w x binary - 1: enabled, 0: disabled what to permit - r: read, w: write, x: execute permission to - user: the owner that create the file/folder group: the users from group that owner is member other: all other users linux command help review the manages on the command: https://linux.die.net/man/1/rsync review: Unraid default docker safe permission is nobody user user group 777 for read write and executable. By owner, group, other 666 removes the file execution permission so linux .sh files or samba window exe file will not be able to run. it is best to set the nobody:user 777 to all files and folder on unraid due to how different services interace with it. cd /mnt/user/path to data to change chmod -R 777 * chown -R nobody:user * Tools > docker safe permisions to set correct unraid permission. Unraid is more focused on the group permission here. services form remotes things liks smaba user, nfs users, dockers. use the other dockers may change some permission for there security and use within the continaer while they exisit on the unraid disks. 666 is not a bad choice here. the script above was an example path. /mnt/user/user_name/ user_name would be the unrad share name or path on the disk. Edited November 28, 20241 yr by bmartino1 typo - data
November 28, 20241 yr Community Expert rclone is the web ui version of this as well has decent comuntiy support which is rsync via a web ui.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.