rSync Backup to Synology Servers


Recommended Posts

I currently have moved everything off my old synology server. I am now trying to use it as a backup, but have had several problems.

 

1. I tried to use syncthing, but is painfully slow over local network

2. Tried rsync, finally got it work, but i have to enter my password

3. I tried to use sshpass, but do not have it on either unraid or synology server

4. How do i create user scripts to back up my files to secondary network storage, on some sort of schedule.

 

Is there a better way to do this. I just want to make network backups for some/all of my shares as needed and have them backup on a schedule without user intervention?

Link to comment
35 minutes ago, mihcox said:

2. Tried rsync, finally got it work, but i have to enter my password

See this rsync backup thread for info on configuring SSH between servers with no password prompt.

 

I use a script to backup all my shares once a week to another unRAID server (there are some users doing it to Synology as a backup as well).

 

With the User Scripts plugin, it is easy to do this on whatever schedule you want via a cron job.  

 

At 1am every Monday my script powers on the backup server via IMPI, backs up the designated shares to the backup server, emails me the results and powers down the backup server until the next time the script runs.

 

It's a piece of cake and runs totally unattended with no user intervention needed.

 

You may struggle a bit to get your head around the unattended SSH part of it, but, once you get it working you never have to think about it again.

  • Like 1
Link to comment
  • 4 months later...

I have a dell server that I’ve ran plex on for several years. I added an 8tb usb hard drive and run a batch file nightly to do aN incremental backup. Can I do something similar with this rsync? 
I’ve been in tech for 20 years and Linux is my biggest deficit. Had to maintain 1k’s of windows machine over those years, so I’m very fluent in that environment... 

Link to comment
8 minutes ago, Goodboy428 said:

Can I do something similar with this rsync? 

Yes.  In fact, the script I use backs up only new/modified files to the backup server.  It DOES NOT delete anything from the backup server that has been deleted since the last backup of the source server. 

 

I have another script that I run manually every three or four months to cleanup deleted source files from the backup.  That usually gives me time to see if anything has been accidentally deleted which has happened twice.  I once lost a whole year of photos on the source server that I was able to recover from the backup server because of the way this is being managed.

Link to comment
  • 1 year later...
On 12/5/2019 at 2:32 PM, Hoopster said:

Yes.  In fact, the script I use backs up only new/modified files to the backup server.  It DOES NOT delete anything from the backup server that has been deleted since the last backup of the source server. 

 

I have another script that I run manually every three or four months to cleanup deleted source files from the backup.  That usually gives me time to see if anything has been accidentally deleted which has happened twice.  I once lost a whole year of photos on the source server that I was able to recover from the backup server because of the way this is being managed.

I'm trying to do the same thing. Could you share your cleanup script?

Link to comment
19 hours ago, Vexedly said:

I'm trying to do the same thing. Could you share your cleanup script?

The cleanup script differs from the weekly backup script on in the addition of the --delete or --delete-after parameter on the rsync lines.

 

Here is the weekly backup script:

#!/bin/bash
#description=This script backs up shares on MediaNAS to BackupNAS
#arrayStarted=true

echo "Starting Sync to BackupNAS"
echo "Starting Sync $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log

# Power On BackupNAS
ipmitool -I lan -H 192.168.1.16 -U admin -P xxxxxxxx chassis power on

# Wait for 5 minutes
# echo "Waiting for BackupNAS to power up..."
sleep 3m

echo "Host is up"
sleep 10s

	# Set up email header
	echo To: [email protected] >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo From: [email protected] >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo Subject: MediaNAS to BackupNAS rsync summary >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo   >> /boot/logs/cronlogs/BackupNAS_Summary.log

				
	# Backup Pictures Share
	echo "Copying new files to Pictures share =====  $(date)"
	echo "Copying new files to Pictures share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Pictures share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Pictures.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Pictures/ [email protected]:/mnt/user/Pictures/  >> /boot/logs/cronlogs/BackupNAS_Pictures.log


	# Backup Videos Share
	echo "Copying new files to Videos share =====  $(date)"
	echo "Copying new files to Videos share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Videos share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Videos.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Videos/ [email protected]:/mnt/user/Videos/  >> /boot/logs/cronlogs/BackupNAS_Videos.log


	# Backup Movies Share
	echo "Copying new files to Movies share =====  $(date)"
	echo "Copying new files to Movies share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Movies share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Movies.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Movies/ [email protected]:/mnt/user/Movies/  >> /boot/logs/cronlogs/BackupNAS_Movies.log


	# Backup TVShows Share
	echo "Copying new files to TVShows share =====  $(date)"
	echo "Copying new files to TVShows share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to TVShows share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_TVShows.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/TVShows/ [email protected]:/mnt/user/TVShows/  >> /boot/logs/cronlogs/BackupNAS_TVShows.log

	# Backup OtherVids Share
	echo "Copying new files to OtherVids share =====  $(date)"
	echo "Copying new files to OtherVids share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to OtherVids share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_OtherVids.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/OtherVids/ [email protected]:/mnt/user/OtherVids/  >> /boot/logs/cronlogs/BackupNAS_OtherVids.log
	
	# Backup FamVideos Share
	echo "Copying new files to FamVideos share =====  $(date)"
	echo "Copying new files to FamVideos share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to FamVideos share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_FamVideos.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/FamVideos/ [email protected]:/mnt/user/FamVideos/  >> /boot/logs/cronlogs/BackupNAS_FamVideos.log

	# Backup Documents Share
	echo "Copying new files to Documents share =====  $(date)"
	echo "Copying new files to Documents share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Documents share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Documents.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Documents/ [email protected]:/mnt/user/Documents/  >> /boot/logs/cronlogs/BackupNAS_Documents.log
	
	echo "moving to end =====  $(date)"
	echo "moving to end =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log

	
	# Add in the summaries
	cd /boot/logs/cronlogs/
	echo ===== > Pictures.log
	echo ===== > Videos.log
	echo ===== > Movies.log
	echo ===== > TVShows.log
	echo ===== > OtherVids.log
	echo ===== > FamVideos.log
	echo ===== > Documents.log
	echo Pictures >> Pictures.log
	echo Videos >> Videos.log
	echo Movies >> Movies.log
	echo TVShows >> TVShows.log
	echo OtherVids >> OtherVids.log
	echo FamVideos >> FamVideos.log
	echo Documents >> Documents.log
	tac BackupNAS_Pictures.log | sed '/^Number of files: /q' | tac >> Pictures.log
	tac BackupNAS_Videos.log | sed '/^Number of files: /q' | tac >> Videos.log
	tac BackupNAS_Movies.log | sed '/^Number of files: /q' | tac >> Movies.log
	tac BackupNAS_TVShows.log | sed '/^Number of files: /q' | tac >> TVShows.log
	tac BackupNAS_OtherVids.log | sed '/^Number of files: /q' | tac >> OtherVids.log
	tac BackupNAS_FamVideos.log | sed '/^Number of files: /q' | tac >> FamVideos.log
	tac BackupNAS_Documents.log | sed '/^Number of files: /q' | tac >> Documents.log
		
	# now add all the other logs to the end of this email summary
	cat BackupNAS_Summary.log Pictures.log Videos.log Movies.log TVShows.log OtherVids.log FamVideos.log Documents.log > allshares.log
	zip BackupNAS BackupNAS_*.log 
	
	# Send email of summary of results
	ssmtp [email protected] < /boot/logs/cronlogs/allshares.log
	cd /boot/logs/cronlogs  
	mv BackupNAS.zip "$(date +%Y%m%d_%H%M)_BackupNAS.zip"
	rm *.log
	
	#Power off BackupNAS gracefully
	sleep 30s
	ipmitool -I lan -H 192.168.1.16 -U admin -P xxxxxxxx chassis power soft
	

The cleanup script has --delete-after before the -e "ssh -i ..." section.

 

--delete caused files to be deleted from the destination during the transfer process whereas --delete-after waits until after files have been transferred to delete files from the destination.  This takes more RAM, but I have plenty.

 

Before actually deleting anything, you should test with the -n or --dry-run parameters as either will show you what will be deleted without actually deleting anything.

 

Here are all the delete options:

--delete
    This tells rsync to delete extraneous files from the receiving side (ones that aren't on the sending side),
    but only for the directories that are being synchronized. You must have asked rsync to send the whole directory
    (e.g. "dir" or "dir/") without using a wildcard for the directory's contents (e.g. "dir/*") since the wildcard is
    expanded by the shell and rsync thus gets a request to transfer individual files, not the files' parent directory.
    Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded
    option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

    Prior to rsync 2.6.7, this option would have no effect unless --recursive was enabled. Beginning with 2.6.7, deletions
    will also occur when --dirs (-d) is enabled, but only for directories whose contents are being copied.

    This option can be dangerous if used incorrectly! It is a very good idea to first try a run using the --dry-run option (-n)
    to see what files are going to be deleted.

    If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled.
    This is to prevent temporary filesystem failures (such as NFS errors) on the sending side from causing a massive deletion
    of files on the destination. You can override this with the --ignore-errors option.

    The --delete option may be combined with one of the --delete-WHEN options without conflict, as well as --delete-excluded.
    However, if none of the --delete-WHEN options are specified, rsync will choose the --delete-during algorithm when
    talking to rsync 3.0.0 or newer, and the --delete-before algorithm when talking to an older rsync.
    See also --delete-delay and --delete-after. 

--delete-before
    Request that the file-deletions on the receiving side be done before the transfer starts. See --delete (which is implied)
    for more details on file-deletion.

    Deleting before the transfer is helpful if the filesystem is tight for space and removing extraneous files would help to
    make the transfer possible. However, it does introduce a delay before the start of the transfer, and this delay might
    cause the transfer to timeout (if --timeout was specified). It also forces rsync to use the old, non-incremental recursion
    algorithm that requires rsync to scan all the files in the transfer into memory at once (see --recursive).

--delete-during, --del
    Request that the file-deletions on the receiving side be done incrementally as the transfer happens.
    The per-directory delete scan is done right before each directory is checked for updates, so it behaves like a more
    efficient --delete-before, including doing the deletions prior to any per-directory filter files being updated.
    This option was first added in rsync version 2.6.4. See --delete (which is implied) for more details on file-deletion.

--delete-delay
    Request that the file-deletions on the receiving side be computed during the transfer (like --delete-during),
    and then removed after the transfer completes. This is useful when combined with --delay-updates and/or --fuzzy, and
    is more efficient than using --delete-after (but can behave differently, since --delete-after computes the deletions
    in a separate pass after all updates are done). If the number of removed files overflows an internal buffer, a temporary
    file will be created on the receiving side to hold the names (it is removed while open, so you shouldn't see it during
    the transfer). If the creation of the temporary file fails, rsync will try to fall back to using --delete-after
    (which it cannot do if --recursive is doing an incremental scan).
    See --delete (which is implied) for more details on file-deletion.

--delete-after
    Request that the file-deletions on the receiving side be done after the transfer has completed.
    This is useful if you are sending new per-directory merge files as a part of the transfer and you want their
    exclusions to take effect for the delete phase of the current transfer. It also forces rsync to use the old,
    non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at
    once (see --recursive). See --delete (which is implied) for more details on file-deletion. 

--delete-excluded
    In addition to deleting the files on the receiving side that are not on
    the sending side, this tells rsync to also delete any files on the receiving side
    that are excluded (see --exclude).

 

Edited by Hoopster
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.