Automated backup of unraid shares

[Divi]

I have setup unraid server and its working nice.

Now I want to have a cold copy of my files for more safety, I'm planning to build a second NAS out of old Dell server. I dont want it to be up all the time as its power hungry and loud, it would be nice to use maybe wake-on-lan once a week or once a month, update backup if anything has changed, and shutdown secondary nas again. 

 

How would you go to setup something like this? Would it be possible to configure the whole thing on unraid (wake, wait for boot until smb is visible, check files and update what necessary, send shutdown command). This way it would be possible to use pretty much any PC with large enough SMB share for this, and unraid would do all the work.

 

Thoughts? Ideas?

[testdasi]

High level idea of top of my head if you want to control from your main server (assuming your secondary server also runs Unraid).

• Use the userscript plugin to schedule a script that sends WOL signal (or there's a plugin that does that too but not sure about scheduling the plugin - could just schedule the command - see quoted post at the end below).
• Mount the SMB share via Unassigned Devices. Use UD script to trigger backup activities if SMB share is mounted.
  • Once backup done, use the same UD script to do a ssh connection to the secondary NAS and send command "powerdown" to shut it down.

Source re sending WOL

 

 

[Hoopster]

1 hour ago, Divi said:

Now I want to have a cold copy of my files for more safety, I'm planning to build a second NAS out of old Dell server. I dont want it to be up all the time as its power hungry and loud, it would be nice to use maybe wake-on-lan once a week or once a month, update backup if anything has changed, and shutdown secondary nas again. 

That's exactly what I do following the instructions found in this thread.

 

Initially, I just had the script wake the server before backup and then put it to sleep afterwards, but then I swapped in a new motherboard with IPMI so now the script powers on the backup server before backup and powers it off afterwards.

 

The script I modified based on the OPs script does the following (It run once a week via cron job from the User Scripts plugin):

 

1. Power on backup server

2. Backup only those files from the indicated shares that have been added or modified since the last backup.  Note: The OP script is designed to copy from disk to disk but the disk configuration is not identical between my two servers so I needed to do share to share)

3. Prepare and send email summary of the files backed up by share

4. Power off the backup server

 

I have another script I run from time to time which deletes from the backup server any files no longer found on the source server.  This gives me time to identify if any important files were accidentally deleted from the source server; this has happened twice and the backup saved my bacon on an entire years worth of my wife's photos.

 

 

 

 

[superloopy1]

22 hours ago, Hoopster said:

That's exactly what I do following the instructions found in this thread.

 

Initially, I just had the script wake the server before backup and then put it to sleep afterwards, but then I swapped in a new motherboard with IPMI so now the script powers on the backup server before backup and powers it off afterwards.

 

The script I modified based on the OPs script does the following (It run once a week via cron job from the User Scripts plugin):

 

1. Power on backup server

2. Backup only those files from the indicated shares that have been added or modified since the last backup.  Note: The OP script is designed to copy from disk to disk but the disk configuration is not identical between my two servers so I needed to do share to share)

3. Prepare and send email summary of the files backed up by share

4. Power off the backup server

 

I have another script I run from time to time which deletes from the backup server any files no longer found on the source server.  This gives me time to identify if any important files were accidentally deleted from the source server; this has happened twice and the backup saved my bacon on an entire years worth of my wife's photos.

 

 

 

 

Care to share those scripts? 

 

I'm sure i've asked this before and at the time did intend to put a backup in place. Unfortunately, other than being able to make the odd amendments, the whole thing was unachievable as i'm sure it is to a lot of forum members. Surely, by now, unRaid should be offering such a backup script as a number of forum members are not so literate with the ins and outs of Linux, never mind rsysc itself yet a backup is always quoted as being of necessity ;>))

Edited February 6, 2020 by superloopy1

[Hoopster]

On 2/6/2020 at 2:01 AM, superloopy1 said:

Care to share those scripts? 

#!/bin/bash
#description=This script backs up shares on MediaNAS to BackupNAS
#arrayStarted=true

echo "Starting Sync to BackupNAS"
echo "Starting Sync $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log

# Power On BackupNAS
ipmitool -I lan -H 192.168.1.16 -U admin -P XXXXXXXXX chassis power on

# Wait for 5 minutes
echo "Waiting for BackupNAS to power up..."
sleep 3m

echo "Host is up"
sleep 10s

	# Set up email header
	echo To: [email protected] >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo From: [email protected]  >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo Subject: MediaNAS to BackupNAS rsync summary >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo   >> /boot/logs/cronlogs/BackupNAS_Summary.log

				
	# Backup Pictures Share
	echo "Copying new files to Pictures share =====  $(date)"
	echo "Copying new files to Pictures share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Pictures share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Pictures.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Pictures/ [email protected]:/mnt/user/Pictures/  >> /boot/logs/cronlogs/BackupNAS_Pictures.log


	# Backup Videos Share
	echo "Copying new files to Videos share =====  $(date)"
	echo "Copying new files to Videos share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Videos share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Videos.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Videos/ [email protected]:/mnt/user/Videos/  >> /boot/logs/cronlogs/BackupNAS_Videos.log


	# Backup Movies Share
	echo "Copying new files to Movies share =====  $(date)"
	echo "Copying new files to Movies share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Movies share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Movies.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Movies/ [email protected]:/mnt/user/Movies/  >> /boot/logs/cronlogs/BackupNAS_Movies.log


	# Backup TVShows Share
	echo "Copying new files to TVShows share =====  $(date)"
	echo "Copying new files to TVShows share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to TVShows share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_TVShows.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/TVShows/ [email protected]:/mnt/user/TVShows/  >> /boot/logs/cronlogs/BackupNAS_TVShows.log

	# Backup OtherVids Share
	echo "Copying new files to OtherVids share =====  $(date)"
	echo "Copying new files to OtherVids share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to OtherVids share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_OtherVids.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/OtherVids/ [email protected]:/mnt/user/OtherVids/  >> /boot/logs/cronlogs/BackupNAS_OtherVids.log

	# Backup Documents Share
	echo "Copying new files to Documents share =====  $(date)"
	echo "Copying new files to Documents share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log
	echo "Copying new files to Documents share =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Documents.log

	rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Documents/ [email protected]:/mnt/user/Documents/  >> /boot/logs/cronlogs/BackupNAS_Documents.log
	
	echo "moving to end =====  $(date)"
	echo "moving to end =====  $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log

	
	# Add in the summaries
	cd /boot/logs/cronlogs/
	echo ===== > Pictures.log
	echo ===== > Videos.log
	echo ===== > Movies.log
	echo ===== > TVShows.log
	echo ===== > OtherVids.log
	echo ===== > Documents.log
	echo Pictures >> Pictures.log
	echo Videos >> Videos.log
	echo Movies >> Movies.log
	echo TVShows >> TVShows.log
	echo OtherVids >> OtherVids.log
	echo Documents >> Documents.log
	tac BackupNAS_Pictures.log | sed '/^Number of files: /q' | tac >> Pictures.log
	tac BackupNAS_Videos.log | sed '/^Number of files: /q' | tac >> Videos.log
	tac BackupNAS_Movies.log | sed '/^Number of files: /q' | tac >> Movies.log
	tac BackupNAS_TVShows.log | sed '/^Number of files: /q' | tac >> TVShows.log
	tac BackupNAS_OtherVids.log | sed '/^Number of files: /q' | tac >> OtherVids.log
	tac BackupNAS_Documents.log | sed '/^Number of files: /q' | tac >> Documents.log
		
	# now add all the other logs to the end of this email summary
	cat BackupNAS_Summary.log Pictures.log Videos.log Movies.log TVShows.log OtherVids.log Documents.log > allshares.log
	zip BackupNAS BackupNAS_*.log 
	
	# Send email of summary of results
	ssmtp [email protected]  < /boot/logs/cronlogs/allshares.log
	cd /boot/logs/cronlogs  
	mv BackupNAS.zip "$(date +%Y%m%d_%H%M)_BackupNAS.zip"
	rm *.log
	
	#Power off BackupNAS gracefully
	sleep 30s
	ipmitool -I lan -H 192.168.1.16 -U admin -P XXXXXXXX chassis power soft

My backup server still exists on the same LAN as the source server (but tucked away in a corner of the basement).  It would need some slight alterations if connecting to a remote server over the Internet.

 

The OP script is for disk-to-disk backup but I have dissimilar disk configurations so I need to do share-to-share backup

 

Yes, that thread I linked can cause your head to explode.  It took me a good four days to get it all working (as you can see from all my questions in the thread).  Some of the steps in the OP did not work for me and some further refinements to SSH key generation and 'go' file entries were need.  By about page three of that thread, I finally had it figured out and working.  It has been set-it-and-forget-it for two years now.

 

A native backup between unRAID servers functionality is one of the 2020 wish list items on which you can vote.

Edited February 7, 2020 by Hoopster

[superloopy1]

Thanks very much!!

 

I've had a go at setting up ssh keys and failed at the first hurdle but, now with the script in front of me, i'm now going to get this working! I'll let you know how i get on. I have two supermicro based servers and have been gradually swapping out each disk from main server with a larger disk and then moving the outgoing disk to my backup server effectively creating the backup as i go along. There should be minimal updates to sync using this method if all goes well. Only problem as i see it is that my 'main' server HAS onboard ipmi (x8sia-f), my backup doesnt (x8sia only, no onboard ipmi) so will need to either switcharound their functionality or find an onboard ipmi plugin for the backup server to enable wol.

[Hoopster]

5 hours ago, superloopy1 said:

my backup doesnt (x8sia only, no onboard ipmi) so will need to either switcharound their functionality or find an onboard ipmi plugin for the backup server to enable wol.

Originally, the backup server motherboard did not support IPMI.  It did support S3 sleep, so, my original script sent a WOL to the server before backup and put it to sleep afterwards.  IPMI was a later addition when I swapped in a new motherboard which works wonderfully in the script if you install ipmitool from the Nerd Tools plugin.