comet424 Posted November 7, 2018 Share Posted November 7, 2018 hi I like to know what you guys recommend for syncing folders from unraid to another unraid same network and 1 offsite so goes through internet.. and I like gui im not so good with command line... as rsync wasn't too bad in freenas but kept replicating itself but it worked.. but I don't see a rysnc in unraid.. id like it to WOL too if possible but id like it to do update either scheduled time through internet.. or instant on the local network backup server and a backup program like you know android backs up your photos to google photo thing id like it to backup my photos from my phone to my network just like that.. is that possible?? Quote Link to comment
Hoopster Posted November 7, 2018 Share Posted November 7, 2018 Yes, unRAID has rsync. I have an rsync script that does the following: 1 - Runs once a week automatically and unattended via User Scripts plugin 2 - Powers on my backup server via IPMI. With my prior server that lacked IPMI, it just woke it from S3 sleep. 3 - Set up email parameters 4 - Backs up everything via SSH login that is new since last backup to my backup server 5 - Backs up share to share as my disk configuration is not identical in both servers 6 - Emails me a backup summary 7 - Powers down the backup server This can be done locally and over the Internet. You could also make the over-the-Internet connection via ZeroTier and run rsync as if the remote server was local as ZeroTier will see your remote server as being in the local LAN. Here is the thread that got me started. I has all my comments (as well as from others) as I learned how to get this setup. Now it is truly set it and forget it. Is just works and runs weekly. I have another script that deletes anything on the backup server that is no longer on the source server. I run this once every 90 days in case I need to restore something that was deleted in the last 90 days. It's not a GUI, but, it is completely unattended and requires zero user interaction, so, who cares? Quote Link to comment
comet424 Posted November 7, 2018 Author Share Posted November 7, 2018 ah ok ill give it a shot.. I didn't see rsync in any options settings plugins apps etc.. but ill check out the link.. ya as long as it works.. I know I having issues with my mini server Acer H340 I using as a backup server 4 bay but doesn't seem to work for WOL and I not sure if network cards newer one if I install would WOL the computer... but ill check out that link first thanks Quote Link to comment
comet424 Posted November 7, 2018 Author Share Posted November 7, 2018 and whats the email for? Quote Link to comment
Hoopster Posted November 7, 2018 Share Posted November 7, 2018 (edited) 7 minutes ago, comet424 said: and whats the email for? It is a summary of the backup activity for each share. Here's a copy of the email I received yesterday. Copying new files to Pictures share ===== Mon Nov 5 01:03:11 MST 2018 Copying new files to Videos share ===== Mon Nov 5 01:04:55 MST 2018 Copying new files to Movies share ===== Mon Nov 5 01:07:35 MST 2018 Copying new files to TVShows share ===== Mon Nov 5 01:07:36 MST 2018 Copying new files to OtherVids share ===== Mon Nov 5 01:12:17 MST 2018 Copying new files to Documents share ===== Mon Nov 5 01:12:17 MST 2018 moving to end ===== Mon Nov 5 01:12:18 MST 2018 ===== Pictures Number of files: 162,933 (reg: 162,534, dir: 399) Number of created files: 409 (reg: 409) Number of deleted files: 0 Number of regular files transferred: 410 Total file size: 1,213,706,202,386 bytes Total transferred file size: 2,355,200,682 bytes Literal data: 2,355,200,682 bytes Matched data: 0 bytes File list size: 196,603 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 2,358,509,718 Total bytes received: 10,985 sent 2,358,509,718 bytes received 10,985 bytes 22,569,576.11 bytes/sec total size is 1,213,706,202,386 speedup is 514.60 ===== Videos Number of files: 17,683 (reg: 17,533, dir: 150) Number of created files: 385 (reg: 384, dir: 1) Number of deleted files: 0 Number of regular files transferred: 384 Total file size: 3,359,231,307,771 bytes Total transferred file size: 13,053,886,464 bytes Literal data: 13,053,886,464 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 13,057,396,877 Total bytes received: 7,721 sent 13,057,396,877 bytes received 7,721 bytes 81,354,545.78 bytes/sec total size is 3,359,231,307,771 speedup is 257.27 ===== Movies Number of files: 632 (reg: 360, dir: 272) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 1,439,609,513,980 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 30,189 Total bytes received: 408 sent 30,189 bytes received 408 bytes 20,398.00 bytes/sec total size is 1,439,609,513,980 speedup is 47,050,675.36 ===== TVShows Number of files: 447 (reg: 388, dir: 59) Number of created files: 24 (reg: 20, dir: 4) Number of deleted files: 0 Number of regular files transferred: 20 Total file size: 528,494,495,875 bytes Total transferred file size: 24,731,864,360 bytes Literal data: 24,731,864,360 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 24,737,922,124 Total bytes received: 561 sent 24,737,922,124 bytes received 561 bytes 87,878,943.82 bytes/sec total size is 528,494,495,875 speedup is 21.36 ===== OtherVids Number of files: 27 (reg: 13, dir: 14) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 21,216,222,489 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 1,213 Total bytes received: 32 sent 1,213 bytes received 32 bytes 2,490.00 bytes/sec total size is 21,216,222,489 speedup is 17,041,142.56 ===== Documents Number of files: 447 (reg: 438, dir: 9) Number of created files: 1 (reg: 1) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 922,799,623 bytes Total transferred file size: 159,898 bytes Literal data: 159,898 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 177,961 Total bytes received: 46 sent 177,961 bytes received 46 bytes 118,671.33 bytes/sec total size is 922,799,623 speedup is 5,184.06 Edited November 7, 2018 by Hoopster Quote Link to comment
JonathanM Posted November 7, 2018 Share Posted November 7, 2018 53 minutes ago, comet424 said: id like it to backup my photos from my phone to my network just like that.. is that possible?? Nextcloud docker along with the nextcloud app on the phone does exactly that. Quote Link to comment
comet424 Posted November 7, 2018 Author Share Posted November 7, 2018 is your script long.. can I copy it and import it... I skimming what guy wrote.. but doesn't look like he has a script just typing stuff I like your setup I need to re read it.. too bad they didn't make it gui version for rsync like freenas does it.. I need to re read this a few time Quote Link to comment
comet424 Posted November 7, 2018 Author Share Posted November 7, 2018 too bad there isn't a video as I learn better with videos part of my learning disability and dyslexia... Quote Link to comment
comet424 Posted November 7, 2018 Author Share Posted November 7, 2018 im kinda confused how to set it up ugh... I have the SMB shares like Photos Videos Music and I wanna send it to my backupserver so main server is called Tower and backup server is BackupServer so I wanna back up those folders. for now or really back up all the folders to the backupserver mirrored I getting confused with the $ rsync -avz -e ssh root@192.168.13.84:/remote/dir /this/dir/ The authenticity of host '192.168.13.221 (192.168.13.84)' can't be established. ECDSA key fingerprint is SHA256:30e/zRmjYVJLB4+5cBc2GjMqEpP7O3sUFcYlzQNYPtY. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.13.84' (ECDSA) to the list of known hosts. [email protected]'s password: that's from the article Quote Link to comment
Hoopster Posted November 7, 2018 Share Posted November 7, 2018 12 minutes ago, comet424 said: is your script long.. can I copy it and import it. See the PM I sent you. Quote Link to comment
Hoopster Posted November 7, 2018 Share Posted November 7, 2018 Other options you may want to look into are Syncthing and Resilio Sync. There are dockers for both and they are configured via a GUI. I used Syncthing before I decided to go down the rsync path. The advantage for me to rsync was that the script could power my backup server on and off as needed. Quote Link to comment
Thorsten Posted November 7, 2020 Share Posted November 7, 2020 On 11/7/2018 at 3:05 AM, Hoopster said: It is a summary of the backup activity for each share. Here's a copy of the email I received yesterday. Copying new files to Pictures share ===== Mon Nov 5 01:03:11 MST 2018 Copying new files to Videos share ===== Mon Nov 5 01:04:55 MST 2018 Copying new files to Movies share ===== Mon Nov 5 01:07:35 MST 2018 Copying new files to TVShows share ===== Mon Nov 5 01:07:36 MST 2018 Copying new files to OtherVids share ===== Mon Nov 5 01:12:17 MST 2018 Copying new files to Documents share ===== Mon Nov 5 01:12:17 MST 2018 moving to end ===== Mon Nov 5 01:12:18 MST 2018 ===== Pictures Number of files: 162,933 (reg: 162,534, dir: 399) Number of created files: 409 (reg: 409) Number of deleted files: 0 Number of regular files transferred: 410 Total file size: 1,213,706,202,386 bytes Total transferred file size: 2,355,200,682 bytes Literal data: 2,355,200,682 bytes Matched data: 0 bytes File list size: 196,603 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 2,358,509,718 Total bytes received: 10,985 sent 2,358,509,718 bytes received 10,985 bytes 22,569,576.11 bytes/sec total size is 1,213,706,202,386 speedup is 514.60 ... Do you still have the script ? Could you also make it available to me. I'm looking for a solution to sync the directories to a USB drive. Quote Link to comment
Hoopster Posted November 7, 2020 Share Posted November 7, 2020 (edited) 2 hours ago, Thorsten said: Do you still have the script ? Could you also make it available to me. I'm looking for a solution to sync the directories to a USB drive. You may want to just use the Unassigned Devices plugin for that. There is a sample script there in the first post that backs up the Pictures share to a USB external drive when it is plugged into the server. I modified that script to backup four of my media shares to a 14TB USB drive. It runs automatically as soon as you connect the drive. You could also run it through User Scripts with a cron schedule to back up on a schedule if the drive is always connected. Here is my Unassigned Devices modified script: #!/bin/bash PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin ## Available variables: # AVAIL : available space # USED : used space # SIZE : partition size # SERIAL : disk serial number # ACTION : if mounting, ADD; if unmounting, UNMOUNT; if unmounted, REMOVE; if error, ERROR_MOUNT, ERROR_UNMOUNT # MOUNTPOINT : where the partition is mounted # FSTYPE : partition filesystem # LABEL : partition label # DEVICE : partition device, e.g /dev/sda1 # OWNER : "udev" if executed by UDEV, otherwise "user" # PROG_NAME : program name of this script # LOGFILE : log file for this script case $ACTION in 'ADD' ) # # Beep that the device is plugged in. # beep -l 200 -f 600 -n -l 200 -f 800 sleep 2 if [ -d $MOUNTPOINT ] then if [ $OWNER = "udev" ] then beep -l 100 -f 2000 -n -l 150 -f 3000 beep -l 100 -f 2000 -n -l 150 -f 3000 logger Started -t$PROG_NAME echo "Started: `date`" > $LOGFILE logger Pictures share -t$PROG_NAME rsync -a -v /mnt/user/Pictures $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Videos share -t$PROG_NAME rsync -a -v /mnt/user/Videos $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Movies share -t$PROG_NAME rsync -a -v /mnt/user/Movies $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Family Videos share -t$PROG_NAME rsync -a -v /mnt/user/FamVideos $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Syncing -t$PROG_NAME sync beep -l 100 -f 2000 -n -l 150 -f 3000 beep -l 100 -f 2000 -n -l 150 -f 3000 beep -r 5 -l 100 -f 2000 logger Unmounting Backup -t$PROG_NAME /usr/local/sbin/rc.unassigned umount $DEVICE echo "Completed: `date`" >> $LOGFILE logger Backup drive can be removed -t$PROG_NAME /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Backup completed" -i "normal" fi else logger Backup Drive Not Mounted -t$PROG_NAME fi ;; 'REMOVE' ) # # Beep that the device is unmounted. # beep -l 200 -f 800 -n -l 200 -f 600 ;; 'ERROR_MOUNT' ) /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not mount Backup" -i "normal" ;; 'ERROR_UNMOUNT' ) /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not unmount Backup" -i "normal" ;; esac Edited November 7, 2020 by Hoopster Quote Link to comment
Hoopster Posted November 7, 2020 Share Posted November 7, 2020 2 hours ago, Thorsten said: Do you still have the script ? Could you also make it available to me. I'm looking for a solution to sync the directories to a USB drive. If you want the script for server to server backup that powers the backup server on and off as needed via IPMI, here it is: !/bin/bash #description=This script backs up shares on MediaNAS to BackupNAS #arrayStarted=true echo "Starting Sync to BackupNAS" echo "Starting Sync $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log # Power On BackupNAS ipmitool -I lan -H 192.168.1.16 -U admin -P XXXXXXXX chassis power on # Wait for 5 minutes # echo "Waiting for BackupNAS to power up..." sleep 3m echo "Host is up" sleep 10s # Set up email header echo To: XXXXXXXXXX.com >> /boot/logs/cronlogs/BackupNAS_Summary.log echo From: XXXXXXXXXXX.com >> /boot/logs/cronlogs/BackupNAS_Summary.log echo Subject: MediaNAS to BackupNAS rsync summary >> /boot/logs/cronlogs/BackupNAS_Summary.log echo >> /boot/logs/cronlogs/BackupNAS_Summary.log # Backup Pictures Share echo "Copying new files to Pictures share ===== $(date)" echo "Copying new files to Pictures share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Pictures share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Pictures.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Pictures/ [email protected]:/mnt/user/Pictures/ >> /boot/logs/cronlogs/BackupNAS_Pictures.log # Backup Videos Share echo "Copying new files to Videos share ===== $(date)" echo "Copying new files to Videos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Videos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Videos.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Videos/ [email protected]:/mnt/user/Videos/ >> /boot/logs/cronlogs/BackupNAS_Videos.log # Backup Movies Share echo "Copying new files to Movies share ===== $(date)" echo "Copying new files to Movies share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Movies share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Movies.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Movies/ [email protected]:/mnt/user/Movies/ >> /boot/logs/cronlogs/BackupNAS_Movies.log # Backup TVShows Share echo "Copying new files to TVShows share ===== $(date)" echo "Copying new files to TVShows share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to TVShows share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_TVShows.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/TVShows/ [email protected]:/mnt/user/TVShows/ >> /boot/logs/cronlogs/BackupNAS_TVShows.log # Backup OtherVids Share echo "Copying new files to OtherVids share ===== $(date)" echo "Copying new files to OtherVids share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to OtherVids share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_OtherVids.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/OtherVids/ [email protected]:/mnt/user/OtherVids/ >> /boot/logs/cronlogs/BackupNAS_OtherVids.log # Backup FamVideos Share echo "Copying new files to FamVideos share ===== $(date)" echo "Copying new files to FamVideos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to FamVideos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_FamVideos.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/FamVideos/ [email protected]:/mnt/user/FamVideos/ >> /boot/logs/cronlogs/BackupNAS_FamVideos.log # Backup Documents Share echo "Copying new files to Documents share ===== $(date)" echo "Copying new files to Documents share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Documents share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Documents.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Documents/ [email protected]:/mnt/user/Documents/ >> /boot/logs/cronlogs/BackupNAS_Documents.log echo "moving to end ===== $(date)" echo "moving to end ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log # Add in the summaries cd /boot/logs/cronlogs/ echo ===== > Pictures.log echo ===== > Videos.log echo ===== > Movies.log echo ===== > TVShows.log echo ===== > OtherVids.log echo ===== > FamVideos.log echo ===== > Documents.log echo Pictures >> Pictures.log echo Videos >> Videos.log echo Movies >> Movies.log echo TVShows >> TVShows.log echo OtherVids >> OtherVids.log echo FamVideos >> FamVideos.log echo Documents >> Documents.log tac BackupNAS_Pictures.log | sed '/^Number of files: /q' | tac >> Pictures.log tac BackupNAS_Videos.log | sed '/^Number of files: /q' | tac >> Videos.log tac BackupNAS_Movies.log | sed '/^Number of files: /q' | tac >> Movies.log tac BackupNAS_TVShows.log | sed '/^Number of files: /q' | tac >> TVShows.log tac BackupNAS_OtherVids.log | sed '/^Number of files: /q' | tac >> OtherVids.log tac BackupNAS_FamVideos.log | sed '/^Number of files: /q' | tac >> FamVideos.log tac BackupNAS_Documents.log | sed '/^Number of files: /q' | tac >> Documents.log # now add all the other logs to the end of this email summary cat BackupNAS_Summary.log Pictures.log Videos.log Movies.log TVShows.log OtherVids.log FamVideos.log Documents.log > allshares.log zip BackupNAS BackupNAS_*.log # Send email of summary of results ssmtp XXXXXXXXXX.com < /boot/logs/cronlogs/allshares.log cd /boot/logs/cronlogs mv BackupNAS.zip "$(date +%Y%m%d_%H%M)_BackupNAS.zip" rm *.log #Power off BackupNAS gracefully sleep 30s ipmitool -I lan -H 192.168.1.16 -U admin -P XXXXXXXX chassis power soft Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.