MaZeeT Posted February 22, 2018 Share Posted February 22, 2018 Hello dear community. I am running a NAS with synology's DSM system. On this NAS there are mostly stored my media collection, family photos and running a few docker containers. It took me the better part of a year to move my DVD/bluray collection onto HDD's back when I lived in a small apartment and I would hate to redo the work, so I need a backup solution. I purchased unRAID and 2x 8tb wd red drives in January to make a setup that can solve my problem. The NAS that stores the media collection is running with 4x 4tb wd red drives in Raid5* (Synology's hybrid raid, SHR) I need the setup to be able to make a backup of the media collection and the photos. It would be nice if some or all of the listed features also could be included. Physical location of the unRAID box not to be the same as the NAS. (I have a DNS for the NAS, but I prefer a VPN if possible). Update the backup once a week (or any different time period) and not a real-time sync solution. A delay on removed files, so files removed on the NAS will be removed from the unRAID backup xx days later. A sleep cycle for the unRAID box so it wakes up when it is time to make a backup and goes to sleep afterwards. Preferably the unRAID box will host the applications used for making the backup. For applications used I prefer plugins over docker and VMs, to remove the need for storage on the HDDs themself. At the moment I am trying to setup the openVPN client plugin to enable me to move the unRAID box from my location, but still without success. I have been searching high and low and a suitable solution for the backup itself but I am having a hard time pinning a good one. --------------------- I am writing this in hope of that some with better knowledge of the unRAID environment can help me with suggestions for applications that can do some of the features I desire or possibly a more complete solution. If suggested solutions can come with a guide or instructions on how to set it up I would be really grateful, but a name of a application and a little description will also be appreciated. Kind regards Mads E. Denmark Link to comment
Hoopster Posted February 23, 2018 Share Posted February 23, 2018 On 2/22/2018 at 3:36 PM, MaZeeT said: I need the setup to be able to make a backup of the media collection and the photos. It would be nice if some or all of the listed features also could be included. Physical location of the unRAID box not to be the same as the NAS. (I have a DNS for the NAS, but I prefer a VPN if possible). Update the backup once a week (or any different time period) and not a real-time sync solution. A delay on removed files, so files removed on the NAS will be removed from the unRAID backup xx days later. A sleep cycle for the unRAID box so it wakes up when it is time to make a backup and goes to sleep afterwards. Preferably the unRAID box will host the applications used for making the backup. For applications used I prefer plugins over docker and VMs, to remove the need for storage on the HDDs themself. Not everything I have done will apply to your situation as I am backing up user shares on one unRAID server to another unRAID server; however, what I am doing is very similar to what you want to do. I do not know anything about the Synology file system and how it presents itself to external clients, but, assuming you can mount the file system so it is visible in unRAID and assuming the Synology supports rsync/SSH what you want to do should be possible. - I have a script that runs automatically (automated via the unRAID User Scripts plugin as a cron job) once a week to perform the backup via rsync over SSH. No Dockers or VMs are involved. - I will soon implement a second script which runs every 90 days to delete files on the backup server that are no longer on the main server. - My script powers on the backup server via IPMI and powers it down once the backup is complete. The first version of the script woke up the server from S3 sleep and then put it to sleep again when backup ended. Since then, I have changed motherboards to one that supports IPMI. - Since the script uses rsync over SSH it is secure and can backup to a server over the Internet in a different physical location. The thread that got me started is this one: I modified the OP's script (which is a disk to disk backup) to my needs which are a user share to user share backup which is independent of the disk configuration in the two servers. Here is another thread on the topic started by a user who wants to do something similar to you between a Synology system and an unRAID server: Link to comment
MaZeeT Posted February 27, 2018 Author Share Posted February 27, 2018 Thanks a lot for the detailed answer. What I have been able to find based on your answer, it surely looks like rsync over SSH is the way to go. It will complete my main objectives of a secure link between the two machines were the backup can work. I will have to look into this and set it up when I get a bit more time on my hands. About the script you are using, I understand that a cron-job is a way to run the scripts when I see fit, but can you comment a bit more about what your script do, maybe with a few lines of the script? Once again, thank you for the answer it has been a great help to guide me in my backup project. Kind regards Mads E. Denmark Link to comment
Hoopster Posted February 27, 2018 Share Posted February 27, 2018 On 2/22/2018 at 8:26 PM, Hoopster said: Here is another thread on the topic started by a user who wants to do something similar to you between a Synology system and an unRAID server: 5 hours ago, MaZeeT said: About the script you are using, I understand that a cron-job is a way to run the scripts when I see fit, but can you comment a bit more about what your script do, maybe with a few lines of the script? If you look at my third post in the discussion linked above, I explained what my script is doing and included a full copy of the script. The good thing about rsync is that it works in both directions. You can initiate the script from either the source or the destination server and copy the files in the desired direction. My script is initiated from the source server and copies files from the source to the destination. In the first discussion I linked in my first post, the OP (tr0910) was initiating the rsync operation from the backup server. The most difficult part is getting the ssh keys properly set up between servers so backup can be automated and there is no prompt to manually enter the server password. That process is well documented in the first discussion and the last few posts were the most helpful for me in getting it functioning properly. Here is what I placed in my go file on each server to make sure the SSH keys are preserved through a server reboot: # Copy SSH files back to /root/.ssh folder and set permissions for files mkdir -p /root/.ssh cp /boot/config/ssh/medianas_key /root/.ssh/id_rsa cp /boot/config/ssh/known_hosts /root/.ssh/known_hosts cat /boot/config/ssh/backupnas_key.pub > /root/.ssh/authorized_keys chmod g-rwx,o-rwx -R /root/.ssh The script is completely automated by a cron job that runs at 1am every Monday (after I had run it manually a few times on some test files to verity it worked). Here is a copy of the email I received on Monday morning when the backup completed. I have the email sent to me so I know the backup took place and I can see how many files were copied. The script only copies new or modified files. Copying new files to Pictures share ===== Mon Feb 26 01:03:11 MST 2018 Copying new files to Videos share ===== Mon Feb 26 01:04:50 MST 2018 Copying new files to Movies share ===== Mon Feb 26 01:31:51 MST 2018 Copying new files to TVShows share ===== Mon Feb 26 01:31:52 MST 2018 Copying new files to Documents share ===== Mon Feb 26 01:33:09 MST 2018 moving to end ===== Mon Feb 26 01:33:11 MST 2018 ===== Pictures Number of files: 154,691 (reg: 154,318, dir: 373) Number of created files: 296 (reg: 289, dir: 7) Number of deleted files: 0 Number of regular files transferred: 290 Total file size: 1,145,633,961,598 bytes Total transferred file size: 3,916,036,423 bytes Literal data: 3,914,157,927 bytes Matched data: 1,878,496 bytes File list size: 131,068 File list generation time: 0.004 seconds File list transfer time: 0.000 seconds Total bytes sent: 3,917,720,209 Total bytes received: 14,875 sent 3,917,720,209 bytes received 14,875 bytes 39,374,221.95 bytes/sec total size is 1,145,633,961,598 speedup is 292.42 ===== Videos Number of files: 16,509 (reg: 16,382, dir: 127) Number of created files: 319 (reg: 298, dir: 21) Number of deleted files: 0 Number of regular files transferred: 298 Total file size: 2,790,369,051,195 bytes Total transferred file size: 165,799,607,126 bytes Literal data: 165,799,607,126 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 165,840,384,270 Total bytes received: 6,068 sent 165,840,384,270 bytes received 6,068 bytes 102,275,911.40 bytes/sec total size is 2,790,369,051,195 speedup is 16.83 ===== Movies Number of files: 268 (reg: 171, dir: 97) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 276,226,499,881 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.003 seconds File list transfer time: 0.000 seconds Total bytes sent: 11,238 Total bytes received: 133 sent 11,238 bytes received 133 bytes 7,580.67 bytes/sec total size is 276,226,499,881 speedup is 24,292,190.65 ===== TVShows Number of files: 178 (reg: 146, dir: 32) Number of created files: 2 (reg: 2) Number of deleted files: 0 Number of regular files transferred: 2 Total file size: 484,957,063,283 bytes Total transferred file size: 8,879,064,784 bytes Literal data: 8,879,064,784 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 8,881,241,107 Total bytes received: 104 sent 8,881,241,107 bytes received 104 bytes 114,596,660.79 bytes/sec total size is 484,957,063,283 speedup is 54.60 ===== Documents Number of files: 445 (reg: 436, dir: 9) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 716,208,692 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 17,911 Total bytes received: 27 sent 17,911 bytes received 27 bytes 7,175.20 bytes/sec total size is 716,208,692 speedup is 39,926.90 This time there were only new files to copy in the Pictures, Videos and TVShows shares. It took a little more then 30 minutes to complete the process. I am happy to help out a fellow Dane. My ancestors came to the U.S. from the Aalborg area of Denmark in the 1860s. Let's just say that my surname could indicate that I am a descendant of someone with your Christian name. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.