Backup Scripts combining Restic and Rclone

[D4v3]

Hi guys,

 

I am a new user of Unraid (still in the 30 day trial period) but already very enthusiastic about the system. However, I had problems at the beginning to implement my backup according to my ideas. I wanted to have my backups backed up both locally and remotely. A decent versioning was also important to me. Therefore, I quickly came across the tools Restic https://restic.net/ and Rclone https://rclone.org/. Both are excellent solutions. Both are excellent solutions that, in combination, leave little to be desired. If you want to know more about the tools, you can read more in my manual or go directly to the source and read the documentation of the tools.

 

Now that I knew which tools I wanted to use, I was faced with the problem of implementation. In true unraid fashion, I am not a fan of installing things bare metal. Therefore, docker is the only true way for me. Restic and Rclone both offer an official container. However, it was very difficult for me to use them at the beginning. It took many hours of searching here and in other forums and some trial and error until I had the information I needed. Since I didn't find any solution or instructions suitable for my idea on the way to implementation, I decided to offer it myself now. Hopefully others will be able to reach their backup goal quickly.

 

Features:

• Simple setup
• Automation with cron jobs possible
• Creates local and remote backups
• Fast and flexible creation of backup jobs with templates
• Automatically uses and installs official Docker containers from Restic and Rclone. No additional installations required
• Automatically stops and starts Docker containers during backups
• Separate schedule for running Forget, Prune and the Rclone transfer
• Blocks execution of the same job while it is still running.

 

The scripts, including instructions on how to set them up and use them, can be found in the corresponding Github repository: https://github.com/DavidKrGH/BackupScripts

 

Maybe some of you will like my solution and enjoy it. Then the many hours of work will have been worth it.

With this in mind, I wish you happy backups.

Edited June 15, 2023 by D4v3
Additional information

[Revan335]

Hello,

 

Awesome!

 

Where can I found your Manual?

 

Whats Docker you are using?

 

This one?

 

 https://github.com/imthenachoman/unraid-templates

 

The Scripts downloaded to the Unraid Shares?

 

Many Thanks!

 

Greetings

Edited June 15, 2023 by Revan335

[D4v3]

Hello Revan336,

you can find the instructions in the GutHub readme under "Installation". There everything is described in detail. If you still have questions, please start a discussion there or directly here.

 

The Docker containers used are the official containers from Restic and Rclone. This ensures optimal compatibility and the current state for the future. The script takes care of the installation and execution itself. The containers are removed after each execution. So you only need an installed Docker Deamon. This is already the case with Unraid by default.

 

I use the appdata share as the storage location for the scripts. In principle, however, any location works. 

 

Best regards

[nitrosont]

Hello D4v3,

I had a look in your "BackupScripts" as well. The following questions popped up:

 

1. The script doesn't run as a docker-container itself, right? It fires up a shell script/some shell scripts and those create containers as needed?!

 

2. In the file

BackupScripts/Config/ResticConfig/restic-repo.password.txt

is states

Quote

"Replace this text with your restic repository password in clear text."

Does this mean the password to encrypt/decrypt my restic repository?

 

3. I only need to edit

BackupScripts/Jobs/BaseJob.sh

All the other files should be left untouched?

 

4. With UnRaid, you would suggest to clone the git-files to a folder i.e. appdata => BackupScripts ?

 

5. Minor one: when the script is running and the containers are up: Do I see them under UnRaid in the GUI?

 

Thanks in advance for any reply!

Edited June 19, 2023 by nitrosont

[Revan335]

Used the Scripts/Docker .... only Read Only Permissions for the Backup? That nobody changed the Source Data. Read Write don't needed for a Backup.

[D4v3]

Hello nitrosont,

 

4 hours ago, nitrosont said:

1. The script doesn't run as a docker-container itself, right? It fires up a shell script/some shell scripts and those create containers as needed?!

Correct. The scripts are written in shell. Only restic and rclone are executed as docker, as they are not installed natively on linux.

 

4 hours ago, nitrosont said:

2. In the file

BackupScripts/Config/ResticConfig/restic-repo.password.txt

is states

Does this mean the password to encrypt/decrypt my restic repository?

Correct. Resitc needs the password with which the repository was encrypted to automate backups. If you already have an existing repository, replace the text in the file with it and remove the .txt extension. If you need to create a new one. Follow the instructions in the manual -> First create the password file and then use it to create and encrypt the repository.

 

4 hours ago, nitrosont said:

3. I only need to edit

BackupScripts/Jobs/BaseJob.sh

All the other files should be left untouched?

Yes, after the setup is complete, only the BaseJob needs to be adjusted. Simply copy the template and adapt it to the desired backup job. All other files should not be changed, except for the "restic-repo.password.txt" and the "Notifier.sh" if you want to activate the notifications under unraid. What to do here is also described in the instructions.

 

5 hours ago, nitrosont said:

4. With UnRaid, you would suggest to clone the git-files to a folder i.e. appdata => BackupScripts ?

In principle, you can use any storage location to which your Docker containers have access. I use the appdata share. Just download the zip from Github and extract it. Git automatically names the directory "BackupScripts-main". You then have to rename it to "BackupScripts" or change the "home_path" in your backup jobs. The directory can then be moved to a location of your choice. Please remember to make the executor files executable according to the instructions.

 

5 hours ago, nitrosont said:

5. Minor one: when the script is running and the containers are up: Do I see them under UnRaid in the GUI?

5. Yes, the containers are visible as long as they are executed. The name of the container is composed of the job_name and the process. When no job is running, only the unused images are visible. Here is a example: 

 

All answers correspond to version 1.0.2.  I hope I was able to answer all your questions in an understandable way 🙂
 

[D4v3]

3 hours ago, Revan335 said:

Used the Scripts/Docker .... only Read Only Permissions for the Backup? That nobody changed the Source Data. Read Write don't needed for a Backup.

In the default setting, read and write access to the source is given. Since version 1.0.2, however, support for Docker volume propagation has been implemented. This makes it possible to use the "slave" mode if the source or the target is located on a hard disk that is mounted with "unassigned devices". This can also be used to restrict access to read-only. e.g. source="/PATH/TO/DATA:ro". 

[nitrosont]

56 minutes ago, D4v3 said:

Hello nitrosont,

...

All answers correspond to version 1.0.2.  I hope I was able to answer all your questions in an understandable way 🙂
 

Yes, definitely! I think with these hints even I myself might be able to set it up!
Thank you very much!

[Revan335]

How Updates working?

How can find the Version of your work?

How can I notify about new Versions?

Have your Work a Backup Function for example the BackupScript Folder?

The Repository Folder is under BackupScripts/Repository/Job Name for example?

Can you @D4v3 add the Repo Folder to your work?

Edited August 3, 2023 by Revan335

[Revan335]

Maybe can help anybody other?

[D4v3]

Hello @Revan335. Sorry didn't have time to reply earlier.

On 7/30/2023 at 7:50 PM, Revan335 said:

How Updates working?

How can find the Version of your work?

Updates work by re-downloading the directory from GitHub. Just copy the content of the "Config" and "FilterFiles" directory from the old version to the new one. As for the backup jobs, the easiest way is to recreate them with the new job template.

 

 

On 7/30/2023 at 7:50 PM, Revan335 said:

How can I notify about new Versions?

You can be notified about updates by following the GitHub project. You can find out whether an update is worthwhile for you from the commits in GitHub. If I make major changes or integrate new functions, it will be posted there.

 

On 7/30/2023 at 7:50 PM, Revan335 said:

Have your Work a Backup Function for example the BackupScript Folder?

The Repository Folder is under BackupScripts/Repository/Job Name for example?

I have the Restic repositories under a different path than the "BackupScripts" directory. I have created a job for "BackupScripts" with the job template. If the repositories are in there, you can exclude them from your backup job with a filter. Please remember that Restic and Rclone each need their own filter files.

 

On 7/30/2023 at 7:50 PM, Revan335 said:

Can you @D4v3 add the Repo Folder to your work?

I don't understand this question. What exactly do you mean by that?

Edited August 17, 2023 by D4v3

[redbeard28]

@D4v3 Dude, you did a really good job of documenting your scripts. I'm a Cloud Architect and frankly, it's not often that you find such detailed documentation.
I don't usually intervene in the forums but I had to congratulate you.

Good Job!

[D4v3]

Hello @redbeard28, thank you for the kind words. I am pleased when the scripts are helpful for other users and the work is appreciated.