[Support] borgmatic


sdub

Recommended Posts

4 hours ago, kri kri said:

Thanks for making this docker. What is my best course of action for backing up my Borg repo to multiple external drives mounted with UD? Rsync, or make them all borg repos themselves? 

I think it’s generally a better practice to create separate repos and not replicate them. It doesn’t take much longer and avoids propagating corruption or errors. 
 

see the official borg FAQ on this point:

https://borgbackup.readthedocs.io/en/stable/faq.html

Link to comment
4 hours ago, sdub said:

I think it’s generally a better practice to create separate repos and not replicate them. It doesn’t take much longer and avoids propagating corruption or errors. 
 

see the official borg FAQ on this point:

https://borgbackup.readthedocs.io/en/stable/faq.html

I agree with sdub

 

It is best practice to create another repo. This is actually very easy, initiate another repo and just update the config.yaml to include the new repo location (ensure the same password is used).

 

I have several config files backup up to multiple locations, 1 local and 2 seperate remote locations without issue. 

Link to comment

Hi all,

 

I am looking in to this as I am not fully happy with Duplicati.

 

My question - is it possible to have only some sources sent to a particular repo?

 

The scenario I am looking at doing would be:

 

Backup Photos and other important smaller files to a local server AND cloud (e.g. Borgbase perhaps).

 

Backup larger files e.g. Movies/Music etc to local server only.

 

Is this possible all from the one confi.yaml file or would I need to have multiple instances of Borgmatic running?

 

Cheers

Link to comment

@nas_nerd

Yes!

If you add multiple repos to one config file, they will run sequentially but be identical.

What you want is to have multiple config files, one for each repo, both located in the Borgmatic config directory. They’ll still run sequentially but you can configure each repo however you like. No need to have two Borgmatic containers.

You can even use include files to avoid duplicating settings common to both files.

See the support article for more detail:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/


Sent from my iPhone using Tapatalk

  • Like 1
Link to comment

Hello,

 

I see in your crontab.txt example you put:

0 1 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1

 

What is the purpose of PATH=$PATH:/usr/bin I know it adds /usr/bin to the users PATH variable, but what is the purpose/need of this in the cron file?

 

Additionally, where do the borg log files end up?

Edited by Solverz
spelling correction
Link to comment
14 hours ago, Solverz said:

Hello,

 

I see in your crontab.txt example you put:

0 1 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1

 

What is the purpose of PATH=$PATH:/usr/bin I know it adds /usr/bin to the users PATH variable, but what is the purpose/need of this in the cron file?

 

Additionally, where do the borg log files end up?

 

To be honest, I think I just copied the PATH statements from an example, and never messed with it.  It might not be strictly necessary, but note that cron runs as root, but not with the full root environment.  So if you have before backup or after backup hooks in your borgmatic config, they might fail to run without explicitly giving the path.

 

By default, borg/borgmatic just log their output to the console.  the "2>&1" at the end of the line redirects error output to the console.   In our Unraid Docker setup, these logs are viewable in the docker log window.

 

Recently, I decided I wanted proper logfiles, so I added the following to my docker config:

Untitled.thumb.png.0328ab3163a5381440478e3a446d76ba.png

 

Then I changed my cron command as follows:

/usr/bin/borgmatic -v 1 --files --stats > /mnt/logs/backup-"`date +"%a-%H"`".log

 

I perform backsups 2x daily, so this creates rolling logfiles named like:

backup-Mon-01.log
backup-Mon-13.log
backup-Tue-01.log
backup-Tue-13.log
...

 

Note that you could also use a date code formatted like `date +"%Y.%m.%d-%H"` and you'd get logfiles like:

backup-2022.01.11-01.log
backup-2022.01.11-13.log
backup-2022.01.12-01.log
backup-2022.01.12-13.log
...

 

and your log directory would just grow indefinitely.  Also note that I removed the 2>&1 from the end of the command.  This way the borg/borgmatic output is logged, but the errors are also shown on the docker log, which makes monitoring easier.

 

I should probably update the docker template and description to make this the default behavior at some point.

 

 

 

Edited by sdub
  • Thanks 1
Link to comment
2 hours ago, sdub said:

 

To be honest, I think I just copied the PATH statements from an example, and never messed with it.  It might not be strictly necessary, but note that cron runs as root, but not with the full root environment.  So if you have before backup or after backup hooks in your borgmatic config, they might fail to run without explicitly giving the path.

 

By default, borg/borgmatic just log their output to the console.  the "2>&1" at the end of the line redirects error output to the console.   In our Unraid Docker setup, these logs are viewable in the docker log window.

 

Recently, I decided I wanted proper logfiles, so I added the following to my docker config:

Untitled.thumb.png.0328ab3163a5381440478e3a446d76ba.png

 

Then I changed my cron command as follows:

/usr/bin/borgmatic -v 1 --files --stats > /mnt/logs/backup-"`date +"%a-%H"`".log

 

I perform backsups 2x daily, so this creates rolling logfiles named like:

backup-Mon-01.log
backup-Mon-13.log
backup-Tue-01.log
backup-Tue-13.log
...

 

Note that you could also use a date code formatted like `date +"%Y.%m.%d-%H"` and you'd get logfiles like:

backup-2022.01.11-01.log
backup-2022.01.11-13.log
backup-2022.01.12-01.log
backup-2022.01.12-13.log
...

 

and your log directory would just grow indefinitely.  Also note that I removed the 2>&1 from the end of the command.  This way the borg/borgmatic output is logged, but the errors are also shown on the docker log, which makes monitoring easier.

 

I should probably update the docker template and description to make this the default behavior at some point.

 

 

 

I understand, maybe settings up the Logrotate daemon in the container also so the log files don't grow too large would help also.

Link to comment
52 minutes ago, Solverz said:

I understand, maybe settings up the Logrotate daemon in the container also so the log files don't grow too large would help also.

I don't control the underlying docker container, so I can't add logrotate, but it does include syslogd right now.  I'm not experienced with this, but I think you can use that to manage local rotating logs, or send them to an Unraid syslog server

  • Thanks 1
Link to comment

Hi,

I'm trying to find an optimal solution for backing up my Linux/Mac computers to my Unraid box. Borg/borgmatic seems like a solid option based upon my research. I have Vorta installed on my Linux laptop and happily backing up to BorgBase.

 

Installed Borgmatic on Unraid box, though I am lost on how to configure and initialize the repositories.

 

I'd greatly appreciate any pointers on how to set it up.

 

Also, is Borgmatic a good solution for backing up a PC to Unraid, or it is meant more for backing up the Unraid box shares? Should I go with something like Duplicacy instead?

 

Thanks in advance!

Link to comment

Hi!

 

I'm using this container in unraid to set up backups to an intermittently connected external drive. Following the borgmatic docs here, I'm trying to probe for the drive using findmnt, however when I run it in the container I get `findmnt: not found`.

 

From googling it seems like findmnt is including in most linux distros, and it works in the unraid shell, so I'm not sure why it doesn't work in the container. Do you know why it may not be available, and if there is any fix or alternative approach to checking if a drive is mounted?

 

Thanks!

 

Link to comment
1 hour ago, shaihulud said:

Hi!

 

I'm using this container in unraid to set up backups to an intermittently connected external drive. Following the borgmatic docs here, I'm trying to probe for the drive using findmnt, however when I run it in the container I get `findmnt: not found`.

 

From googling it seems like findmnt is including in most linux distros, and it works in the unraid shell, so I'm not sure why it doesn't work in the container. Do you know why it may not be available, and if there is any fix or alternative approach to checking if a drive is mounted?

 

Thanks!

 


Docker containers are by design minimal and not full Linux distros. You could open an issue with the container manager (https://github.com/borgmatic-collective/docker-borgmatic) and see if they’re willing to add it. 
 

I can think of several alternatives. Probably the simplest would be to invoke the command on the Unraid host via ssh. 
 

replace:

findmnt /mnt/removable > /dev/null || exit 75

 

with: (where 192.168.1.100 is your Unraid server IP)

ssh [email protected] findmnt /mnt/disks/borg_backup > /dev/null || exit 75

 

in your borgmatic config file and you should be in business.  If you haven't already, you will need to set up keyfile authentication with the Unraid host, so it doesn't ask for a password.  
 

Edited by sdub
  • Thanks 1
Link to comment
On 1/14/2022 at 7:31 AM, Reinis said:

Hi,

I'm trying to find an optimal solution for backing up my Linux/Mac computers to my Unraid box. Borg/borgmatic seems like a solid option based upon my research. I have Vorta installed on my Linux laptop and happily backing up to BorgBase.

 

Installed Borgmatic on Unraid box, though I am lost on how to configure and initialize the repositories.

 

I'd greatly appreciate any pointers on how to set it up.

 

Also, is Borgmatic a good solution for backing up a PC to Unraid, or it is meant more for backing up the Unraid box shares? Should I go with something like Duplicacy instead?

 

Thanks in advance!

 

You need to manually initialize the repo as described here:

https://torsion.org/borgmatic/docs/how-to/set-up-backups/#initialization

 

You can either follow my configuration guide in the first post in this thread or go right to the source for help with the config files:

https://torsion.org/borgmatic/docs/reference/configuration/

 

I think Borg is a great solution for any backup needs.  I currently have it running backups:

1.  From my Unraid array to a local backup drive

2. From my Unraid array to a remote Unraid server

3. From a windows PC running the Docker Borgmatic image

4. From a windows PC running borgmatic under WSL2

5. From a Macbook running Vorta

 

All of them are running very reliably.  Unless you're using Vorta, you have to be willing to mess with text-based config files, but it's really not that complicated.

 

I've gotten a Vorta docker container to run under Unraid, but I'm hesitant to publish it to the CA appstore since I don't actively use it.  I'm not really a developer, just an enthusiastic user!

Link to comment
On 1/19/2022 at 7:03 AM, sdub said:

 

You need to manually initialize the repo as described here:

https://torsion.org/borgmatic/docs/how-to/set-up-backups/#initialization

 

You can either follow my configuration guide in the first post in this thread or go right to the source for help with the config files:

https://torsion.org/borgmatic/docs/reference/configuration/

 

I think Borg is a great solution for any backup needs.  I currently have it running backups:

1.  From my Unraid array to a local backup drive

2. From my Unraid array to a remote Unraid server

3. From a windows PC running the Docker Borgmatic image

4. From a windows PC running borgmatic under WSL2

5. From a Macbook running Vorta

 

All of them are running very reliably.  Unless you're using Vorta, you have to be willing to mess with text-based config files, but it's really not that complicated.

 

I've gotten a Vorta docker container to run under Unraid, but I'm hesitant to publish it to the CA appstore since I don't actively use it.  I'm not really a developer, just an enthusiastic user!

Thanks for the tips, will give another shot at configuring Borg and report back.

Link to comment
On 1/18/2022 at 11:03 PM, sdub said:

 

 

 

I've gotten a Vorta docker container to run under Unraid, but I'm hesitant to publish it to the CA appstore since I don't actively use it.  I'm not really a developer, just an enthusiastic user!

 

There is a vorta docker on ca apps now, just an FYI. 

Link to comment

Thanks for bringing borgmatic to unraid, I have managed to get it working with unassigned drive.

 

The ultimate goal is to have a Raspberry Pi hosted remotely for backups - I have managed to do a test backup, but it looks like the encryption was done on the Pi?

 

My plan is to set a wireguard connection schedule to connect the Pi to my unraid, run borgmatic backup and then disconnect the pi from my unraid. 

 

What is the best way to:

  • Make sure the encryption is done on the more powerful unraid server
  • Connect and disconnect the Raspberry Pi Wireguard connection from the unraid server.

Furthermore, my healthchecks only show this in notification emails:

Quote

Starting a backup. Finished a backup.

Is there a way to have more detailed --stats output?

 

I have followed sdub's guide on creating persistent logs in appdata and can see the output. According to borgmatic documentation the notification should already contain some log up to 10kb? https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#healthchecks-hook

 

Link to comment
On 1/11/2022 at 2:42 PM, sdub said:

 

To be honest, I think I just copied the PATH statements from an example, and never messed with it.  It might not be strictly necessary, but note that cron runs as root, but not with the full root environment.  So if you have before backup or after backup hooks in your borgmatic config, they might fail to run without explicitly giving the path.

 

By default, borg/borgmatic just log their output to the console.  the "2>&1" at the end of the line redirects error output to the console.   In our Unraid Docker setup, these logs are viewable in the docker log window.

 

Recently, I decided I wanted proper logfiles, so I added the following to my docker config:

Untitled.thumb.png.0328ab3163a5381440478e3a446d76ba.png

 

Then I changed my cron command as follows:

/usr/bin/borgmatic -v 1 --files --stats > /mnt/logs/backup-"`date +"%a-%H"`".log

 

I perform backsups 2x daily, so this creates rolling logfiles named like:

backup-Mon-01.log
backup-Mon-13.log
backup-Tue-01.log
backup-Tue-13.log
...

 

Note that you could also use a date code formatted like `date +"%Y.%m.%d-%H"` and you'd get logfiles like:

backup-2022.01.11-01.log
backup-2022.01.11-13.log
backup-2022.01.12-01.log
backup-2022.01.12-13.log
...

 

and your log directory would just grow indefinitely.  Also note that I removed the 2>&1 from the end of the command.  This way the borg/borgmatic output is logged, but the errors are also shown on the docker log, which makes monitoring easier.

 

I should probably update the docker template and description to make this the default behavior at some point.

 

 

 

I have tried to set this up but it does not seem to work:

20 4 * * 1,4,6,7 PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --files --stats > /mnt/logs/backup-"`date +"%a-%H"`".log
20 4    * * 3 PATH=$PATH:/usr/bin /usr/bin/borgmatic check -v 1 2>&1

First added the statement with notepad but read elsewhere that it might be causing issues so recreated the file in VSC, the issue persists. 

 

image.png.05e6ea8cc09ef597ec3fb9788510b928.png

 

Logs are created but with a strange name:

image.thumb.png.ca77fce5dea18b6cc0136e364c06ada6.png

Docker log is full of cron errors as well 

image.thumb.png.d42bdd7eca5aaceda917b1e61601c7b9.png

 

Any idea what is going wrong?

 

Link to comment
10 minutes ago, kilonde said:

I have tried to set this up but it does not seem to work:

20 4 * * 1,4,6,7 PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --files --stats > /mnt/logs/backup-"`date +"%a-%H"`".log
20 4    * * 3 PATH=$PATH:/usr/bin /usr/bin/borgmatic check -v 1 2>&1

First added the statement with notepad but read elsewhere that it might be causing issues so recreated the file in VSC, the issue persists. 

 

image.png.05e6ea8cc09ef597ec3fb9788510b928.png

 

Logs are created but with a strange name:

image.thumb.png.ca77fce5dea18b6cc0136e364c06ada6.png

Docker log is full of cron errors as well 

image.thumb.png.d42bdd7eca5aaceda917b1e61601c7b9.png

 

Any idea what is going wrong?

 

I mean, the logs say "error at 1,4,6,7"

 

Looking at the part "1,4,6,7" in your crontab.txt file, I do not understand what this is here for and Cron does not understand what it is.

Link to comment
5 minutes ago, Solverz said:

I mean, the logs say "error at 1,4,6,7"

 

Looking at the part "1,4,6,7" in your crontab.txt file, I do not understand what this is here for and Cron does not understand what it is.

Thanks, I replaced it with 20 4 * * 1,4,6,0 and it appears to have worked! 7 is not Sunday :)

 

 

Link to comment
  • 2 weeks later...
On 1/18/2022 at 11:44 PM, sdub said:


Docker containers are by design minimal and not full Linux distros. You could open an issue with the container manager (https://github.com/borgmatic-collective/docker-borgmatic) and see if they’re willing to add it. 
 

I can think of several alternatives. Probably the simplest would be to invoke the command on the Unraid host via ssh. 
 

replace:

findmnt /mnt/removable > /dev/null || exit 75

 

with: (where 192.168.1.100 is your Unraid server IP)

ssh [email protected] findmnt /mnt/disks/borg_backup > /dev/null || exit 75

 

in your borgmatic config file and you should be in business.  If you haven't already, you will need to set up keyfile authentication with the Unraid host, so it doesn't ask for a password.  
 

 

Would you be able to point me in the right direction for setting up keyfile authentication? I tried googling but wasn't able to find much on running ssh from inside an unraid docker container.

 

I found that running this command instead of `findmnt` seems to work sometimes

test -d mnt/backup_1_repo/data > /dev/null || (echo "Backup disk 1 not found, failing over" && exit 75)

 

This works correctly when I run borgmatic manually inside the container, However, for some reason it's not working correctly when running from my crontab -- the soft failure always occurs, even when the disk is mounted. So I'm going back to trying the ssh/findmnt suggestion to see if that will work any better.

Link to comment

Can this container be used as the server/target for backing up clients? I want to back up my clients to my Unraid server. I previously had this working with Synology as the target, but the Synology Borgmatic app runs natively rather than in a container, so there's no Docker layer. 

 

The container doesn't appear to be configured with any port mappings by default. I tried adding a port mapping so I could SSH into the container but couldn't get it to work. I checked the permissions on the SSH folder and files without success. Can someone point me in the right direction?

Edited by WalkerJ
Link to comment
6 minutes ago, WalkerJ said:

Can this container be used as the server/target for backing up clients? I want to back up my clients to my Unraid server. I previously had this working with Synology as the target, but the Synology Borgmatic app runs natively rather than in a container, so there's no Docker layer. 

 

The container doesn't appear to be configured with any port mappings by default. I tried adding a port mapping so I could SSH into the container but couldn't get it to work. I checked the permissions on the SSH folder and files without success. Can someone point me in the right direction?

Unsure, "should" be possible as it just has borg installed, however do not know enough about how the container was buikt to confirm this.

 

BUT, you can install borgbackup directly to unraid using the nerd pack plugin and then set a script or somthing to run the borg backup server command on boot or array start.

Link to comment
4 hours ago, Solverz said:

Unsure, "should" be possible as it just has borg installed, however do not know enough about how the container was buikt to confirm this.

 

BUT, you can install borgbackup directly to unraid using the nerd pack plugin and then set a script or somthing to run the borg backup server command on boot or array start.

This is what I’ve done… I have 2 Unraid servers that back up to each other. The backups are initiated from within docker containers but connect to borg binaries that I installed natively to Unraid using the nerd pack. 
 

Unfortunately the b3vis docker container does not include an SSH server, only the client.   Without that I’m not sure there’s a way for the remote machine to access the borg binary inside the local docker or Vice versa. 

  • Like 1
Link to comment

I'm mostly an Unraid newbie. I'm trying to automate borg backups from my laptop to my Unraid box. From what I understand this should better be done using a docker rather than by directly running a borg server on the main Unraid OS. Am I correct that I could use the borgmatic docker to achieve this, since it already has borgbackup and some ssh infrastructure?
 

I'd be grateful for any pointers regarding how to set up the docker to do this. Should I use my Unraid box's local IP address with a specific port to reach the docker? Where do I define this port? Should I generate SSH keys from inside the docker and save the public key to my laptop, or the other way around? The answers to these questions might be obvious but not to me, and Google has failed me so far. Thanks in advance for any advice.

 

- Mathieu

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.