[Support] borgmatic

RinxKninks · December 30, 2022

1 hour ago, Solverz said:

Please just say what you exactly want to do without using prolonged sentences and fancy words. Your post is like a riddle that we'd need to break down to understand what you are talking about.

Plain & simple: I started using borgmatic writing to rotating removable hard drives, each containing another repositorie.

With the same "run" of borgmatic I'd like to copy data already being deduplicated etc. to this target disks, that I'd like not to process by the borgmatic backup process again. So I think about rsnapshot.

But the borgmatic docker doesn't have access to the rsnapshot bin on the unraid system - which I understand, nor can I just copy and use rsnapshot into the borgmatic docker.

So do you have an idea on how to use rsnapshot as a after_backup hook in the borgmatic/config?

Edited December 30, 2022 by RinxKninks

Solverz · December 30, 2022

13 minutes ago, RinxKninks said:

Plain & simple: I started using borgmatic writing to rotating removable hard drives, each containing another repositorie.

With the same "run" of borgmatic I'd like to copy data already being deduplicated etc. to this target disks, that I'd like not to process by the borgmatic backup process again. So I think about rsnapshot.

But the borgmatic docker doesn't have access to the rsnapshot bin on the unraid system - which I understand, nor can I just copy and use rsnapshot into the borgmatic docker.

So do you have an idea on how to use rsnapshot as a after_backup hook in the borgmatic/config?

So you have two repos, each on a separate external hard drive? Got that.

Not sure what you mean on the rest?

Quote

copy data already being deduplicated etc.

What do you mean here?

Edited December 30, 2022 by Solverz

Solverz · December 30, 2022

1 hour ago, RinxKninks said:

Plain & simple: I started using borgmatic writing to rotating removable hard drives, each containing another repositorie.

With the same "run" of borgmatic I'd like to copy data already being deduplicated etc. to this target disks, that I'd like not to process by the borgmatic backup process again. So I think about rsnapshot.

But the borgmatic docker doesn't have access to the rsnapshot bin on the unraid system - which I understand, nor can I just copy and use rsnapshot into the borgmatic docker.

So do you have an idea on how to use rsnapshot as a after_backup hook in the borgmatic/config?

Borgbackup, backs up to a repo. You can choose what you want to be backed up to this repo.

So what what do you need to use rsnapshot for that borgbackup is not doing already.

RinxKninks · December 30, 2022

2 hours ago, Solverz said:

Borgbackup, backs up to a repo. You can choose what you want to be backed up to this repo.

So what what do you need to use rsnapshot for that borgbackup is not doing already.

OK - got it.

Revan335 · January 1, 2023

What's the URL for a Client (Vorta for example) Connection to the Repository?

No GUI or Webui?

Edited January 1, 2023 by Revan335

sdub · January 1, 2023

The Borgmatic image does not have ssh built in, so you’d need install Borgmatic using nerd tools plug-in, then you could use “ssh://user@host/path/to/repo”

no built in gui because the config files are pretty straight forward to configure, and something intended to be set and forget.

Edited January 1, 2023 by sdub

Greygoose · January 1, 2023

No gui needed, as sdub said once the configs are setup its leave alone and let it do its thing.

The most i need to do is break-lock if i restart or stop the docker during the backup process.

Revan335 · January 2, 2023

Its only worked its the SSH Service and Default SSH Port enabled and not working with a different for example the rsync Docker Container?

sdub · January 2, 2023

17 minutes ago, Revan335 said:

Its only worked its the SSH Service and Default SSH Port enabled and not working with a different for example the rsync Docker Container?

that won’t work since that container doesn’t have the borg binary.

you could try something like the nold360/borgserver container, which is basically just borg and ssh. Just feed in the path to your borg repo.

Revan335 · January 2, 2023

8 hours ago, sdub said:

that won’t work since that container doesn’t have the borg binary.

you could try something like the nold360/borgserver container, which is basically just borg and ssh. Just feed in the path to your borg repo.

Many Thanks!

I tried!

The Vorta Docker doesn't work to, correct?

Revan335 · January 2, 2023

13 hours ago, sdub said:

you could try something like the nold360/borgserver container, which is basically just borg and ssh. Just feed in the path to your borg repo.

OK, new Message.

Error: Repository path not allowed.

ssh://borg@Unraidname:Port/Borg/backup/Clientname

The Permissions of 1000:1000 are correct.

~# ls -la /mnt/user/Borg/
total 0
drwxrwxrwx 1 nobody users  26 Jan  2 13:59 ./
drwxrwxrwx 1 nobody users 172 Jan  2 14:47 ../
drwxr-xr-x 1   1000  1000  10 Jan  2 14:52 backup/
drwxr-xr-x 1   1000  1000  22 Jan  2 14:46 sshkeys/

Revan335 · January 4, 2023

Anybody a Idea why the Repository path not allowed?

Revan335 · January 8, 2023

Found the crucial hint in the Ubuntuusers Wiki.

// is important for the path. Either this is not in the docs of Vorta/Pika or Borg sites or I must have over seen it despite multiple checks.

ssh://borg@Unraidname:Port//backup/Clientname

Test it now further.

nicjames2378 · January 8, 2023

FIXED: This issue was caused by the mounted /root/.ssh path on the borgmatic end being protected. After making a new path within appdata and copying the necessary files from UnRaid's /root/.ssh, everything works without unexpected errors. I've left the below errors in case it helps someone else in the future.

I've got this working using b3vis/borgmatic:latest-msmtp on the client being backed up, and using nold360/borgserver:1.2.2 on the remote server. The only issue I have encountered (once it was all setup correctly, that is) is that each emailed report from the msmtp comes through as "Failed" due to an error. I know this isn't meant to be a borg support forum, but I've only encountered this while using this container so perhaps someone here knows what's going on?

I assumed that since the issues seems to be due to "Operation not permitted", it was a permissions problem. However, I can copy the file via command line when I docker exec -it into the local bormatic container with no problems, so that does not seem to be the case?

Remote: hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted
Remote: update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted
ssh://[email protected]:PORT/backup/SERVERNAME: Error running actions for repository
Command 'borg check --glob-archives backup-* ssh://[email protected]:PORT/backup/SERVERNAME' returned non-zero exit status 2.
Error during prune/create/check.
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
ssh://[email protected]:PORT/backup/SERVERNAME: Error running actions for repository
Remote: hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted
Remote: update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1039, in check
    objects = list(self.io.iter_objects(segment))
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1512, in iter_objects
    size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1606, in _read
    data = fd.read(length)
OSError: [Errno 5] Input/output error
Borg server: Platform: Linux 1e2c94ae5f5a 5.19.17-Unraid #2 SMP PREEMPT_DYNAMIC Wed Nov 2 11:54:15 PDT 2022 x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.3  Python: CPython 3.10.9 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 343  CWD: /backup/SERVERNAME
Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--restrict-to-path', '/backup/SERVERNAME']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve'
Platform: Linux 990a970c11d1 5.15.46-Unraid #1 SMP Fri Jun 10 11:08:41 PDT 2022 x86_64
Linux: Unknown Linux
Borg: 1.2.2  Python: CPython 3.10.8 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 108  CWD: /
sys.argv: ['/usr/local/bin/borg', 'check', '--glob-archives', 'backup-*', 'ssh://[email protected]:PORT/backup/SERVERNAME']
SSH_ORIGINAL_COMMAND: None
Command 'borg check --glob-archives backup-* ssh://[email protected]:PORT/backup/SERVERNAME' returned non-zero exit status 2.

Edited January 20, 2023 by nicjames2378
Found solution to my problem.

DizkoDan · February 17, 2023

First, thanks for this. I've been using it for a long time, and it's been pretty much rock solid.

Just wanted to make a note here, cuz I hadn't seen it anywhere. They apparently moved the image for this from dockerhub to google a few weeks ago. If you look at your containers in unraid, you'll notice an error for this one not being found. Just update the Repository to the following, and you can grab 1.76.

http://ghcr.io/borgmatic-collective/borgmatic

sdub · February 17, 2023

1 hour ago, DizkoDan said:

First, thanks for this. I've been using it for a long time, and it's been pretty much rock solid.

Just wanted to make a note here, cuz I hadn't seen it anywhere. They apparently moved the image for this from dockerhub to google a few weeks ago. If you look at your containers in unraid, you'll notice an error for this one not being found. Just update the Repository to the following, and you can grab 1.76.

http://ghcr.io/borgmatic-collective/borgmatic

Thanks... it looks like they're still pushing the updates to dockerhub (at least for now), but I updated the template as suggested. (I think you meant to say they moved from dockerhub to github)

I'm not sure if template updates affect existing containers. To move existing containers to the new address, open the borgmatic container settings, go to advanced and change the following lines:

Repository: ghcr.io/borgmatic-collective/borgmatic

Docker Hub URL: https://ghcr.io/borgmatic-collective/borgmatic

Icon URL: https://github.com/borgmatic-collective/borgmatic/raw/master/docs/static/borgmatic.png

Edited February 17, 2023 by sdub

DizkoDan · February 17, 2023

Correct, I mean github, used to dealing with google repo, so missed the h there.

I most definitely was not getting the updated version. Unraid changed to showing repo not found or something similar at some point in the past couple of weeks. I checked the log output of the previous run, and it was still showing borgmatic 1.7.5 for the version.

kilonde · February 25, 2023

I am confused how to update my repo path in the configuration to avoid this error:

Remote repository paths without ssh:// syntax are deprecated.

My repo configuration is like this:

[email protected]:/mnt/disk/borg_backup/repo

and the ssh command looks like this:

    ssh_command: ssh -i /root/.ssh/Pi

Also, is it advised to keep the same version of borg on client and the server?

witten · March 9, 2023

On 2/25/2023 at 2:20 PM, kilonde said:
I am confused how to update my repo path in the configuration to avoid this error:
Remote repository paths without ssh:// syntax are deprecated.
My repo configuration is like this:
[email protected]:/mnt/disk/borg_backup/repo
and the ssh command looks like this:
    ssh_command: ssh -i /root/.ssh/Pi

"[email protected]:/mnt/disk/borg_backup/repo" would be rewritten as "ssh://[email protected]/mnt/disk/borg_backup/repo". Note both the "ssh://" and the removed ":" after the IP.

Newer versions of borgmatic will show you this as part of the deprecation warning:

test.yaml: Remote repository paths without ssh:// syntax are deprecated. Interpreting "[email protected]:/mnt/disk/borg_backup/repo" as "ssh://[email protected]/mnt/disk/borg_backup/repo"

On 2/25/2023 at 2:20 PM, kilonde said:

Also, is it advised to keep the same version of borg on client and the server?

It doesn't have to be exactly the same, but it's good to stay within the same major version if you can. (E.g. 1.x and 1.x.) There are some exceptions to that, like there were some major changes in Borg 1.2.

Edited March 9, 2023 by witten

kilonde · March 25, 2023

On 3/9/2023 at 1:30 AM, witten said:
"[email protected]:/mnt/disk/borg_backup/repo" would be rewritten as "ssh://[email protected]/mnt/disk/borg_backup/repo". Note both the "ssh://" and the removed ":" after the IP.

Newer versions of borgmatic will show you this as part of the deprecation warning:
test.yaml: Remote repository paths without ssh:// syntax are deprecated. Interpreting "[email protected]:/mnt/disk/borg_backup/repo" as "ssh://[email protected]/mnt/disk/borg_backup/repo"
It doesn't have to be exactly the same, but it's good to stay within the same major version if you can. (E.g. 1.x and 1.x.) There are some exceptions to that, like there were some major changes in Borg 1.2.

Thanks for your help, this has worked!

bastl · April 12, 2023

First of all, thanks for this container.

I have a question @sdub

In your second post of this thread you showed your config as an example which I used to test it on my system. I have an issue with it. As long as I keep the prefix line in the retention block

retention:
    keep_hourly: 2
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 10
    prefix: 'backup-'

I get the following error during sheduled backups:

/mnt/borg-repository: Error running actions for repository
Command 'borg prune --keep-hourly 2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --glob-archives backup-* --glob-archives backup-* --list /mnt/borg-repository' returned non-zero exit status 2.
Error during prune/create/check.
/etc/borgmatic.d/config.yaml: An error occurred

borg prune: error: argument -a/--glob-archives: There can be only one.
Command 'borg prune --keep-hourly 2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --glob-archives backup-* --glob-archives backup-* --list /mnt/borg-repository' returned non-zero exit status 2.

The argument "--glob-archives backup-*" is added 2 times somehow.

Why do you add it at the retention section in your yaml file? Is it really needed or is this line save to remove?

Gaurhoth · April 12, 2023

8 hours ago, bastl said:

The argument "--glob-archives backup-*" is added 2 times somehow.

A recent update changed the way prefix option works. Generally, you can replace multiple prefixes (for example if you had same "prefix:" under retention and consistency) with archive_name_format under storage.

Example:

[old way]

storage:
    archive_name_format: 'homepage-{now}'
retention:
    prefix: 'homepage-'
consistency:
    prefix: 'homepage-'

Can be replaced with:

[new way]

storage:
    archive_name_format: 'homepage-{now}'

If you need different 'prefix' values for some reason you can look at the match_archives option under storage. This change is documented at https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#code-35

witten · April 14, 2023

While Gaurhoth is correct that you can update your configuration to drop the deprecated "prefix" option with recent versions of borgmatic, the fact that "prefix" started breaking as described above was actually a bug. It has been fixed however in borgmatic 1.7.12.

darkside40 · April 15, 2023

Was something recently changed in this Image?

Using it for over a year but in the past couple of weeks it seems that the cron runs quite random. Normally it should rum at 18:00 every day, but now it sometimes simply runs the script when the server comes from sleep etc.

Anyone also experienced this?

witten · April 15, 2023

3 hours ago, darkside40 said:

Was something recently changed in this Image?

Using it for over a year but in the past couple of weeks it seems that the cron runs quite random. Normally it should rum at 18:00 every day, but now it sometimes simply runs the script when the server comes from sleep etc.

Anyone also experienced this?

The docker-borgmatic image this is based on switched to supercronic a while back. But presumably if that introduced the changes you're seeing, you would've seen it well before now. You might consider filing a ticket describing your problem at: https://github.com/borgmatic-collective/docker-borgmatic

[Support] borgmatic

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

sdub

sdub

Greygoose

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation