[Support] borgmatic


sdub

Recommended Posts

I haven't monitored the whole conversation. But first let's make sure your configuration works and you can backup files. To do so, please open the console of the borgmatic container and execute a backup yourself via  "/usr/bin/borgmatic prune create -v 1 --stats 2>&1" (without the ""). For testing purpose only include a small file in your backup via the `config.yaml`. Let me know how this goes.

 

Just to make sure: Do you know how cron works and what the entries mean? The backup is only triggered when the cron expression evaluates to true. You may have to restart the container after making changes to the crontab.txt file.

Edited by T0a
Link to comment

ISSUE: Cron not running

 

When I console into the borgmatic container I can run backups fine.  I can even paste the cron entry, and it works, the cron is functioning as I monitored the logs just now, and it tried to run the cron again.

 

If i paste this into the console it works and backs up correctly.

PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1

 

 

borgmatic_cron.png

Edited by Greygoose
added colour header RED
Link to comment

FIXED: CRON schedule not running

 

Ok sorted it.

 

I recreated the crontab.txt file in the docker console using vi.

 

Now the cron is working. It must have been a permissions issue, I normally edit the txt file in notepad+ in my Windows 10 machine via the shared network drive.   I will play around and see how I caused it and report back

Edited by Greygoose
added header colour green
  • Thanks 2
Link to comment

Extracting archives

 

Just in case anyone else has this issue, when extracting archives using the borgmatic command

(make sure you in the extract folder) - I created another mount point in the borgmatic docker to /mnt/extract

 

borgmatic extract --archive myarchivename

 

bormatic will look into the config.yaml files, I had my config to back up folder 1 to borgbase repo first then locally.  I wanted to restore from the local backup, however the command above kicked up issues as I had the config file set to back up to the borgbase repo first before the local location.  When I change the order in the config file it restores. 

 

 

 

Link to comment

I'm hoping someone can help clarify the config.yaml and crontab.txt options that I've placed in my /appdata/borgmatic/config folder. I've created my own version of config.yaml based on the master configuration reference. 

 

Now I'm stuck on the crontab.txt. 

0 1 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1

What is /usr/bin? Do I need to change '/usr/bin' to '/appdata/borgmatic' ? Also, will the borgmatic docker container automatically run the crontab itself?

I should also clarify, I don't have a backup .sh script, just the config.yaml file. I'm hoping the cronjob will run this file.

Thank you for your help!

Link to comment

@Rick_Sanchez

 

You only need to change the cron schedule,

 

Please see my schedule below, the first line has been changed so that it runs twice daily at 1am and 1pm. The second line is as per the original post and set to run weekly at 6am.

 

Quote

0 1,13 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1
0 6   * * 3 PATH=$PATH:/usr/bin /usr/bin/borgmatic check -v 1 2>&1

 

The path does not need to be changed, this is the configuration within the docker and borgmatic.

 

Please also see this link for further details on the cron setup, please let me know how it goes.

https://linuxize.com/post/scheduling-cron-jobs-with-crontab/

Link to comment

@Rick_Sanchez

 

To follow on from your post.

 

Borgmatic will read the crontab.txt automatically, this is part of the docker setup.  When you change the commands in crontab.txt you will need to restart the docker to load the changes.

 

If you console into your borgmatic docker you can test your config files by simply typing borgmatic. borgmatic will read the .yaml files in the order they are list "alphabetically". I have 2 .yaml files, nextcloud.yaml and personal.yaml  Both of these run two different backups.

 

I typically use the following command in console to see if everything is running smoothly

borgmatic --verbosity 2

 

Also this to list my archives

borgmatic --list

 

Please let me know how you get on with everything, any further questions please ask. 

Edited by Greygoose
Grammer
  • Thanks 1
Link to comment

Hi Rick,

 

Thats great.

 

When running the borgmatic command i dont believe this shows in the logs.

 

However when your cron scedule runs this will show in the docker log (click the docker icon and click logs).

 

I recommend using the heathchecks.io function, this allows push notifications/emails on backup up times. It can be set to alert you if the backup fails.

 

I also recommend borgbase for online backups, their website is full of borg and borgmatic information.

Link to comment

Hey @Greygoose - I setup my crontab.txt as follows, which is found in /appdata/borgmatic/config/ (note, this is the exact file - I have not changed any PATH directories)

0 1 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1
0 6 * * 3 PATH=$PATH:/usr/bin /usr/bin/borgmatic check -v 1 2>&1

The docker log output:

crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1
/bin/ash: redir error

I thought it may be a permissions issue, but the txt file is set as -rw-rw-rw-. 

 

I did find a post where someone put this in their crontab, but i'd prefer to use the proper method if possible:

0 6 * * * borg create -v --stats --compression lz4 /backup_repository::$(date +'%Y-%m-%d') /backup_sources

 

Link to comment

I’m going to try and sudo nano the crontab.txt and see if that fixes the /bin/ash error I’m getting. I just read the comments above.

 

Update: I've tried using the unraid terminal to nano the file and save it, and I've tried running the borgmatic container as an admin, but i'm still getting the 

crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1
/bin/ash: redir error

Anyone have tips on how to fix this? Does the file need to be accessible to the 'root' user of unraid?

 

The Fix: Same as above. 

  1. Copy contents of crontab.txt
  2. move current crontab.txt file
    mv /path/to/crontab.txt /path/to/crontab.txt.old

     

  3. ssh into unraid, or, use the unraid terminal
  4. nano /path/to/crontab.txt
  5. Paste contents of crontab.txt
  6. Restart borg container
Edited by Rick_Sanchez
Updates
  • Thanks 2
Link to comment
  • 1 month later...
On 12/24/2020 at 8:25 AM, Greygoose said:

The above post was my error.

 

The creation of the key needs to be done in the docker container, as explained on the borgbase site.

https://docs.borgbase.com/faq/ssh/

 

Now working with borgbase without any issues.

What did you put in "Borg Repo (Backup Destination):"  ? since you dont use local repo ?

Edited by ZappyZap
Link to comment

hi together,
I would very much like to create a simple backup with the Borgmatic. Borgmatic is installed, all paths are entered and the init is performed. so far so good. I would now like to create a daily backup. the backup should then only new or deleted files to the backup repo add / remove.

Can someone help me how I have to create the corn and config file for this?

I would be very happy about your help on this.

 

greetings kilian

Link to comment
  • 2 weeks later...

Hi,

is it possible to mount an unassigned device just before borg starts and unmount the drive after all is finished? I have a separate drive for backups and would like (security/safety) to have access just for backup/restore run.

I tried from inside container, but it seems there are not enough rights for mounting the drive from there. Hope there is a workaround.

Thanks...

Link to comment
1 hour ago, HagenS said:
Hi,
is it possible to mount an unassigned device just before borg starts and unmount the drive after all is finished? I have a separate drive for backups and would like (security/safety) to have access just for backup/restore run.
I tried from inside container, but it seems there are not enough rights for mounting the drive from there. Hope there is a workaround.
Thanks...


There are hooks in the Borgmatic config file for “before_backup” and “after_backup” that could be used to invoke an SSH command to tell the Unraid parent to mount/unmount volumes.

For example (where Unraid's IP is 192.168.1.6)

before_backup:
ssh 192.168.1.6 mount /dev/sdj1 /mnt/disks/borg_backup

after_backup:
ssh 192.168.1.6 umount /mnt/disks/borg_backup

 

Edited by sdub
  • Like 1
  • Thanks 1
Link to comment
  • 1 month later...
On 12/16/2020 at 10:04 AM, T0a said:

No, I totally get your point and your arguments seems reasonable to me. I run a docker container with a filesystem-based SQLite database that has no build-in database export like you mentioned. That's why I asked this question in the first place.

 

I installed your container today and I really like it so far. Up until now, I did my offsite backups to Google drive via rclone. However, this solution didn't let me sleep well to be honest - especially the Google drive part. So, I ordered a storage box from Hetzner today and did my first backup to it with borgmatic!

 

The last piece missing is stopping the docker container I mentioned above. The plan is to use "HA dockermon"  from within the borgmatic container. Would you mind adding curl to the docker container for me? Then, I would be able to stop any container via:


curl -v -X POST <ha_dockermon_ip>:8126/container/container_name --header 'content-type: application/octet-stream' --data '{"state": "stop"}'

Thanks for bringing borgmatic to my attention :)

 

This is really amazing, thank you @T0a and @sdub

I didn't know about the HA dockermon, like T0a, I'm looking to stop some containers before doing the backup with borgmatic, and start the container after the backup is done.

Link to comment
On 12/21/2020 at 10:44 AM, Greygoose said:

@sdub

I have an issue which i'm not sure is my configuration or is not possible at all.

 

I have the waseh rclone plugin installed, unraid is set to boot and mounts the OneDrive folder using the user scripts. I then have the folder set as a repository in the Borgmatic docker, I can see the folder mounts correctly in Borgmatic console.,

 

When I start the backup in borgmatic after init a new repo on the OneDrive rclone folder it backs up for a few minutes then the array crashes and I need to reboot.  When I login to onedrive I can see the backup has started to upload onto the microsoft servers.  I have tried a few times and always the same outcome.

 

 

@Greygoose below is the config I have on my User Scripts to mount rclone, in my case I'm using Google Drive, but it is working fine, no errors so far.
Keep in mind that you have to set up a folder for rclone cache, and use the vsf-cache-mode full.

image.thumb.png.6def4a0640b8893735d10d1643ff67ea.png

 

I'm using inside the config.yaml the repo (/mnt/borg-repository)

location:
    source_directories:
        - /mnt/user/sourcerepo1
    repositories:
        - /mnt/borg-repository
    one_file_system: true
    files_cache: mtime,size

But in the docker template I point the repo to the rclone mounted (in my case secure)

image.png.26be82ae482fcfcebbbe1ddb7ff68b1f.png

 

I hope these screenshots help others... 👍

Link to comment

@sdub

- Can you see if there is a way to add log file to borgmatic yaml config?
 

- Is there a place in the docker where I could see all logs from borgmatic and/or borg?

I went to look for all logs in the docker and couldn't find any in /var/log/

 

- Is there a way to send all logs from both applications to a syslog server?

 

Also, I tried to use the validate-borgmatic-config but the docker can't find it, I think there is something missing there..

Link to comment
  • 2 months later...

I'm slightly confused about how I'm supposed to init the borg repo. Do I do it from within the borgmatic container, or do I do it in UnRaid through the Nerdpack tools?

I'm trying to do a manual backup from within the container after initing the repo in UnRaid itself, but I'm getting the following errors:

Quote

/ # borgmatic
Starting a backup.
/mnt/borg-repository: Error running actions for repository
Command 'borg prune --keep-hourly 2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --prefix backup- /mnt/borg-repository' returned non-zero exit status 2.
Error during prune/create/check.
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/mnt/borg-repository: Error running actions for repository
Failed to create/acquire the lock /mnt/borg-repository/lock (timeout).
Command 'borg prune --keep-hourly 2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --prefix backup- /mnt/borg-repository' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

 

My config is as follows:

location:
    source_directories:
        - /boot
        - /mnt/user/appdata
        - /mnt/user/Backup
        - /mnt/user/NextCloud
        - /mnt/user/Files
    repositories:
        - /mnt/borg-repository
    one_file_system: true
    files_cache: mtime,size
    patterns:
        - '- [Tt]rash'
        - '- [Cc]ache'
    exclude_patterns:
        - /mnt/user/Files/Games
    exclude_if_present:
        - .nobackup
        - .NOBACKUP

storage:
    encryption_passphrase: "***"
    compression: zstd
    ssh_command: ssh -i /root/.ssh/id_rsa
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 2
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 10
    prefix: 'backup-'

consistency:
    checks:
        - repository
        - archives
    prefix: 'backup-'

hooks:
    before_backup:
        - echo "Starting a backup."
    after_backup:
        - echo "Finished a backup."
    on_error:
        - echo "Error during prune/create/check."

 

Link to comment
5 hours ago, TrikkStar said:

I'm slightly confused about how I'm supposed to init the borg repo. Do I do it from within the borgmatic container, or do I do it in UnRaid through the Nerdpack tools?

I'm trying to do a manual backup from within the container after initing the repo in UnRaid itself, but I'm getting the following errors:

 

 

 

The specific reason for your error appears to be that "/mnt/borg-repository" doesn't exist in the environment where you're running the borgmatic command.

 

Everything should be done from within the borgmatic container... as a matter of fact you don't even need to install the nerdpack borg tools at all in Unraid.  This docker container is the only thing you need. 

 

The whole point of this CA package was to allow you to keep a "stock" Unraid installation, which does not include borg or borgmatic.

 

 

  • Thanks 1
  • Upvote 1
Link to comment
  • 2 weeks later...

Hi at all. 

 

First I have to say that I use unRAID only since two weeks and therefore I'm new and inexperienced.

 

In search of a suitable backup solution for me I found borgmatic and I absolutely love it. For me it's the best way to backup all my important stuff in a secure way. Many thanks for making this possible!

 

At the moment I'm using CA Backup to backup my appdata folder which is located on my cache drive. But for some other schedules it is really annoying that all containers has to be stopped during the backup. I already know that there is a way not to stop some containers but I'm searching for a easier solution. Therefore I would like to backup my appdata folder with borgmatic. 

 

But there are a few things I have not yet understood like why Docker containers are not allowed to run during a backup of the appdata folder. What can happen if they run in worst case and why?

 

I need this information to contrast the potentially risk with the advantages of backing up the appdata Folder with borgmatic without stoping the containers.

 

On 11/22/2020 at 9:51 PM, T0a said:

How do you make sure that files are not getting written by your docker containers while the backup is running? The CA backup stops containers to prevent file corruption AFAIK. I cannot see such a mechanism in your solution. Technically, this would be possible with the before_backup and after_backup hooks.

 

This topic has already been addressed but until now I couldn't find a solution to stop containers with before_backup and after_backup hooks. I wonder if there is an easy way to control docker daemon on unRAID from inside of a docker container.

 

Thanks in advance and sorry for the possibly stupid questions.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.