sdub
-
Posts
88 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Posts posted by sdub
-
-
3 hours ago, xxnumbxx said:
Does anyone know how Borg works in regard to deduplication, specific to files of the same name. I am curious of photo backups specifically. Will it determine files of the same name and check if the are different files. My camera repeats the names of files and I want to make sure all photos are retained regardless of file names.
The filename makes no difference at all. What matters is the binary content of the files.
Borg breaks up files into 2MB "chunks" (by default) that are deduplicated irrespective of metadata like filename or path. In this way even large files that have been partially changed have the possibility of partial deduplication, and renamed files will be fully deduplicated. In your example, different files with the same file name would both be broken into chunks and fully retained in the backup. When the last backup archive that uses a chunk is purged, the chunk is purged also.
Reading up on the "chunker" might help you get an appreciation. I've found (as a borg user) it's very efficient.
https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#chunker-details
-
1
-
-
I struggled with something similar and i think the issue is once it’s in the cache it won’t stop backing it up. Only applies going forward for new folders.
I bet if you do a simple test on a new repo with that folder it will work as expected.Might want to ask the guys over at the borg GitHub repo since it’s a borg behavior that’s causing it.
-
1
-
-
I think the issue is that you created the borg repo then moved it, or are trying to access it using a different path.
Run this borg command manually and confirm that you’re ok with the move. Then try running Borgmatic again.
borg list 10.10.1.184:/mnt/user/borg-
1
-
-
I’ve never successfully exposed a fuse mount in one docker container to another. I usually perform browse and restore operations from the command line within the Borgmatic docker
A couple of options that may work are:
1) install vorta container, which gives a borg gui so you can at least visually browse the archive, even if it’s not really a file manager.
2) install borg and llfuse to the base Unraid image using nerdpack. I would think this should work but I’ve never gotten the fuse mount to appear via samba or a file manager docker container. Not sure why because I think it should work.
3) install an Ubuntu VM with borg and llfuse and try to mount it there, browsing the archive visually through VNC. I haven’t actually tried this one yet.
Let us know if you figure out a working solution!
-
48 minutes ago, donbruno said:
and the repro /mnt/borg-repository/NAS2 is a remote nfs mount point from another NAS
I wonder if there is some lower level error happening with the NFS mount. I would start with running:
borg prune --verbose --keep-hourly 2 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --prefix backup- /mnt/borg-repository/NAS2to see if there are any more detailed error messages (note the --verbose) you can get from borg (bypassing the borgmatic script altogether)
This might be a better question to ask in the borg github issues page to see if they can help understand the specific borg error.
-
12 minutes ago, donbruno said:
ok, I delete all images... but when I pull the borgmatic repo, I see that the container with id df9b9388f04a
was not pulled, it still exits... 😞 !?!?!
How can I delete it?
From the command line:
docker image remove df9b9388f04abut I seriously doubt the image is corrupted. A little more context on the error would be helpful. It's probably an error in the config.yaml, the repo can't be contacted (especially if it's a remote repo), or there's a stale lock.
Can you execute the following command from within the container?
borg list /path/to/repo -
7 hours ago, Solverz said:
Stop using CA-Backup and use borg exclusively.
If you’re mapping in /mnt/user it should already be there… you just need to add to the config.yaml
otherwise yes just add a new path mapping into the container and add that container path to borgmatic’s config.yaml
-
6 hours ago, Solverz said:
Stop using CA-Backup and use borg exclusively.
Agree… no reason you can’t back up /boot and /mnt/use/appdata with borg. I don’t like CA appdata because the backup files are huge and not very friendly with deduplication. I’ve used my borg archive to restore Plex multiple times without issue.
With daily backups the risk of corruption on successive backups is limited, especially if you’re using the built-in DB backup capabilities of the various docker containers.
-
1
-
-
On 2/19/2022 at 11:23 AM, -Mathieu- said:
I'm mostly an Unraid newbie. I'm trying to automate borg backups from my laptop to my Unraid box. From what I understand this should better be done using a docker rather than by directly running a borg server on the main Unraid OS. Am I correct that I could use the borgmatic docker to achieve this, since it already has borgbackup and some ssh infrastructure?
I'd be grateful for any pointers regarding how to set up the docker to do this. Should I use my Unraid box's local IP address with a specific port to reach the docker? Where do I define this port? Should I generate SSH keys from inside the docker and save the public key to my laptop, or the other way around? The answers to these questions might be obvious but not to me, and Google has failed me so far. Thanks in advance for any advice.
- Mathieu
In general, I like to keep Unraid as "stock" as possible and install any applications as containers. This is generally viewed as a best practice. Note, however that the borgmatic container is intended to be a backup client... it does not have an SSH server built into it. If you're running borgmatic or vorta on your laptop (via docker or otherwise), your easiest path will be to just install the borg binary from the nerdpack so your unraid box can act as a borg server (note that borg doesn't run as a server daemon, it's just a binary that gets invoked over SSH in "server mode" when necessary)
If you go that path, you'll need to set up an SSH keypair between your laptop and unraid box that has nothing to do with borg. Once you can SSH in passwordless to the Unraid box, you'll simply add "1.2.3.4:/mnt/disks/backup_disk/laptop_repo" as a destination in your borgmatic config file, where 1.2.3.4 is your Unraid IP and /mnt/disks/backup_disk/laptop_repo" is the path on the Unraid box where you want your backup to reside.
You can try running a borgserver docker as an alternative, but you'll have to manage SSH keys within that, pick an alternative SSH port, and expose this port to your LAN. Doable, but a bit more complicated.
-
1
-
-
On 2/14/2022 at 10:20 PM, sdub said:
I opened an issue on the maintainer's github project. We'll see what they say.
https://github.com/borgmatic-collective/docker-borgmatic/issues/98
The package maintainer declined to add ssh-server to the borgmatic docker container... see the discussion in the link. You'll either need to install this simple borgserver docker or you'll need the borg binary installed on the machine you're trying to remotely backup to. If the target is also running Unraid this is easy enough with nerdpack.
-
1
-
-
On 3/3/2022 at 10:01 AM, Lonnie LeMaster said:
Is it possible to mount an archive from within the gui?
On 3/5/2022 at 9:06 AM, Ranbato said:No, but you might try putting in a feature request for the app
Just wrapped this up with the vorta-docker container owner. The next version of the docker container should include support for mounting archives within Vorta.
Once the update is pushed out, you'll need to add device "/dev/fuse" and "--cap-add SYS_ADMIN" under "extra parameters" in the advanced view.
After mounting the archive in the GUI, you'll need to open a command line in the vorta docker as the "app" user, since that's the user the vorta application runs as. The default "console" command from the GUI won't do it.
Run the following from the Unraid terminal:
docker exec -it --user app vorta shAlternatively, you could just skip mounting the archive from wihtin the gui and open a standard vorta console. MAnually mount the repo with:
borg mount /destination/repo /destination/mountpointProbably easier in the long run.
-
1
-
-
Mines broken too. If you set the image to modem7/wordle:legacy it works again
-
1
-
-
I submitted a feature request to include the llfuse module in the docker container today.
Is there a way to specify where logfiles are kept? Other than things being sent to the syslog that's viewable in the Docker logfile, I would like to be able to see the more detailed logfiles that are referenced in the GUI. Can't seem to find them anywhere.
-
1
-
-
Copied flash to a big old Sandisk Cruzer Glide I found in the bottom of my sock drawer. Voila! Back online. Thanks. It's interesting that it was too corrupt to boot off of, but just fine for copying files to a new drive.
Hopefully it lasts at least a year in case I need to replace it again. I can't believe that Samsung Fit Plus died sometime between 12-18 months. Seriously people, Don't use those little stubby USB drives!
I guess that's the problem with only rebooting every 6 months... no idea if latent failures are lurking. My new best practice is to perform a reboot before I do any service or upgrades on the hardware. Make sure it all comes back up cleanly to rule out the new stuff causing problems.
-
I was able to get the diagnostic file off the RAM drive using another USB disk.
I'll try plugging the flash drive into another port. It's the internal USB port on the Mobo... the one I've always used.
-
My Unraid system (X9DRH-7F mobo) has been running smoothly for 1.5 years. It has onboard dual GbE and a dual 10GbE NIC installed.
I powered down for the first time in 6 months to replace my PCIe NVM riser card with a dual NVMe card. Long story short, my mobo doesn't support bifurcation. Oh well. I removed the card and reinstalled the old single NVMe riser that has worked since the initial build.
Now when I boot, NONE of the network cards are appearing and I'm getting a dummy IP address.
Furthermore, when I log in as root, I'm not prompted for a password, and it says "root@Tower" at the login prompt. When I go to /boot I don't see the flash drive.. just some RAM filesystem with few files in it. I tried running "diagnostic" but the file that it writes to the /boot/logs directory isn't there when I pull the USB drive.
If I boot to GUI mode the web browser gives a 404 error on localhost. It almost seems like it's all stemming back to a problem with the flash drive not mounting properly at the start.
Any help where to start?? In the meantime, I'm going to try to make an Ubuntu boot USB to see what that does.
-
I opened an issue on the maintainer's github project. We'll see what they say.
https://github.com/borgmatic-collective/docker-borgmatic/issues/98
-
2
-
-
4 hours ago, Solverz said:
Unsure, "should" be possible as it just has borg installed, however do not know enough about how the container was buikt to confirm this.
BUT, you can install borgbackup directly to unraid using the nerd pack plugin and then set a script or somthing to run the borg backup server command on boot or array start.
This is what I’ve done… I have 2 Unraid servers that back up to each other. The backups are initiated from within docker containers but connect to borg binaries that I installed natively to Unraid using the nerd pack.
Unfortunately the b3vis docker container does not include an SSH server, only the client. Without that I’m not sure there’s a way for the remote machine to access the borg binary inside the local docker or Vice versa.
-
1
-
-
On 1/14/2022 at 7:31 AM, Reinis said:
Hi,
I'm trying to find an optimal solution for backing up my Linux/Mac computers to my Unraid box. Borg/borgmatic seems like a solid option based upon my research. I have Vorta installed on my Linux laptop and happily backing up to BorgBase.
Installed Borgmatic on Unraid box, though I am lost on how to configure and initialize the repositories.
I'd greatly appreciate any pointers on how to set it up.
Also, is Borgmatic a good solution for backing up a PC to Unraid, or it is meant more for backing up the Unraid box shares? Should I go with something like Duplicacy instead?
Thanks in advance!
You need to manually initialize the repo as described here:
https://torsion.org/borgmatic/docs/how-to/set-up-backups/#initialization
You can either follow my configuration guide in the first post in this thread or go right to the source for help with the config files:
https://torsion.org/borgmatic/docs/reference/configuration/
I think Borg is a great solution for any backup needs. I currently have it running backups:
1. From my Unraid array to a local backup drive
2. From my Unraid array to a remote Unraid server
3. From a windows PC running the Docker Borgmatic image
4. From a windows PC running borgmatic under WSL2
5. From a Macbook running Vorta
All of them are running very reliably. Unless you're using Vorta, you have to be willing to mess with text-based config files, but it's really not that complicated.
I've gotten a Vorta docker container to run under Unraid, but I'm hesitant to publish it to the CA appstore since I don't actively use it. I'm not really a developer, just an enthusiastic user!
-
1 hour ago, shaihulud said:
Hi!
I'm using this container in unraid to set up backups to an intermittently connected external drive. Following the borgmatic docs here, I'm trying to probe for the drive using findmnt, however when I run it in the container I get `findmnt: not found`.
From googling it seems like findmnt is including in most linux distros, and it works in the unraid shell, so I'm not sure why it doesn't work in the container. Do you know why it may not be available, and if there is any fix or alternative approach to checking if a drive is mounted?
Thanks!
Docker containers are by design minimal and not full Linux distros. You could open an issue with the container manager (https://github.com/borgmatic-collective/docker-borgmatic) and see if they’re willing to add it.
I can think of several alternatives. Probably the simplest would be to invoke the command on the Unraid host via ssh.
replace:
findmnt /mnt/removable > /dev/null || exit 75with: (where 192.168.1.100 is your Unraid server IP)
ssh [email protected] findmnt /mnt/disks/borg_backup > /dev/null || exit 75in your borgmatic config file and you should be in business. If you haven't already, you will need to set up keyfile authentication with the Unraid host, so it doesn't ask for a password.
-
1
-
-
52 minutes ago, Solverz said:
I understand, maybe settings up the Logrotate daemon in the container also so the log files don't grow too large would help also.
I don't control the underlying docker container, so I can't add logrotate, but it does include syslogd right now. I'm not experienced with this, but I think you can use that to manage local rotating logs, or send them to an Unraid syslog server
-
1
-
-
14 hours ago, Solverz said:
Hello,
I see in your crontab.txt example you put:
0 1 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1
What is the purpose of PATH=$PATH:/usr/bin I know it adds /usr/bin to the users PATH variable, but what is the purpose/need of this in the cron file?
Additionally, where do the borg log files end up?
To be honest, I think I just copied the PATH statements from an example, and never messed with it. It might not be strictly necessary, but note that cron runs as root, but not with the full root environment. So if you have before backup or after backup hooks in your borgmatic config, they might fail to run without explicitly giving the path.
By default, borg/borgmatic just log their output to the console. the "2>&1" at the end of the line redirects error output to the console. In our Unraid Docker setup, these logs are viewable in the docker log window.
Recently, I decided I wanted proper logfiles, so I added the following to my docker config:
Then I changed my cron command as follows:
/usr/bin/borgmatic -v 1 --files --stats > /mnt/logs/backup-"`date +"%a-%H"`".log
I perform backsups 2x daily, so this creates rolling logfiles named like:
backup-Mon-01.log backup-Mon-13.log backup-Tue-01.log backup-Tue-13.log ...Note that you could also use a date code formatted like `date +"%Y.%m.%d-%H"` and you'd get logfiles like:
backup-2022.01.11-01.log backup-2022.01.11-13.log backup-2022.01.12-01.log backup-2022.01.12-13.log ...and your log directory would just grow indefinitely. Also note that I removed the 2>&1 from the end of the command. This way the borg/borgmatic output is logged, but the errors are also shown on the docker log, which makes monitoring easier.
I should probably update the docker template and description to make this the default behavior at some point.
-
1
-
-
@nas_nerd
Yes!
If you add multiple repos to one config file, they will run sequentially but be identical.
What you want is to have multiple config files, one for each repo, both located in the Borgmatic config directory. They’ll still run sequentially but you can configure each repo however you like. No need to have two Borgmatic containers.
You can even use include files to avoid duplicating settings common to both files.
See the support article for more detail:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/
Sent from my iPhone using Tapatalk-
1
-
-
4 hours ago, kri kri said:
Thanks for making this docker. What is my best course of action for backing up my Borg repo to multiple external drives mounted with UD? Rsync, or make them all borg repos themselves?
I think it’s generally a better practice to create separate repos and not replicate them. It doesn’t take much longer and avoids propagating corruption or errors.
see the official borg FAQ on this point:
[Support] borgmatic
in Docker Containers
Posted
I backup my server to another server where my connection is ~15Mbps up and the remote server is 200Mbps down.
I performed a backup last night that consisted of 5.8GB against a 7.1TB archive. It took 82 minutes. The equivalent local backup took 55 minutes.
… so if it all scales linearly I’d say my backups take about 5 minutes per TB (total archive) plus copy time.
If it’s local that’s about 1 minute per GB and remote (with my line speed) that’s about 7 minutes per GB changed.
Remember that a lot do the time during the backup, it’s scanning files on the remote machine using borg in server mode and not transmitting data.