[Support] binhex - UrBackup


Recommended Posts

  • 2 weeks later...
  • 3 weeks later...

I just set this up and am starting to consider what I should be backing up. I like the idea of a full system image, but frankly I probably don't need it since I can pretty easily re-install Windows and re-download Steam ;-) I'm curious what everyone else is doing--full images? Documents/Pictures only?

Link to comment
  • 4 weeks later...
16 hours ago, eds said:

Has anyone tried a full image backup and then restore with this?

I did and the restore got as far as 23/24% and then bsod. 

Yes I tried it on a vm and it was successful, are you restoring to the same hardware that the backup was taken from?

 

if not the case then you are most probably seeing a windows driver that was included on the source machine attempting to be loaded on the destination machine and failing (missing/different hardware) which then causes the BSOD.

 

if you are restoring to different hardware then you would need to sysprep the source machine before backing it up or run some sort of windows repair, or alternatively do file/folder level backups and then you can restore to any machine running windows.

Edited by binhex
Link to comment
21 hours ago, binhex said:

Yes I tried it on a vm and it was successful, are you restoring to the same hardware that the backup was taken from?

 

if not the case then you are most probably seeing a windows driver that was included on the source machine attempting to be loaded on the destination machine and failing (missing/different hardware) which then causes the BSOD.

 

if you are restoring to different hardware then you would need to sysprep the source machine before backing it up or run some sort of windows repair, or alternatively do file/folder level backups and then you can restore to any machine running windows.

Hi binhex, 

 

Thanks for the quick response.  

I was restoring to the same hard drive the image was created from. 

 

Curious if you think the fact that the restore went from 23%-100% in seconds and said restore was complete was at all strange? 

 

The backup took something like 800 minutes.   The restore took less than 200.  I thought that was odd too. 

Link to comment
4 hours ago, eds said:

Curious if you think the fact that the restore went from 23%-100% in seconds and said restore was complete was at all strange? 

dont take this as fact, but i would suspect the reason the restore went from 23% to 100% very quickly is because the rest of the image that was restored is free space, thus urbackup is clever enough to know it doesnt need to write each byte as there is no data present - have a look at the source machine, do you have a large proportion of free space on the drive?.

 

of course a backup of a drive will have to be done in its entirety, as urbackup obviously doesn't know where the data is written on the disk, so it does a complete backup of the disk, thus the backup will take longer in most cases than the restore.

Link to comment
5 hours ago, binhex said:

dont take this as fact, but i would suspect the reason the restore went from 23% to 100% very quickly is because the rest of the image that was restored is free space, thus urbackup is clever enough to know it doesnt need to write each byte as there is no data present - have a look at the source machine, do you have a large proportion of free space on the drive?.

 

of course a backup of a drive will have to be done in its entirety, as urbackup obviously doesn't know where the data is written on the disk, so it does a complete backup of the disk, thus the backup will take longer in most cases than the restore.

Yes, the drive had mostly empty space so it does make sense the restore would move 23% - 100%.

 

Good to know that urbackup would take that into consideration on the restore. 

 

Thanks again. 

Link to comment
  • 3 weeks later...
On 11/19/2019 at 9:51 AM, SuperDan said:

...this image seems to be missing BTRFS tools

I'm curious about this, too, @binhex!

 

I've been attempting incremental backups (directly to a /mnt/cache/... path within a cache-only share folder on a BTRFS cache drive) and finding that UrBackup is making none of the expected BTRFS subvols or snapshots.

 

I'm absolutely not up to speed on what all is involved in a Docker container performing BTRFS-specific operations on an "external" BTRFS volume.

 

So, following up on @SuperDan's question, might this be because certain BTRFS resources are excluded from the binhex-urbackup image? And, if so, is that strategic?

 

If it isn't strategic, it would be lovely to see them added. And when I have a bit of free time, if I won't be duplicating someone else's efforts, I'll take a shot at it myself.

 

EDIT: After some consideration and experimentation, I'm not even sure I'm thinking about this correctly.

 

I installed btrfs-progs within the binhex-urbackup container (# pacman -Fy && pacman -S btrfs-progs) as an experiment, but my next incremental backup still didn't create the BTRFS subvolume I was hoping for.

 

On the UrBackup Developer Blog, it says that "[e]very file backup is put into a separate sub-volume" if "the backup storage path points to a btrfs file system and btrfs supports cross-sub-volume reflinks."

 

So, admitting I'm more than a touch out of my depth here, perhaps: 1) Unraid btrfs doesn't support cross-sub-volume reflinks for some reason, or 2) I shouldn't expect it to work from within a Docker container accessing a filesystem that's outside the container, or 3) ...something else.

 

Any insight is appreciated, and I'll post here if I happen to get it figured out.

Edited by bland328
More information
Link to comment
On 1/19/2020 at 1:05 AM, bland328 said:

So, admitting I'm more than a touch out of my depth here, perhaps: 1) Unraid btrfs doesn't support cross-sub-volume reflinks for some reason, or 2) I shouldn't expect it to work from within a Docker container accessing a filesystem that's outside the container, or 3) ...something else.

i think you are correct, it probably is one or more of the above possibilities, but i honestly dont know which. i have a problem helping out, in as much as i dont have any btrfs formatted drives, and on top of that i am concerned about adding in the btrfs tools to this image, due to the previous issues with sparklyballs (another developer) urbackup image causing docker corruption, which (or may not) be due to the use of btrfs tools conflicting with the btrfs formatted docker loopback mounted image.

 

so im afraid you are on your own with this one, if you do figure it out and its stable then i would be ok with creating a test tagged image with the change, this way we can ensure it doesnt screw up docker before we release to the wider community.

 

 

Link to comment

First, thanks for your work, it's an amazing docker app.

 

I'm trying to use it with cache enabled (yes setup in the share) to do the backups faster.

The problem is that the docker is filling the cache disk, until it's full.

Then the docker apps and VM crashes, and the urBackup stop the backup because the cache disk is full.

 

There are any way to do the backups using the cache disk? I suposed that urBackup will use the cache disk until it's full, and then continue using the array instead the cache disk.

 

If i use the setting "cache = No" in the share, all is working fine. Just backups are longer than using the cache.

https://gyazo.com/5a6752ba3f9914ed2642f594dc1b9b2d

Link to comment
11 minutes ago, jcabello7 said:

I suposed that urBackup will use the cache disk until it's full, and then continue using the array instead the cache disk.

no, this is not how it works, if you set a share to use cache then any files written to the share will be written to the cache drive instead, if the cache drive fills up then no further copies will be permitted, it (unraid or the app) will NOT automatically switch to writing to the array when the cache is full.

 

In the situation where the cache drive is full you have two options, manually copy the files from the cache drive to the array to free up space, or wait for the 'mover' to move the files from the cache drive to the array for you, also freeing up space.

 

In short, if your cache drive is too small to contain your urbackup files for a 24 hour period (mover normally runs nightly) then you either need to buy a larger cache drive or not use the cache and write directly to the array, the choice is yours.

Edited by binhex
Link to comment
12 minutes ago, binhex said:

no, this is not how it works, if you set a share to use cache then any files written to the share will be written to the cache drive instead, if the cache drive fills up then no further copies will be permitted, it (unraid or the app) will NOT automatically switch to writing to the array when the cache is full.

 

In the situation where the cache drive is full you have two options, manually copy the files from the cache drive to the array to free up space, or wait for the 'mover' to move the files from the cache drive to the array for you, also freeing up space.

 

In short, if your cache drive is too small to contain your urbackup files for a 24 hour period (mover normally runs nightly) then you either need to buy a larger cache drive or not use the cache and write directly to the array, the choice is yours.

You explained it really well. The problem is that the image backup that I'm trying to do is 1TB disk, and the cache is 512GB.

So, the unRaid help is not 100% clear, because it says:

 

Quote

Yes indicates that all new files and subdirectories should be written to the Cache disk/pool, provided enough free space exists on the Cache disk/pool.
If there is insufficient space on the Cache disk/pool, then new files and directories are created on the array.

 

So, I should use the No setting to save data direct to the array. Thank you!

Link to comment
1 minute ago, jcabello7 said:

Yes indicates that all new files and subdirectories should be written to the Cache disk/pool, provided enough free space exists on the Cache disk/pool.
If there is insufficient space on the Cache disk/pool, then new files and directories are created on the array.

hmm ok i stand corrected there, i dont think it used to do this, and i personally dont use mover so wasnt aware of the change.

 

Even with this possibility its still going to be an issue if urbackup is generating a large single file, which it can do when creating a disk based backup of a machine, as it wont be possible to split the file in two and write some of it to cache and some of it to the array when the cache drive is full - which looking at your initial post seems to be the case.

 

4 minutes ago, jcabello7 said:

So, I should use the No setting to save data direct to the array. Thank you!

i think this is your only reasonable course of action for now, maybe reconsider this again when you bump up the size of your cache drive.

 

FWIW i do not use cache drive for caching, most people don't, its used to store docker images, vm vdisks, and metadata, writing to the array, especially when 'CA Auto Turbo Write Mode' (check CA for details) is installed, is fast enough for most use cases IMHO.

 

Link to comment

Try to set the backup share to "Prefer".
From the help description:
"Prefer indicates that all new files and subdirectories should be written to the Cache disk/pool, provided enough free space exists on the Cache disk/pool.
If there is insufficient space on the Cache disk/pool, then new files and directories are created on the array.
When the mover is invoked, files and subdirectories are transferred off the array and onto the Cache disk/pool."

Link to comment

Hey binhex,

 

I am running into this error when trying to initiate file backups from my windows machine.  I will note that after initial setup it DID start a file backup, but I adjusted some client side settings, since I really only wanted to start with personal files rather than my entire drive.

 

Any help would be appreciated.  Should I reinstall at this point?

 

DEBUG  Script does not exist urbackup/post_incr_filebackup

 

DEBUG  Script does not exist urbackup/post_full_filebackup

 

Edit:  It seems to have kicked off a full image backup, but is still doing the full disk.  I will have to do some digging to understand this a bit better.

Edited by WhatPlantsCrave
Link to comment

A client only version of this Docker would be very useful - for users that want to just backup selected UNRAID folders to other URBACKUP servers.

 

I have been using URBACKUP for about a year now to backup onsite and offsite & just started using UNRAID as a central file server. So need to set up a URBACKUP client on my UNRAID server now.

  • Like 1
Link to comment

Thank you for another excellent docker! I am planning on utilizing Urbackup to backup my vital files on my Unraid server. I am just not sure whether to implement/configure certain needs.

 

The configuration

- An Unraid server with about 15 shares (all exported read-only to prevent ransomware attacks).

- A dedicated Windows VM where I would install an Urbackup client and enable read-only access to all shares to be backed up (file backups).

- Windows desktop where I would install an Urbackup and configure an image backup of the computer.

- Another Unraid server (my friend's) in a different location, where I would sync the backup databases.

- My friends backup databases would be synced to my Unraid server also.

- The backup traffic between our Unraid servers would go via a WireGuard tunnel.

 

The questions

- Backing up the shares would be easy if I could have an Urbackup client inside the Unraid server. However, would the above configuration achieve the same?

- I would also backup the Unraid system files using this. That is, I need to export them as SMB shares (read-only). I guess there aren't any issues on this?

- What would be the best way do the backup database syncs between our Unraid servers? Any examples?

 

Thank you for any info!

 

Link to comment
On 1/31/2020 at 4:27 AM, WhatPlantsCrave said:

Hey binhex,

 

I am running into this error when trying to initiate file backups from my windows machine.  I will note that after initial setup it DID start a file backup, but I adjusted some client side settings, since I really only wanted to start with personal files rather than my entire drive.

 

Any help would be appreciated.  Should I reinstall at this point?

 

DEBUG  Script does not exist urbackup/post_incr_filebackup

 

DEBUG  Script does not exist urbackup/post_full_filebackup

 

Edit:  It seems to have kicked off a full image backup, but is still doing the full disk.  I will have to do some digging to understand this a bit better.

ive not seen that particular messages in my logs and i use urbackup on a daily basis, i can only assume that is an issue on the urbackup client side, which urbackup client did you deploy to your machine?.

 

also i would advise configuring the client file selection from the urbackup server web ui, its much better to push the config from the server to the client, than the other way around.

Link to comment
1 hour ago, Ruato said:

Backing up the shares would be easy if I could have an Urbackup client inside the Unraid server. However, would the above configuration achieve the same?

im assuming you are thinking about mapping the share on the windows client and then configuring the urbackup client to backup the mapped drives, right?, if so im not sure if the urbackup client wll allow you to select mapped drives, you could certainly try it, although its far from optimal performance of course, it couldf be a bit/very slow.

 

1 hour ago, Ruato said:

I would also backup the Unraid system files using this. That is, I need to export them as SMB shares (read-only). I guess there aren't any issues on this?

if by system files you mean the flash drive, then yep you could do this, as it would just appear as another share (if configured).

 

1 hour ago, Ruato said:

What would be the best way do the backup database syncs between our Unraid servers? Any examples?

ive not attempted this myself, are you purely talking about the sqlite internal urbackup database here?, so not the actual backed up files/folders?, off the top of my head i would say probably rsync is the best tool for the job here, i dont really want to include an example here as there is a TON of options for rsync dependong on what you want it to do, check the man pages.

  • Thanks 1
Link to comment

First of all, thank you for the instant reply and support!

 

22 hours ago, binhex said:

im assuming you are thinking about mapping the share on the windows client and then configuring the urbackup client to backup the mapped drives, right?, if so im not sure if the urbackup client wll allow you to select mapped drives, you could certainly try it, although its far from optimal performance of course, it couldf be a bit/very slow.

Yes, correct. I would prefer to do it internally inside the Unraid server but there is not a Urbackup client available and it is needed. Or have I misunderstood something?

 

Does Urbackup ignore the files that have not changed? That is, it transfers only the changed files since the previous backup? That way the performance hit is not so bad after the first backup.

 

22 hours ago, binhex said:

ve not attempted this myself, are you purely talking about the sqlite internal urbackup database here?, so not the actual backed up files/folders?, off the top of my head i would say probably rsync is the best tool for the job here, i dont really want to include an example here as there is a TON of options for rsync dependong on what you want it to do, check the man pages.

Yes, I was planning on syncing the Urbackup database. However, I might have limited understanding of the issue. So, what I want to acchieve is that if my Unraid server and the Urbackup installation inside it gets correupted / destroyed, I can still restore the backups using the files stored in my friends Unraid server. So, I need to sync everything that is needed for that. Do you know, whether the Urbackup internal database is enough or should I need to sync something else also?

 

Link to comment
On 1/18/2020 at 5:05 PM, bland328 said:

 

 

I installed btrfs-progs within the binhex-urbackup container (# pacman -Fy && pacman -S btrfs-progs) as an experiment, but my next incremental backup still didn't create the BTRFS subvolume I was hoping for.

 

On the UrBackup Developer Blog, it says that "[e]very file backup is put into a separate sub-volume" if "the backup storage path points to a btrfs file system and btrfs supports cross-sub-volume reflinks."

 

So, admitting I'm more than a touch out of my depth here, perhaps: 1) Unraid btrfs doesn't support cross-sub-volume reflinks for some reason, or 2) I shouldn't expect it to work from within a Docker container accessing a filesystem that's outside the container, or 3) ...something else.

 

 

Im curious if you tried running the test script "urbackup_snapshot_helper test" from the container console to verify that Urbackup can use the BTRFS snapshot feature?

 

Edited by SuperDan
Link to comment
On 2/5/2020 at 12:34 PM, SuperDan said:

Im curious if you tried running the test script "urbackup_snapshot_helper test" from the container console to verify that Urbackup can use the BTRFS snapshot feature?

 

I somehow missed that, so I did not. Thanks for the tip! I'll give it a try when I have a few free minutes and report back.

 

I tried it and got an immediate failure of:

Backupfolder not set

I don't have time at the moment to look into that deeply, but I did find food for thought at https://forums.urbackup.org/t/urbackup-mount-helper-test-backupfolder-not-set/5271.

Edited by bland328
Link to comment
On 2/11/2020 at 9:08 AM, bland328 said:

I somehow missed that, so I did not. Thanks for the tip! I'll give it a try when I have a few free minutes and report back.

 

I tried it and got an immediate failure of:


Backupfolder not set

I don't have time at the moment to look into that deeply, but I did find food for thought at https://forums.urbackup.org/t/urbackup-mount-helper-test-backupfolder-not-set/5271.

Yeah it does that because it requires a folder and file to be created named:

/etc/urbackup/backupfolder

Inside the file backupfolder you put the path to where urbackup stores it's backup ie: /mnt/user/Urbackup

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.