Need a simple off-site unraid-unraid incremental backup solution


maxse

Recommended Posts

On 3/18/2019 at 8:08 PM, strike said:

 

I totally agree with this!

 

Talking about folder structure, there is a huge "design flaw" using rsync with user shares on the first sync. Rsync will create the entire folder structure of a share on the first disk it chooses before it starts transferring the files. This will result in all files going to the first disk causing it to totally fill up and ultimately fail with out of space errors.

 

This is only a problem with rsyncing user shares I think, and also only a problem on the first sync. Or I should say only a problem syncing to an empty server. I prefer using user shares, but I knew about this issue before I started transferring the files to the backup server so I didn't use rsync on the first transfer. I just mounted the server via UD and transferred all the files via Midnight Commander, locally. But I have been using rsync with user shares ever since then and it works great, just not for the first big transfer to an empty server.

Just to make sure as I am almost done transferring all of the files and just changed out the fans on the backup server case.

 

Since I am rsync shares only, not the disks... Is this a problem when the share is empty or just the entire server? I have transferred some files, but created more shares to be organized and those haven't been transferred to the backup server yet, so that share is empty on the backup server, while other shares on the backup server do have files already...

 

Or should I at least transfer 1 file into the new share on the backup server and then can I rsync with this method? Or will I still have this issue?

Link to comment

If the share is empty on the backup server and the share you're going to transfer are too big for the next disk that unraid chooses you're going to have this issue. This is because as mentioned rsync will create the ENTIRE folder structure of the share on the next disk unraid chooses (based on allocation settings) BEFORE the file transfer starts. So all files in that share will go to that disk because all folders are created on that disk.

 

I guess you could solve it by setting split level to automatically split all files as required in the share settings. I haven't tried it but it should work. But if you have any other split level it most definitely won't work (assuming the share is too big for the next disk unraid chooses).

 

So you gotta choose, either you don't care where files end up (set split level to split all files as required), or you care and have another split level set. If any other split level, then either check that the share can fit on the disk or transfer the files manually.

 

Edit: And if you choose any other split level then "split any files as required" you're gonna have this issue when disks start to get full (not rsync's fault, but because of the split level). So you gotta watch the disk usage and either exclude the disk that is getting full from the share or you're gonna have to manually move files around to make room for future files going to that disk

Edited by strike
Link to comment
  • 2 weeks later...

I have never had this issue as defined by @strike as I do most of my backups by disk.  rsync mainserver.mnt/disk1 -> backupserver.mnt/disk1 

 

If you can do it this way, you won't have the problems he listed.  However if you want to backup mnt/user/everything -> mnt/user/everything and expect rsync to manage everything for you, it won't be so easy.

Link to comment
  • 3 months later...
On 4/27/2019 at 3:10 PM, strike said:

If the share is empty on the backup server and the share you're going to transfer are too big for the next disk that unraid chooses you're going to have this issue. This is because as mentioned rsync will create the ENTIRE folder structure of the share on the next disk unraid chooses (based on allocation settings) BEFORE the file transfer starts. So all files in that share will go to that disk because all folders are created on that disk.

 

I guess you could solve it by setting split level to automatically split all files as required in the share settings. I haven't tried it but it should work. But if you have any other split level it most definitely won't work (assuming the share is too big for the next disk unraid chooses).

 

So you gotta choose, either you don't care where files end up (set split level to split all files as required), or you care and have another split level set. If any other split level, then either check that the share can fit on the disk or transfer the files manually.

 

Edit: And if you choose any other split level then "split any files as required" you're gonna have this issue when disks start to get full (not rsync's fault, but because of the split level). So you gotta watch the disk usage and either exclude the disk that is getting full from the share or you're gonna have to manually move files around to make room for future files going to that disk

Wow time flies! Believe it or not guys I am still working on this!!! I am still in the process of converting files to X265, it's taking almost half a year LOL, but I should be done in a month or so. In the meantime I've been playing around with a raspberry pi and was able to set up DuckDNS on it successfully so I should be ready to start this process soon.

 

So back to this just to make sure again. I will be copying share-share and not disk/disk, the disks on the remote backup are not all the same size as the source. There's not way I want to be able to monitor and exclude disks on the remote server, this needs to be a set it and forget it type of thing once it's running...

 

Do you mean to set the share to ""Automatically split any directory as require?" I don't have an option that says "...file as required." So that would be fine then?

 

Also I thought to get around this I could copy just one folder to the backup server on the share, and then I would be able to run rsync and it would just copy over the new files automatically skipping what's already on the target, and then I won't have the issue stated by @strike? I thought that's all I had to do is just make sure that each share has at least one file in it, and then just rsync the rest and it won't create this problem?

Link to comment
1 hour ago, maxse said:

""Automatically split any directory as require?"

Yes, that's the one. I've haven't tested it but I think if you have this you'll be fine even on the first big transfer (I think). But I want to rephrase my previous statement a little. In my previous statements I've been saying this is an issue only on the first big transfer, but that's not correct. Once again to be clear this issue happens if your split level is not set to  "Automatically split any directory as require?" and you're trying to rsync a large batch of files. It don't even have to be that large of a transfer to trigger this. What happens when you do a rsync transfer is that rsync will create ALL the folders on the first available disk according to allocation method BEFORE transferring any files.

 

Split level is about trying to keep files together so they don't end up on different disks, so if you have the wrong split level unraid will try to force some files to a disk which is already full, trying to keep the files together according to set split level. Since this is just a backup I wouldn't worry about where files end up anyway.

 

So if you take those two paragraphs and put them together you'll see that this issue can happen on even a small transfer. Lets says you have transferred all your files and in 2 months time you have 500 new folders with pictures in them, roughly 5 gig in total. Which you want to back up. The next disk in line to get new files based on allocation method is disk 3, which only has 2 GB available space left. Your split level is set wrong and because of that and this rsync issue all 500 folders will be created on disk 3, even if some of the files in those folders won't make it to disk 3 because it's completely full by the time rsync gets to them and it won't choose another disk either because split level goes before everything, so the transfer will just fail with out of space errors.

 

So for this to be set it and forget it you will have to choose the first split level like mentioned, you will have to set a min free limit so that unraid will choose another disk if that disk has less space then the limit. Since this is just a backup I think I would set the allocation method to fill up. If you're not concerned about evenly filling the disks that is. I try to keep at least 30 GB of available space left on all my disks.

Edited by strike
Link to comment
  • 2 months later...

Hey @maxse I just read through this whole thread. It was interesting watching you lean and grow as the thread progressed. Brought a tear to my eye. 😥 Anyhow.. I'm interested in setting us something very similar at a friend's place. I can probably convince them to change out their main router/firewall to an Edgerouter or an USG. But if you don't mind could you give a little summary of what you decided to do? And what equipment and applications you will use? I like the idea of waking the backup server through IPMI as well. 

 

I'm running my array disks as xfs encrypted at my home unRAID box. Not sure if that matters and I'd like to keep the backup server as light as possible. I'm considering at site-to-site L2TP IPsec VPN between the two routers and just allowing one VLAN to utilize the tunnel. 

 

Please give us an update? 🙏

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.