What's going on:

• I have a 1 TB file that I routinely backup with rsync --inplace from one unraid server to another unraid server.
• Both the source and destination are using BTRFS for their filesystems (single-disk).
• The file is encrypted and has been backed up before so the full 1 TB for the file is fully allocated on the destination
• I recently changed about 500 GB of the file and backed it up again. During the transfer I noticed that my free space on the disk was going down and by the end of the transfer I had lost about 500 GB of free space. I would expect it to stay the same, as it should be overwriting already allocated space.
• I unmounted, restarted, remounted but it stayed the same size
• du -s showed about what I expected, but df was off by about 500 GB, which makes me think the issue has something to do with the filesystem
• I deleted the file and sure enough it freed up over 1500 GB

I'd appreciate any insight into this issue. I'm guessing that not transferring with inplace would 'fix' it, but I don't want to do a 1 TB write every backup when usually I only need to transfer a gig or two. Not to mention all my VM disks. Also while in this case it was happening with one large file making it obvious, I assume it happens with files of all sizes.

 

Also here is the rsync command I used:
rsync -aHXxrv --numeric-ids --inplace --no-whole-file --delete "$source" "$destination"

Edited May 4, 20233 yr by fry_the_solid

Not sure why the file is growing since it's not something I ever used, but since both filesystems are btrfs why not use snapshots with incremental send/receive?

I didn't know about incremental send/receive, I'll use that instead. Thanks!

@JorgeB

I did a little reading on it and if I'm understanding correctly, this won't be possible in Unraid if the destination disks are not of equal or greater size to the source disks. For example I couldn't backup a 16TB disk to 2 x 8TB disks.

That's not a problem, as long as the 2 x 8TB disk are in the same btrfs pool.

Thank you again for the advice Jorge, this solution is so much better that what I was using before. Both my primary and backup server are using disks of the same size, and (single-disk) btrfs, so I am able to backup the disks 1:1. Fortunately for me it was a perfect solution. In case anyone is in the same situation and wants to look into it, start with btrbk. I spent about 3 hours writing my own script before I stumbled upon it. It's easy to configure and does just about everything you could want.

 

However, this alternative may not work for everyone. If the source filesystem was not btrfs, this wouldn't work. Similarly, if the disks in the destination are not equal in size to the source, it will be a challenge/not possible to backup using this solution either. It's true that if the disks are in the same btrfs pool this wouldn't be an issue, but then you need to use either raid0 with no redundancy or raid1cX where at most you're able to utilize 50% of your raw storage capacity. In the context of Unraid, I'd guess that this would be an issue for many users.

 

edit: I believe I understand the issue now, it is a problem with BTRFS. Whenever you overwrite existing data in BTRFS it is not actually overwritten but rather a new block is written to, but for whatever reason with these inplace transfers (and similar cases like writing to VM disk images) it does not delete the old block leading to unusable/wasted space. You can use a tool called btdu to view how your space is being used; the space that has been overwritten but not deleted is listed as "unreachable". The only solution I found were to run a btrfs defrag (though I didn't test this myself), or transfer the file off of the volume then back on, however neither of these are something you'd want to do often. All in all an unfortunate limitation (bug?) of BTRFS.

Edited May 31, 20233 yr by fry_the_solid
solved