Rsync errors when trying to backup 2+Tb Nextcloud Install

[jeffmackinnon]

So I have two Unraid installs, one is my main server and the other is purely for backups. On the main server I have all my apps installed in dockers (Plex, Nextcloud, Sabnzbd, etc).

 

On the backup I only have a Ubuntu VM.

 

The plan was to backup the nextcloud share called "fog" on the main server to the backup.

 

To do this I mounted fog and the backup on the Ubuntu VM and then ran the following command

rsync -Aax --exclude=".*" /mnt/fog /mnt/backup/FOG-Backup

However, regardless every time I run this, I get an error that looks like this:

rsync: close failed on "/mnt/backup/FOG-Backup/fog/.nextcloud.log.7Zkz3M": No space left on device (28)

I have spent the last couple of days trying to figure out what it is, I have tried direct tarballing, I have tried different switches on the rsync, etc.

 

Any tips to make this work will be very appreciated.

[jeffmackinnon]

A bit of an update. I think it is because my drives aren't balanced, and the first drive as part of the Backup share is full. Is this a quirk of how the shares work? Do I need to level out the drives first?

[pwm]

You are talking about a Ubuntu VM, but that doesn't give us much information about actual host paths that are involved. Maybe at least start by explaining exactly what the mounts are, in relation to actual paths on the two unRAID machines.

 

Are you trying to backup from one user share on one unRAID to a user share on the other unRAID? And how is then the target unRAID configured to split file writes to actual disks?

 

Or are you trying to perform a backup from one disk share to another disk share? In which case does the target share have the required space?

[jeffmackinnon]

8 minutes ago, pwm said:

You are talking about a Ubuntu VM, but that doesn't give us much information about actual host paths that are involved. Maybe at least start by explaining exactly what the mounts are, in relation to actual paths on the two unRAID machines.

 

Are you trying to backup from one user share on one unRAID to a user share on the other unRAID? And how is then the target unRAID configured to split file writes to actual disks?

 

Or are you trying to perform a backup from one disk share to another disk share? In which case does the target share have the required space?

1. /mnt/fog is on the main server as a share called fog, /mnt/backup is on the backup server as a share called backup. In the VM, which is located on the backup server, the /etc/fstab file is configured to auto mount those two shares. The main server and backup server are in different boxes attached to the same switch (for the time-being)
2. Yes
3. The target is configured as attached here. Originally it was "Fillup" but I have switched to High-Water and added a new drive.
4. No, I am running the backup b/w servers, not on a single server. That is what I meant by two Unraid installs.

[jonp]

Your issue is likely the result of not having a minimum free space value specified for the share.  Because we don't stripe data in the array, individual storage devices store individual complete files.  As such, when a disk is nearing capacity, if you attempt to write a file to that individual disk that is larger than the capacity that disk has remaining, it will result in an out of space error.  The way to solve for this is to set the minimum free space value to the size of the largest file you intend to copy to that share.  So if we were dealing with a "movies" share, lets say, 50GB would probably suffice for most folks, though that might be larger if we're talking 4K content.  Depending on how your backups work, you could be archiving one giant file (not ideal) or there could be a large set of smaller files (better).

[jeffmackinnon]

58 minutes ago, jonp said:

Your issue is likely the result of not having a minimum free space value specified for the share.  Because we don't stripe data in the array, individual storage devices store individual complete files.  As such, when a disk is nearing capacity, if you attempt to write a file to that individual disk that is larger than the capacity that disk has remaining, it will result in an out of space error.  The way to solve for this is to set the minimum free space value to the size of the largest file you intend to copy to that share.  So if we were dealing with a "movies" share, lets say, 50GB would probably suffice for most folks, though that might be larger if we're talking 4K content.  Depending on how your backups work, you could be archiving one giant file (not ideal) or there could be a large set of smaller files (better).

Thanks Jon, 

 

I have added 50GB as the minimum free space (my largest file will be around 30GB), removed the old in-finished backup, and started a new one.

 

I should know in the morning if it worked.

Thanks, 

JM

[jonp]

18 hours ago, jeffmackinnon said:

Thanks Jon, 

 

I have added 50GB as the minimum free space (my largest file will be around 30GB), removed the old in-finished backup, and started a new one.

 

I should know in the morning if it worked.

Thanks, 

JM

 

Awesome!!  Please do let us know if everything works after this adjustment. 

[jeffmackinnon]

So everything seems to be working smoothly on the rsync backup side now. 

 

The rest should be normal troubleshooting the completed system, and then building a third system for offsite storage. I'm not a fan of the cloud yet, I rather build my own fog.