rsync Incremental Backup


Recommended Posts

Thank you.


I also thought about the hardlink limit. However, it's at 1023 and I only made it to 20 backups.

 

The use case should be that someone who knows very little about computers can use the disk (family photos) without my help.

 

That's why I didn't want to use a unix file system. Since I don't know where the person will use the disk, I also didn't want to use a system that only works with additional software under Windows.

 

What else could depress the read/write speed so much that the disk sort of stands still?

 

Could another standard job that is started in parallel cause the problem?

 

Certain files should not be it, right?
 

Is there any way I can turn on a debug mode?

 

many greetings
bonbonJaeger

Link to comment
1 hour ago, bonbonJaeger said:

--compress

This option is useless as you are copying locally.

 

1 hour ago, bonbonJaeger said:

Does anyone have any idea what is causing the script to crash?

What is written to the logs?

 

How do you know that the script is crashed? Did you try to kill rsync by "pkill -xc rsync"?

 

1 hour ago, bonbonJaeger said:

If I then force the system to reboot with "reboot", the parity check is started every time.

I had a similar problem in the past. It seems Unraid is not able to restart if a rsync process is still running. Because of that I created a stop file with the following content:

IMG_20210806_132937.thumb.jpg.2fe26459b78185b093289129c66c9530.jpg

 

Unraid executes the stop file automatically if the system reboots / is shutdown.

Link to comment

Thanks.

 

1 hour ago, mgutt said:
2 hours ago, bonbonJaeger said:

--compress

This option is useless as you are copying locally.

I have removed the switch.

 

1 hour ago, mgutt said:
2 hours ago, bonbonJaeger said:

Does anyone have any idea what is causing the script to crash?

What is written to the logs?

 

How do you know that the script is crashed? Did you try to kill rsync by "pkill -xc rsync"?

I thought it crashed because nothing happened for hours. I tried to kill the processes with

kill -9

 

1 hour ago, mgutt said:

I had a similar problem in the past. It seems Unraid is not able to restart if a rsync process is still running. Because of that I created a stop file with the following content:

I have created the script. Do I need to make it executable? If so how? A

chmod +x

shows no effect.

 

I will run the script again and then post the logs here.

Link to comment
14 minutes ago, mgutt said:
1 hour ago, bonbonJaeger said:

I have created the script. Do I need to make it executable?

Don't you use the user scripts plugin?

Your screenshot showed the config editor and a different path than the one for the user scripts. That's why I didn't get the idea.
Should I create the script via the UserScripts and then copy it to /boot/config?

Link to comment

it happens again. The syslog shows:

 

image.thumb.png.87180a04aa45fffdbcc29378e1d5a665.png

 

The unassigned devices - 0 free and 0 used space - no read or write speed:

image.thumb.png.78e81c66890756ef84a5d235b1d176bf.png

 

I had a window open with tail to view the output of the rsync script.

 

image.thumb.png.12b085922537c5ba74eee9709583fc78.png


Is there anything else I can check? I would keep running this now to see if it recovers.

Link to comment
20 minutes ago, bonbonJaeger said:

it happens again

While the backup is running? Should be "normal". As I said. Its a timeout of the UD Plugin and not an error.

 

21 minutes ago, bonbonJaeger said:

I had a window open with tail to view the output of the rsync script.

Does it still copy files or is it frozen?

Link to comment
36 minutes ago, bonbonJaeger said:

But the screenshot of your stop script pointed to /boot/config. 

Ah ok. Simply create the file:

touch /boot/config/stop

 

Should be sufficient. It gets the same chmod as the other files, so it should work:

image.png.6d7de3ffaf97abdc2a6d8b1ddf5e8979.png

 

For Copy & Paste:

#!/bin/bash

# -------------------------------------------------
# Kill all rsync processes
# -------------------------------------------------
pkill -x rsync

 

Link to comment
2 hours ago, mgutt said:

While the backup is running? Should be "normal". As I said. Its a timeout of the UD Plugin and not an error.

 

Does it still copy files or is it frozen?

I can't tell if it's copying files. Because in the main window for the disk everything is set to 0.
The rsync-log has no new entries.

image.thumb.png.197cedb95a7febdb137bea2e52163c54.png
But the syslog has many error messages.

image.thumb.png.58ed5d00153bff981bf5c4190a08ae30.png

Link to comment
27 minutes ago, bonbonJaeger said:

But the syslog has many error messages.

Yes, this is normal as you access the WebGUI which trys to reload the unassigned devices usage every 10 seconds or so (which fails because of the timeout).

 

27 minutes ago, bonbonJaeger said:

The rsync-log has no new entries.

Ok, so it's dead.

 

At first you should kill rsync: (x = exact match, c = count killed processes)

pkill -xc rsync

 

Then add "--timeout=10" to your rsync command. Rsync does never timeout without this flag.

 

But finally it does not solve your problem.

 

Maybe you should try to copy the folder manually with the "cp" command. And I would try to start a diagnosis with a vendor tool. As an example if it's an WD drive, the Data Lifeguard Diagnostic Tool.

 

Link to comment
1 hour ago, mgutt said:

At first you should kill rsync: (x = exact match, c = count killed processes)


pkill -xc rsync

image.thumb.png.47bda2ce2be464715f933ddac32e175f.png

that doesn't work. I will reboot. Unfortunately, the reboot does not work because the disk does not respond.

image.png.fd6b4b32ad22fb5fa66a4ca946823294.png

I now have to force the restart. But this also means a new parity check.

Tomorrow I will check the disk with the diagnostic tool.

Link to comment
2 hours ago, bonbonJaeger said:

that doesn't work. I will reboot

Then the target disk is completely blocking rsync. Maybe because of the massive amount of sub shells calling "df"... I will ask the dev of UD what he thinks about that. 

 

Instead of rebooting: Wouldn't it be possible to simply disconnect the drive?

Link to comment

So far, removing the disk from unraid has not been detected. The same error messages were always displayed.

 

I have downloaded the dashboard from WD. Unfortunately, only a SMART test is offered here. I could have also started in unraid.

 

By the way, the dashboard shows that everything is good with the disk.

 

I remembered that backups via a virtual machine with the software HardlinkBackup runs without errors.

 

Could there be a problem with the USB port?

 

For now, the advanced SMART test is running.

Link to comment
  • 2 weeks later...
On 3/17/2021 at 8:22 AM, mgutt said:

Add a new local user through the settings with a strong password and name it "backup". After that right mouse click on "documents" > share with... Select the new user from the drop-down. It should be "read only". Repeat this step for all dirs you want to share. Do not share the root username dir. It contains the hidden "appdata" dir and a massive amount of other files which aren't needed.

 

After that you can mount this "smb server" through the Unassigend Disks Plugin.

 

Hello - I want to follow this concept, but from an Ubuntu client (instead of Windows)

However how do I do this ?

Create additional user... should be ok

created additional samba-user - should be also ok

However finally I cannot mount the ubuntu ext4 disk in unassigned devices plug in

It says permission denied - does it NEED write access ?

Because when I grant full write access there is no problem....

 

Link to comment

Hi,

 

if i execute the script i become the folloing error:

 

root@Avalon:/boot/config/scripts# sh backup_rsync.sh 
backup_rsync.sh: line 74: syntax error near unexpected token `>'
backup_rsync.sh: line 74: `    exec &> >(tee "${backup_path}/logs/${new_backup}.log")'
root@Avalon:/boot/config/scripts# 

 

How can i solve this?

 

Thank you!

  • Like 1
Link to comment
  • 2 weeks later...
1 hour ago, Braulio said:

The backup size always increases.

Each backup is a full backup which uses hardlinks to already existing files in previous backups. So if you create an archive from one backup, it can not take advantage of the hardlinks. Instead it copies all files.

 

The only solution is to create archives of multiple backups like one archive per week. But the archive type must support hardlinks preserving. Rar supports it for NTFS:

https://documentation.help/WinRAR/HELPSwOH.htm

 

But I do not suggest to use NTFS. NTFS has an extremely bad hardlink performance and it is not possible to backup all files that could be present on a Linux disk. For example you can't save files with a question mark in the filename and many others which are supported in Linux. Use ext4 or BTRFS. There is a free windows driver for BTRFS available.

 

Link to comment
3 hours ago, mgutt said:

Each backup is a full backup which uses hardlinks to already existing files in previous backups. So if you create an archive from one backup, it can not take advantage of the hardlinks. Instead it copies all files.

 

The only solution is to create archives of multiple backups like one archive per week. But the archive type must support hardlinks preserving. Rar supports it for NTFS:

https://documentation.help/WinRAR/HELPSwOH.htm

 

But I do not suggest to use NTFS. NTFS has an extremely bad hardlink performance and it is not possible to backup all files that could be present on a Linux disk. For example you can't save files with a question mark in the filename and many others which are supported in Linux. Use ext4 or BTRFS. There is a free windows driver for BTRFS available.

 

 

I thought I had only one folder with all the files (incremental) and each new folder (backup) pointed to the file location.

Thanks for the explanations. Your script turned out really good. Congratulations

 

Link to comment

Hi,

I am new to UnRaid. But not totally to Linux - I had a Open Media Vault Box since last week and changed the setup to UNRAID. All is working great an I am very pleased.

Now I need a backup solution and found this script. This seems to be great. Sadly I do not find a explanation how to set it up...I read the whole thread but did not find anything...so please point me in the right direction.

Thanks.

Best.

F.

 

 

Edited by FFrank
Link to comment

OK, I tried to get it done...But I failed 😞

 

I am not sure what triggers the script. Is it only on schedule or on connection of the drive? You mention both in the first post:

 

*******

Use the User Scripts Plugin (Unraid Apps) to execute it by schedule

Use the Unassigned Devices Plugin (Unraid Apps) to execute it after mounting a USB drive

*******

 

I found two places to enter the script. One in the Unassigned Devices Plugin and one in the User Scripts Plugin? How/where is it supposed to be?

 

It would be perfect if I could just plug in the drive and after getting a message(mail) that the backup is done I just unplug the drive.

 

I really want to learn more - so just point me to the right place where I find the info.

 

I will investigate further - hope I will manage later the day....

Thanks.

F.

 

Link to comment
48 minutes ago, FFrank said:

just plug in the drive

Then you want to use an Unassigned Devices script. Take a look at the UD support thread, there are examples in the first page.

 

You can go directly to the support thread for any of your plugins by clicking its Support Link on the Plugins page.

Link to comment
7 minutes ago, trurl said:

Then you want to use an Unassigned Devices script. Take a look at the UD support thread, there are examples in the first page.

 

You can go directly to the support thread for any of your plugins by clicking its Support Link on the Plugins page.

Great, thanks for the information.

But still it would be nice to know how this plugin is supposed to work.

 

Is the drive attached the whole time?

Is it always mounted?

How do I have to set it up?

 

Thanks so much for more information. It seams to be a great way to backup. But I want to understand it before fiddling arround ;-)

 

Best.

F.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.