rsync Incremental Backup


Recommended Posts

3 minutes ago, LyDjane said:

like this?

  "/mnt/user/appdata --exclude="/mnt/user/appdata/plex"

No. The appdata path in front is too much.

 

3 minutes ago, LyDjane said:

and if i want to exclude multiple folders?

Add multiple --exclude lines.

 

  • Like 1
Link to comment
7 hours ago, LyDjane said:

okay. So i give only exclude path?

You should check the script. You are looking at the wrong position. The exclude paths are not added to the jobs section. Scroll down to the rsync options.

Screenshot_20220308_062411.thumb.png.f353b9a685c1a2ffc91c37554d079ae2.png

 

And note: exclude paths are relative to the source path. So if you want to backup /mnt/user/appdata, but not /mnt/user/appdata/plex then you need to add --exclude=/plex/

 

If you want to exclude any subdir with the name plex, then use --exclude=plex/

 

If you want to exclude only the content of a dir use --exclude=plex/*

 

More examples:

https://askubuntu.com/a/469191/227119

  • Like 1
Link to comment
On 12/19/2021 at 8:17 PM, mgutt said:

I've released version 0.7 with the following enhancements:

# - Added support for SSH destinations (replaced find, rm and mv commands against pure rsync hacks)

 

....as I am going to need this feature in the near future...anyone cared to try this with opensshd running under Win11 (https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse)?

 

I need some extra space in order to swap/move some Backups around and the only space available is a couple of TB on my Win11 Desktop (running a Stablebit Drivepool and a 10G connection).

Just wanted to make sure, that I am not going to waste my time on this Windows based endeavor ;-)

 

 

TIA,

ford

Link to comment
  • 2 weeks later...

Short Question.

I want to transfer my backup to a remote server via script.

 

# user-defined rsync command
alias rsync='sshpass -p "PUT MY PASSWORD HERE" rsync -e "ssh -o StrictHostKeyChecking=no"'

# user-defined ssh command
alias ssh='sshpass -p "PUT MY PASSWORD HERE" ssh -o "StrictHostKeyChecking no"'

 

Is this right?
If i run the script ive got the error: sshpass command not found!

 

Thank you :)

Link to comment

Hello and thank you for your script. I´m using it to backup some important shares over to a remote server through a wireguard connection. My connection supports 50 Mbps up and the remote server has 300 Mbps down (fiber). I can see that my server is using the full bandwith of 50 Mbps when the script transfers files. The contents of the shares don´t change much on a day to day basis, but the script runs upwards of 10 hours a day, even when the log says it transfers almost no data per share. Maybe i dont fully understand the concept of this rsync script, but i thought, when the files don´t change it just creates a hardlink. When i read the logs correctly, the most time consuming part is the "managing" of folders etc., not the transfer itself. The remote mashine only has a Celeron J4125 (it´s only purpose is the backup), would an upgrade help?

Link to comment
56 minutes ago, Marc Heyer said:

i thought, when the files don´t change it just creates a hardlink.

Correct

 

Do the daily logs contain each day all files? (Should not)

 

59 minutes ago, Marc Heyer said:

through a wireguard connection

Which protocol are you using to transfer the files? SMB, SFTP, SSH?

Link to comment

I don´t think so. It lists them all, but transfers almost none. For example my nextcloud data share is listed with 142000 files, but created 0, deleted 0, transferred 0. Total bytes sent 3.75M, received 24,84K, speedup 18,959.32. The other shares list way less of that and the speedup is 784,059,192.20 on a 200GB share.

 

I have an NFS share mounted from the remote server to transfer to.

 

Everytime i check the logs, it is at the clean up stage of one of the shares.

 

Edit: almost forgot: Thank you for your reply!

Edited by Marc Heyer
Link to comment
  • 2 weeks later...
On 3/29/2022 at 2:13 PM, Marc Heyer said:

Everytime i check the logs, it is at the clean up stage of one of the shares.

 

Hmm.. then it needs to long to delete hardlinks of old backups. Does it display a message like "Delete outdated backup: xxx ... through rsync (slow)"?

Link to comment
23 hours ago, mgutt said:

Hmm.. then it needs to long to delete hardlinks of old backups. Does it display a message like "Delete outdated backup: xxx ... through rsync (slow)"?

No, doenst seem like it. I attached the output of one share below. Nothing with slow in this one and all the others. The script ran 15 hours today 😅

Spoiler

# #####################################
last_backup: '20220410_170850/'
Create incremental backup from /mnt/user/paperless_ng to /mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220411_164621 by using last backup /mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220410_170850/
cd..t...... ./

Number of files: 145 (reg: 138, dir: 7)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 278.25M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.003 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 6.66K
Total bytes received: 32

sent 6.66K bytes  received 32 bytes  382.63 bytes/sec
total size is 278.25M  speedup is 41,554.14
File count of rsync is 145
Make backup visible ...
... through local mv
renamed '/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/.20220411_164621' -> '/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220411_164621'
# #####################################
Clean up outdated backups
Keep daily backup: 20220411_164621
Keep daily backup: 20220410_170850
Keep daily backup: 20220407_160435
Keep daily backup: 20220406_160918
Keep daily backup: 20220405_220930
Keep daily backup: 20220404_164806
Keep daily backup: 20220403_164706
Keep daily backup: 20220402_163523
Keep daily backup: 20220401_162148
Keep daily backup: 20220331_173318
Keep daily backup: 20220330_140503
Keep daily backup: 20220329_140329
Keep daily backup: 20220328_202244
Keep daily backup: 20220327_143332
Delete outdated backup: 20220326_141141
Keep monthly backup: 20220227_155229
Keep monthly backup: 20220131_182110

 

Link to comment

And the "20220326_141141" contained only 145 files, too?!

 

What happens if you execute this through the terminal?

rm -r "/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220327_143332"

 

If this dir contains only around 145 files, it should be done in seconds. If not, then your problem is NFS or the target. Maybe the target filesystem is so slow while deleting hardlinks?!

Link to comment

Hello, and thanks for your fantastic script.
I use my UnRaid server mainly as a media server and would like to implement a "recycle bin" function against accidental deletion and encryption trojans. Is it possible to implement this with your script, so that the media library is backed up via hardlink on another share of the same HDD. Would this work or would you recommend another solution for this task without the need for extra HDD space?

Edited by IpanemaX
Link to comment
5 hours ago, IpanemaX said:

Would this work

It would cost twice storage space, so I don't think it's useful for you.

 

5 hours ago, IpanemaX said:

would you recommend another solution for this task without the need for extra HDD space?

I would use "cp --reflink=always" to sync all your files on the same HDD to a different directory.

 

As cp does a one-way sync, deleted files aren't deleted from the destination.

 

And with reflink the file does not occupy extra storage space (works only on the same disk and the mover kills reflinks as well, so a cache pool wouldn't be allowed).

 

But a simple sync would not help against ransomware or allow old versions of a document. For this it would be needed to change the destination. Maybe on daily basis and the destination folder gets the name Monday, Tuesday, etc?! It depends on your targets.

 

reflinks are really usefull for vdisk, too:

 

Another option would be the unraid plugin Recycle Bin, which works for SMB network shares.

 

 

Edited by mgutt
Link to comment
On 4/11/2022 at 8:17 PM, mgutt said:

And the "20220326_141141" contained only 145 files, too?!

 

What happens if you execute this through the terminal?

rm -r "/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220327_143332"

 

If this dir contains only around 145 files, it should be done in seconds. If not, then your problem is NFS or the target. Maybe the target filesystem is so slow while deleting hardlinks?!

I tried it, but modified the command to the delete the oldest daily backup because the one you picked was already gone. And it ran just a couple of seconds, maybe 10 seconds or so. The target is a remote unraid xfs share.

 

I found another user on reddit, who does it the same way and his script runs much shorter. Maybe i try from scratch sometime, when i have the time and access to the mashine.

Link to comment
  • 3 weeks later...

Hello - some basic questions from me I'm afriad, having used rsync for the inital transfers onto a new unraid from my existing separate drives (following Spaceinvader One's tutorial).

 

-Is there any kind of of verification built into rsync? Most of my archive is original raw video cards from shoots which are large and which I alwasy need to maintain the file structure for. Before unraid I used apps like Shotput Pro where there is a verification process as part of the transfer giving peace of mind. With these transfers I've just been doing the old right-click-properties and comparing the source and destination copies, and with rsync they seem to be accurate down to the last bit.. so just wondering if that's becasue there is a check of some kind, and I can not worry about doing that manual comparason?

 

-Am I right in thinking that once I've started a transfer using the console, the transfer will continue even if I turn off my PC (in this case the drives are connected direct to unraid as UD. 

 

-Is there any command I can type to check progress on a transfer, other than adding --progress to the origonal command? (for example if I close/reopen the console and lose the original progress, or if I log in on another computer at home and therefore a new console winow..

 

-Is there a command than can stack transfers? Say I have 3x UDs plugged in and want the transfers to happen one after the other overnight..

 

Thanks very much

 

Link to comment
10 hours ago, hughs said:

Is there any kind of of verification built into rsync?

If you add the --checksum Option to the rsync options it will verify. But this will be done on every backup and takes a huge amount of time.

 

Maybe it would be better to add the script twice and execute it with checksum only once per month.

  • Like 1
Link to comment

Thank you very much - can I check, does anything in the responce below indicate success? Or would there be an error message of some kind of it didn't pass the checksum?

 

Bill Sevesi HOF footage/Sevesi stills/welcome to the orange.tiff
          2.18M 100%    8.37MB/s    0:00:00 (xfr#97, to-chk=0/100)

sent 163.16G bytes  received 1.88K bytes  47.77M bytes/sec
total size is 163.12G  speedup is 1.00

Link to comment
On 5/4/2022 at 4:16 AM, hughs said:

-Am I right in thinking that once I've started a transfer using the console, the transfer will continue even if I turn off my PC (in this case the drives are connected direct to unraid as UD.

If you started rsync from the command line then by default the rsync command terminates when the console session is closed.

 

To avoid this install the 'screen' app (using the Nerd Tools plugin) and first start a screen session from the command line, and then run rsync under that.

Link to comment

For the moment I just want to say Thank You for this work. My use case is off site backup from an Unraid server to a small old Synology NAS with some USB disks attached. After first tries I'm very confident that it will suit my needs. The last time I did something similar, I used dirvish on a dedicated backup server. I liked it a lot, and originally was looking for a way to get that working on Unraid, when I stumbled upon this thread.

 

 

 

Link to comment
  • 1 month later...

Very much thank you for this,

 

i got my own stupid version of this running without the additional backup part :)

 

but one question, is there a way to get a more human friendly directory naming?

 

like

 

date +%Y-%m-%d_%H-%M-%S

 

or would this kill the script if i would change it everywhere?

Edited by Marcel40625
Link to comment
On 6/11/2022 at 9:06 AM, Marcel40625 said:

or would this kill the script if i would change it everywhere?

Yes and no.

 

At this position it obtains for example the month by extracting the 5th char of the dirname.

    year=${backup_name:0:4}
    month=${backup_name:4:2}
    day=${backup_name:6:2}

 

But I integrated a check, which will stop the script if it was not able to obtain only numbers:

    # all date parts must be integer
    if ! [[ "$year$month$day" =~ ^[0-9]+$ ]]; then
      echo "Error: $backup_name is not a backup!"
      continue
    fi

 

So it will fail, but not kill any backups or similar. Feel free to change the numbers, to it extracts the correct numbers. This should work in your case:

 

    year=${backup_name:0:4}
    month=${backup_name:5:2}
    day=${backup_name:8:2}

 

Link to comment
On 6/16/2022 at 9:12 AM, mgutt said:

Yes and no.

 

At this position it obtains for example the month by extracting the 5th char of the dirname.

    year=${backup_name:0:4}
    month=${backup_name:4:2}
    day=${backup_name:6:2}

 

But I integrated a check, which will stop the script if it was not able to obtain only numbers:

    # all date parts must be integer
    if ! [[ "$year$month$day" =~ ^[0-9]+$ ]]; then
      echo "Error: $backup_name is not a backup!"
      continue
    fi

 

So it will fail, but not kill any backups or similar. Feel free to change the numbers, to it extracts the correct numbers. This should work in your case:

 

    year=${backup_name:0:4}
    month=${backup_name:5:2}
    day=${backup_name:8:2}

 

 

Thanks for the answer, got it working myself 3 days ago :D  the grep was a little obstacle but got it working (needed to add the "-")
 

 last_backup=$(echo "$last_backup" | grep -oP "[0-9-_/]*" | sort -r | head -n1)

 

Link to comment
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.