mgutt Posted March 7, 2022 Author Share Posted March 7, 2022 3 minutes ago, LyDjane said: like this? "/mnt/user/appdata --exclude="/mnt/user/appdata/plex" No. The appdata path in front is too much. 3 minutes ago, LyDjane said: and if i want to exclude multiple folders? Add multiple --exclude lines. 1 Quote Link to comment
LyDjane Posted March 7, 2022 Share Posted March 7, 2022 3 hours ago, mgutt said: No. The appdata path in front is too much. Add multiple --exclude lines. okay. So i give only exclude path? Quote Link to comment
mgutt Posted March 8, 2022 Author Share Posted March 8, 2022 7 hours ago, LyDjane said: okay. So i give only exclude path? You should check the script. You are looking at the wrong position. The exclude paths are not added to the jobs section. Scroll down to the rsync options. And note: exclude paths are relative to the source path. So if you want to backup /mnt/user/appdata, but not /mnt/user/appdata/plex then you need to add --exclude=/plex/ If you want to exclude any subdir with the name plex, then use --exclude=plex/ If you want to exclude only the content of a dir use --exclude=plex/* More examples: https://askubuntu.com/a/469191/227119 1 Quote Link to comment
Ford Prefect Posted March 14, 2022 Share Posted March 14, 2022 On 12/19/2021 at 8:17 PM, mgutt said: I've released version 0.7 with the following enhancements: # - Added support for SSH destinations (replaced find, rm and mv commands against pure rsync hacks) ....as I am going to need this feature in the near future...anyone cared to try this with opensshd running under Win11 (https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse)? I need some extra space in order to swap/move some Backups around and the only space available is a couple of TB on my Win11 Desktop (running a Stablebit Drivepool and a 10G connection). Just wanted to make sure, that I am not going to waste my time on this Windows based endeavor TIA, ford Quote Link to comment
LyDjane Posted March 25, 2022 Share Posted March 25, 2022 Short Question. I want to transfer my backup to a remote server via script. # user-defined rsync command alias rsync='sshpass -p "PUT MY PASSWORD HERE" rsync -e "ssh -o StrictHostKeyChecking=no"' # user-defined ssh command alias ssh='sshpass -p "PUT MY PASSWORD HERE" ssh -o "StrictHostKeyChecking no"' Is this right? If i run the script ive got the error: sshpass command not found! Thank you Quote Link to comment
mgutt Posted March 26, 2022 Author Share Posted March 26, 2022 sshpass is not available in unRAID. You need to install it manually. Easiest method is to create the dir "/boot/extra" and paste this file and reboot: https://slackware.pkgs.org/current/slackers/sshpass-1.09-x86_64-1cf.txz.html Quote Link to comment
Marc Heyer Posted March 29, 2022 Share Posted March 29, 2022 Hello and thank you for your script. I´m using it to backup some important shares over to a remote server through a wireguard connection. My connection supports 50 Mbps up and the remote server has 300 Mbps down (fiber). I can see that my server is using the full bandwith of 50 Mbps when the script transfers files. The contents of the shares don´t change much on a day to day basis, but the script runs upwards of 10 hours a day, even when the log says it transfers almost no data per share. Maybe i dont fully understand the concept of this rsync script, but i thought, when the files don´t change it just creates a hardlink. When i read the logs correctly, the most time consuming part is the "managing" of folders etc., not the transfer itself. The remote mashine only has a Celeron J4125 (it´s only purpose is the backup), would an upgrade help? Quote Link to comment
mgutt Posted March 29, 2022 Author Share Posted March 29, 2022 56 minutes ago, Marc Heyer said: i thought, when the files don´t change it just creates a hardlink. Correct Do the daily logs contain each day all files? (Should not) 59 minutes ago, Marc Heyer said: through a wireguard connection Which protocol are you using to transfer the files? SMB, SFTP, SSH? Quote Link to comment
Marc Heyer Posted March 29, 2022 Share Posted March 29, 2022 (edited) I don´t think so. It lists them all, but transfers almost none. For example my nextcloud data share is listed with 142000 files, but created 0, deleted 0, transferred 0. Total bytes sent 3.75M, received 24,84K, speedup 18,959.32. The other shares list way less of that and the speedup is 784,059,192.20 on a 200GB share. I have an NFS share mounted from the remote server to transfer to. Everytime i check the logs, it is at the clean up stage of one of the shares. Edit: almost forgot: Thank you for your reply! Edited March 29, 2022 by Marc Heyer Quote Link to comment
mgutt Posted April 10, 2022 Author Share Posted April 10, 2022 On 3/29/2022 at 2:13 PM, Marc Heyer said: Everytime i check the logs, it is at the clean up stage of one of the shares. Hmm.. then it needs to long to delete hardlinks of old backups. Does it display a message like "Delete outdated backup: xxx ... through rsync (slow)"? Quote Link to comment
Marc Heyer Posted April 11, 2022 Share Posted April 11, 2022 23 hours ago, mgutt said: Hmm.. then it needs to long to delete hardlinks of old backups. Does it display a message like "Delete outdated backup: xxx ... through rsync (slow)"? No, doenst seem like it. I attached the output of one share below. Nothing with slow in this one and all the others. The script ran 15 hours today 😅 Spoiler # ##################################### last_backup: '20220410_170850/' Create incremental backup from /mnt/user/paperless_ng to /mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220411_164621 by using last backup /mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220410_170850/ cd..t...... ./ Number of files: 145 (reg: 138, dir: 7) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 278.25M bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.003 seconds File list transfer time: 0.000 seconds Total bytes sent: 6.66K Total bytes received: 32 sent 6.66K bytes received 32 bytes 382.63 bytes/sec total size is 278.25M speedup is 41,554.14 File count of rsync is 145 Make backup visible ... ... through local mv renamed '/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/.20220411_164621' -> '/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220411_164621' # ##################################### Clean up outdated backups Keep daily backup: 20220411_164621 Keep daily backup: 20220410_170850 Keep daily backup: 20220407_160435 Keep daily backup: 20220406_160918 Keep daily backup: 20220405_220930 Keep daily backup: 20220404_164806 Keep daily backup: 20220403_164706 Keep daily backup: 20220402_163523 Keep daily backup: 20220401_162148 Keep daily backup: 20220331_173318 Keep daily backup: 20220330_140503 Keep daily backup: 20220329_140329 Keep daily backup: 20220328_202244 Keep daily backup: 20220327_143332 Delete outdated backup: 20220326_141141 Keep monthly backup: 20220227_155229 Keep monthly backup: 20220131_182110 Quote Link to comment
mgutt Posted April 11, 2022 Author Share Posted April 11, 2022 And the "20220326_141141" contained only 145 files, too?! What happens if you execute this through the terminal? rm -r "/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220327_143332" If this dir contains only around 145 files, it should be done in seconds. If not, then your problem is NFS or the target. Maybe the target filesystem is so slow while deleting hardlinks?! Quote Link to comment
IpanemaX Posted April 14, 2022 Share Posted April 14, 2022 (edited) Hello, and thanks for your fantastic script. I use my UnRaid server mainly as a media server and would like to implement a "recycle bin" function against accidental deletion and encryption trojans. Is it possible to implement this with your script, so that the media library is backed up via hardlink on another share of the same HDD. Would this work or would you recommend another solution for this task without the need for extra HDD space? Edited April 14, 2022 by IpanemaX Quote Link to comment
mgutt Posted April 14, 2022 Author Share Posted April 14, 2022 (edited) 5 hours ago, IpanemaX said: Would this work It would cost twice storage space, so I don't think it's useful for you. 5 hours ago, IpanemaX said: would you recommend another solution for this task without the need for extra HDD space? I would use "cp --reflink=always" to sync all your files on the same HDD to a different directory. As cp does a one-way sync, deleted files aren't deleted from the destination. And with reflink the file does not occupy extra storage space (works only on the same disk and the mover kills reflinks as well, so a cache pool wouldn't be allowed). But a simple sync would not help against ransomware or allow old versions of a document. For this it would be needed to change the destination. Maybe on daily basis and the destination folder gets the name Monday, Tuesday, etc?! It depends on your targets. reflinks are really usefull for vdisk, too: Another option would be the unraid plugin Recycle Bin, which works for SMB network shares. Edited April 14, 2022 by mgutt Quote Link to comment
Marc Heyer Posted April 15, 2022 Share Posted April 15, 2022 On 4/11/2022 at 8:17 PM, mgutt said: And the "20220326_141141" contained only 145 files, too?! What happens if you execute this through the terminal? rm -r "/mnt/remotes/100.72.190.35_backup/Shares/paperless_ng/20220327_143332" If this dir contains only around 145 files, it should be done in seconds. If not, then your problem is NFS or the target. Maybe the target filesystem is so slow while deleting hardlinks?! I tried it, but modified the command to the delete the oldest daily backup because the one you picked was already gone. And it ran just a couple of seconds, maybe 10 seconds or so. The target is a remote unraid xfs share. I found another user on reddit, who does it the same way and his script runs much shorter. Maybe i try from scratch sometime, when i have the time and access to the mashine. Quote Link to comment
hughs Posted May 4, 2022 Share Posted May 4, 2022 Hello - some basic questions from me I'm afriad, having used rsync for the inital transfers onto a new unraid from my existing separate drives (following Spaceinvader One's tutorial). -Is there any kind of of verification built into rsync? Most of my archive is original raw video cards from shoots which are large and which I alwasy need to maintain the file structure for. Before unraid I used apps like Shotput Pro where there is a verification process as part of the transfer giving peace of mind. With these transfers I've just been doing the old right-click-properties and comparing the source and destination copies, and with rsync they seem to be accurate down to the last bit.. so just wondering if that's becasue there is a check of some kind, and I can not worry about doing that manual comparason? -Am I right in thinking that once I've started a transfer using the console, the transfer will continue even if I turn off my PC (in this case the drives are connected direct to unraid as UD. -Is there any command I can type to check progress on a transfer, other than adding --progress to the origonal command? (for example if I close/reopen the console and lose the original progress, or if I log in on another computer at home and therefore a new console winow.. -Is there a command than can stack transfers? Say I have 3x UDs plugged in and want the transfers to happen one after the other overnight.. Thanks very much Quote Link to comment
mgutt Posted May 4, 2022 Author Share Posted May 4, 2022 10 hours ago, hughs said: Is there any kind of of verification built into rsync? If you add the --checksum Option to the rsync options it will verify. But this will be done on every backup and takes a huge amount of time. Maybe it would be better to add the script twice and execute it with checksum only once per month. 1 Quote Link to comment
hughs Posted May 4, 2022 Share Posted May 4, 2022 Thank you very much - can I check, does anything in the responce below indicate success? Or would there be an error message of some kind of it didn't pass the checksum? Bill Sevesi HOF footage/Sevesi stills/welcome to the orange.tiff 2.18M 100% 8.37MB/s 0:00:00 (xfr#97, to-chk=0/100) sent 163.16G bytes received 1.88K bytes 47.77M bytes/sec total size is 163.12G speedup is 1.00 Quote Link to comment
mgutt Posted May 5, 2022 Author Share Posted May 5, 2022 If it's copied, then the checksum differs. Quote Link to comment
itimpi Posted May 5, 2022 Share Posted May 5, 2022 On 5/4/2022 at 4:16 AM, hughs said: -Am I right in thinking that once I've started a transfer using the console, the transfer will continue even if I turn off my PC (in this case the drives are connected direct to unraid as UD. If you started rsync from the command line then by default the rsync command terminates when the console session is closed. To avoid this install the 'screen' app (using the Nerd Tools plugin) and first start a screen session from the command line, and then run rsync under that. Quote Link to comment
kannznichkaufen Posted May 9, 2022 Share Posted May 9, 2022 For the moment I just want to say Thank You for this work. My use case is off site backup from an Unraid server to a small old Synology NAS with some USB disks attached. After first tries I'm very confident that it will suit my needs. The last time I did something similar, I used dirvish on a dedicated backup server. I liked it a lot, and originally was looking for a way to get that working on Unraid, when I stumbled upon this thread. Quote Link to comment
Marcel40625 Posted June 11, 2022 Share Posted June 11, 2022 (edited) Very much thank you for this, i got my own stupid version of this running without the additional backup part but one question, is there a way to get a more human friendly directory naming? like date +%Y-%m-%d_%H-%M-%S or would this kill the script if i would change it everywhere? Edited June 11, 2022 by Marcel40625 Quote Link to comment
mgutt Posted June 16, 2022 Author Share Posted June 16, 2022 On 6/11/2022 at 9:06 AM, Marcel40625 said: or would this kill the script if i would change it everywhere? Yes and no. At this position it obtains for example the month by extracting the 5th char of the dirname. year=${backup_name:0:4} month=${backup_name:4:2} day=${backup_name:6:2} But I integrated a check, which will stop the script if it was not able to obtain only numbers: # all date parts must be integer if ! [[ "$year$month$day" =~ ^[0-9]+$ ]]; then echo "Error: $backup_name is not a backup!" continue fi So it will fail, but not kill any backups or similar. Feel free to change the numbers, to it extracts the correct numbers. This should work in your case: year=${backup_name:0:4} month=${backup_name:5:2} day=${backup_name:8:2} Quote Link to comment
Marcel40625 Posted June 17, 2022 Share Posted June 17, 2022 On 6/16/2022 at 9:12 AM, mgutt said: Yes and no. At this position it obtains for example the month by extracting the 5th char of the dirname. year=${backup_name:0:4} month=${backup_name:4:2} day=${backup_name:6:2} But I integrated a check, which will stop the script if it was not able to obtain only numbers: # all date parts must be integer if ! [[ "$year$month$day" =~ ^[0-9]+$ ]]; then echo "Error: $backup_name is not a backup!" continue fi So it will fail, but not kill any backups or similar. Feel free to change the numbers, to it extracts the correct numbers. This should work in your case: year=${backup_name:0:4} month=${backup_name:5:2} day=${backup_name:8:2} Thanks for the answer, got it working myself 3 days ago the grep was a little obstacle but got it working (needed to add the "-") last_backup=$(echo "$last_backup" | grep -oP "[0-9-_/]*" | sort -r | head -n1) Quote Link to comment
Archonw Posted July 21, 2022 Share Posted July 21, 2022 How do i restore my backup? my backup is in /mnt/user/unraidbackup/appdata/20220717_043012/mariadb/ and has to be restored to /mnt/user/appdata/mariadb/ which attributes do i need for rsync? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.