schoppehermann Posted January 2, 2022 Share Posted January 2, 2022 (edited) Thank you for the Script and Update! I would like to change the Skript Version from v0.6 to 1.3 and the Path. What´s the best Way to change the Backup Paht and preserve my old Rsync Backups? "/mnt/user/backup/shares/daten" >>> "/mnt/user/backup/daten" Edited January 2, 2022 by schoppehermann Quote Link to comment
mgutt Posted January 2, 2022 Author Share Posted January 2, 2022 6 hours ago, schoppehermann said: What´s the best Way to change the Backup Paht and preserve my old Rsync Backups? You could move everything to a new path as follows (change diskX as needed): mkdir /mnt/diskX/backup/daten mv /mnt/diskX/backup/shares/daten/* /mnt/diskX/backup/daten/ The old backups will be preserved, but on the next backup hardlinking (space saving) does not work. If you even want that, you need to move the files of your most recent backup in the upper dir as follows: mv /mnt/diskX/backup/daten/20220102_044001/daten/* /mnt/diskX/backup/daten/20220102_044001 Quote Link to comment
schoppehermann Posted January 3, 2022 Share Posted January 3, 2022 (edited) Thank´s for your fast Reply, it works werry well! I moved the last Backup one Level up: Quote mv /mnt/user/backup/daten/20220103_044001/daten/* /mnt/user/backup/daten/20220103_044001 And for the same Structure in my Monthly Backups: Quote mv /mnt/user/backup/daten/20210701_044001/daten/* /mnt/user/backup/daten/20210701_044001/ mv /mnt/user/backup/daten/20210801_044001/daten/* /mnt/user/backup/daten/20210801_044001/ mv /mnt/user/backup/daten/20210901_044001/daten/* /mnt/user/backup/daten/20210901_044001/ mv /mnt/user/backup/daten/20211001_044001/daten/* /mnt/user/backup/daten/20211001_044001/ mv /mnt/user/backup/daten/20211101_044001/daten/* /mnt/user/backup/daten/20211101_044001/ I hope second Step works to Edited January 3, 2022 by schoppehermann Quote Link to comment
mgutt Posted January 3, 2022 Author Share Posted January 3, 2022 8 hours ago, schoppehermann said: And for the same Structure in my Monthly Backups: Was not necessary (but does not hurt) as hardlinks won't break if they are in a subdir and my script uses only the most recent backup to compare and decide which files only need hardlinks. That's why I posted only one mv command. Quote Link to comment
Shantarius Posted January 4, 2022 Share Posted January 4, 2022 Hi mgutt, thank you for the superb script. I have a feature request: Is it possible to have an option for excluding files or paths? Best regards Chris Quote Link to comment
mgutt Posted January 4, 2022 Author Share Posted January 4, 2022 2 hours ago, Shantarius said: thank you for the superb script. I have a feature request: Is it possible to have an option for excluding files or paths? Its already there: --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP" --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP" --exclude="Cache/" # exclude dirs with the name "Cache" Quote Link to comment
Shantarius Posted January 4, 2022 Share Posted January 4, 2022 (edited) Hi mgutt, i have changed the rsync command in the script (V0.6) to rsync -av --stats --exclude-from="/boot/config/scripts/exclude.list" --delete --link-dest="${backup_path}/${last_backup}" "${source_path}" "${backup_path}/.${new_backup}" The file exclude.list contains this: /mnt/zpool/VMs/Debian11_105/ /mnt/zpool/VMs/Ubuntu21_102/ /mnt/zpool/VMs/Win10_106/ /mnt/zpool/Docker/Dockerimage/ But the script don't ignore the directorys in the exclude.list Can you say why? Thanks! Edited January 4, 2022 by Shantarius Quote Link to comment
mgutt Posted January 4, 2022 Author Share Posted January 4, 2022 37 minutes ago, Shantarius said: Can you say why? I'd say you need to use the relative path and not the absolute path. This means if you backup from "/mnt/zpool/" to /mnt/user/Backups, you need to add "VMs/Debian11_105/" to your exclude list file. Quote Link to comment
itsalljustdata Posted January 5, 2022 Share Posted January 5, 2022 (edited) I got myself moderately tangled up changing my parameters when copying over your latest version (user stupidity on my half), so do you fancy making these changes so that it can be config based (on a yaml file) if you call the script with no parameters then it uses whatever values are hardcoded in the script. Otherwise it'll parse the YAML file you specify and use those values. It also allows you to more easily have multiple backup sets running in parallel with a single backup script # ##################################### # Settings # ##################################### # rsync options which are used while creating the full and incremental backup rsync_options=( # --dry-run --archive # same as --recursive --links --perms --times --group --owner --devices --specials --human-readable # output numbers in a human-readable format --itemize-changes # output a change-summary for all updates --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP" --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP" --exclude="[Cc][Aa][Cc][Hh][Ee]/" # exclude dirs with the name "Cache" or "cache" ) # set empty dir empty_dir="/tmp/${0//\//_}" if [ "${1}" == "" ]; then backupBase="[email protected]:/mnt/user/backup/beast" # backup source to destination backup_jobs=( # source # destination "/boot" "$backupBase/boot" "/mnt/user/scan" "$backupBase/scan" ) # keep backups of the last X days keep_days=14 # keep multiple backups of one day for X days keep_days_multiple=1 # keep backups of the last X months keep_months=12 # keep backups of the last X years keep_years=3 # keep the most recent X failed backups keep_fails=3 # notify if the backup was successful (1 = notify) notification_success=0 # notify if last backup is older than X days notification_backup_older_days=30 # create destination if it does not exist create_destination=1 # backup does not fail if files vanished during transfer https://linux.die.net/man/1/rsync#:~:text=vanished skip_error_vanished_source_files=1 # backup does not fail if source path returns "host is down". # This could happen if the source is a mounted SMB share, which is offline. skip_error_host_is_down=1 # backup does not fail if file transfers return "host is down" # This could happen if the source is a mounted SMB share, which went offline during transfer skip_error_host_went_down=1 # backup does not fail, if source path does not exist, which for example happens if the source is an unmounted SMB share skip_error_no_such_file_or_directory=1 # a backup fails if it contains less than X files backup_must_contain_files=2 # a backup fails if more than X % of the files couldn't be transfered because of "Permission denied" errors permission_error_treshold=20 else if ! [ -f yaml.sh ]; then wget https://raw.githubusercontent.com/jasperes/bash-yaml/master/script/yaml.sh fi if ! [ -f "${1}" ]; then echo "File \"${1}\" not found" exit 1 fi source yaml.sh create_variables "${1}" empty_dir+="." empty_dir+=$(basename "${1}") fi my YAML file backup_jobs: -/boot [email protected]:/mnt/user/backup/beast/boot -/mnt/user/scan [email protected]:/mnt/user/backup/beast/scan keep: days: 14 days_multiple: 14 months: 12 years: 3 fails: 3 notification: # notify if the backup was successful (1 = notify) success: 0 # notify if last backup is older than X days backup_older_days: 30 # create destination if it does not exist create_destination: 1 # a backup fails if it contains less than X files backup_must_contain_files: 2 # a backup fails if more than X % of the files couldn't be transfered because of "Permission denied" errors permission_error_treshold: 20 skip_error: # backup does not fail if files vanished during transfer https://linux.die.net/man/1/rsync#:~:text=vanished vanished_source_files: 1 # backup does not fail, if source path does not exist, which for example happens if the source is an unmounted SMB share no_such_file_or_directory: 1 host: # backup does not fail if source path returns "host is down". # This could happen if the source is a mounted SMB share, which is offline. is_down: 1 # backup does not fail if file transfers return "host is down". # This could happen if the source is a mounted SMB share, which went offline during transfer went_down: 1 also # make script race condition safe if [[ -d "${empty_dir}" ]] || ! mkdir "${empty_dir}"; then echo "Script is already running!" && exit 1; fi; trap 'rmdir "${empty_dir}"' EXIT; and obv remove the setting of "empty_dir" just above the loop Edited January 5, 2022 by Meles Meles Quote Link to comment
itsalljustdata Posted January 5, 2022 Share Posted January 5, 2022 (edited) Also... i'm backing up from one unRAID server to another (via ssh), and for some reason I get errors pop up every time it makes a SSH connection. Quote hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted so that I can get rid of these in the (final) log files, i'm "sed"ding them out just before I move the log to the destination # hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted # update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted sed -i '/hostfile_replace_entries/d' "$log_file" sed -i '/update_known_hosts/d' "$log_file" # move log file to destination rsync --remove-source-files "$log_file" "$dst_path/$new_backup/" || rsync --remove-source-files "$log_file" "$dst_path/.$new_backup/" could you either a) pop this into your code or b) give me some sort of clue as to why the blazes i'm getting this message! ta Edited January 5, 2022 by Meles Meles Quote Link to comment
mgutt Posted January 5, 2022 Author Share Posted January 5, 2022 Had an idea, which I need to note for myself: Before rsync starts, create reflink copies of vdisks in the new backup dir. Needs testing, but I hope it will update only some bytes, instead if copying the whole file. But I don't know if rsync compares the destination dir at all, if link-dest is used. Or isn't this needed at all, because rsync already transfers only the changed bytes? Quote Link to comment
Shantarius Posted January 6, 2022 Share Posted January 6, 2022 On 1/4/2022 at 10:24 PM, mgutt said: I'd say you need to use the relative path and not the absolute path. This means if you backup from "/mnt/zpool/" to /mnt/user/Backups, you need to add "VMs/Debian11_105/" to your exclude list file. Hi mgutt, thank you for your answer. This workes for me! source_paths=( "/mnt/zpool/Docker" "/mnt/zpool/VMs" "/mnt/zpool/nextcloud_data" ) backup_path="/mnt/user/Backup_zpool" ... rsync -av --exclude-from="/boot/config/scripts/exclude.list" --stats --delete --link-dest="${backup_path}/${last_backup}" "${source_path}" "${backup_path}/.${new_backup}" excludes.list Debian11_105/ Ubuntu21_102/ Win10_106/ 1 Quote Link to comment
mgutt Posted January 7, 2022 Author Share Posted January 7, 2022 On 1/5/2022 at 4:33 AM, Meles Meles said: hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted This happens because /root/.ssh is linked to the flash drive and the flash drive does not support hardlinks (as it uses FAT): ls -lah | grep .ssh lrwxrwxrwx 1 root root 21 Apr 7 2021 .ssh -> /boot/config/ssh/root/ On 1/5/2022 at 4:33 AM, Meles Meles said: why the blazes i'm getting this message! It is produced by OpenSSH and is patched in various projects: https://github.com/termux/termux-packages/issues/2909 https://github.com/haikuports/haikuports/issues/6018 I don't know if it is patched in OpenSSH itself?! As Unraid 6.9.3 uses a relatively OpenSSH version, it does not seem so: ssh -V OpenSSH_8.4p1, OpenSSL 1.1.1j 16 Feb 2021 One solution would be to remove the link and copy the file from the usb flash drive: rm /root/.ssh cp /boot/config/ssh/root/ /root/.ssh The downside of this solution is that any changes to /root/.ssh are not permanently saved in your unraid configuration and are lost on reboot. So you maybe should add an daily check as a script which copies the file back to your flash drive if the timestamp differs. Quote Link to comment
mgutt Posted January 7, 2022 Author Share Posted January 7, 2022 On 1/5/2022 at 3:52 AM, Meles Meles said: if you call the script with no parameters At first this script is created to use it as an Unraid users script which does not allow the usage of parameters, but yes, it would be nice having an external configuration file or being able to pass parameters to use it as an stand-alone script. I'm annoyed by myself changing the settings on each update and once I even forgot a setting as you did😅 But I don't think we should use bash and YAML at the same time as this would confuse the user. Instead we should use only bash like "source settings.conf" or change the settings area so it uses YAML, too. For example (untested): cat > /tmp/${0//\//_}_settings.yaml << EOF # ##################################### # Settings # ##################################### backup_jobs: -/boot [email protected]:/mnt/user/backup/beast/boot -/mnt/user/scan [email protected]:/mnt/user/backup/beast/scan keep: days: 14 days_multiple: 14 months: 12 years: 3 fails: 3 # ##################################### # Script # ##################################### EOF # parse parameters for i in "$@"; do case $i in --include=*) settings_file="${i#*=}"; shift ;; *) echo "Unknown parameter passed: $1"; exit 1 ;; esac done # use inline settings if [[ ! "$settings_file" ]]; then $settings_file="/tmp/${0//\//_}_settings.yaml" fi # parse settings source yaml.sh create_variables "$settings_file" Example to use an external config file: rsynclink.sh include=/boot/config/backupjob.yaml Then we only need to decide if we use only bash or only YAML. Quote Link to comment
bclinton Posted January 12, 2022 Share Posted January 12, 2022 (edited) Greetings. I have been happily using .60 for a while now and noticed I am a few versions behind in added features. I ran the 1.3 script on a spare 8TB HD to see what changed. These are my backup paths at total about 4 TB of data. The 1.3 script takes about 2 days to run with the drive in a USB cradle. I was wondering if there was a way to have all backup paths included in one destination folder. When the script finished, I had 3 separate folders since the backup time spanned 3 days. In other words, can I change something in the script to place all 5 backup locations in one destination folder even if the backup takes multiple days? So, if I start the backup on 1-12-22 I will have one folder on the backup disk with all 5 backups in that folder (1-12-22)? (similar to the way script .6 named backup destination folders?) backup paths "/mnt/user/data" "/mnt/disks/backup3/data" "/mnt/disk2" "/mnt/disks/backup3/disk2" "/mnt/user/backups" "/mnt/disks/backup3/backups" "/mnt/user/domains" "/mnt/disks/backup3/domains" "/mnt/user/isos" "/mnt/disks/backup3/isos" Edited January 12, 2022 by bclinton Quote Link to comment
mgutt Posted January 12, 2022 Author Share Posted January 12, 2022 4 hours ago, bclinton said: , if I start the backup on 1-12-22 I will have one folder on the backup disk with all 5 backups in that folder Sorry, not possible. Even if your jobs would start at the same day: the timestamp is accurate to the second. It will be never the same. Maybe it's possible by passing only /mnt as source and using the rsync options to include / exclude what you want in your backup. I'll test that. PS thank you for your donation! Quote Link to comment
bclinton Posted January 12, 2022 Share Posted January 12, 2022 (edited) 5 hours ago, mgutt said: Sorry, not possible. Even if your jobs would start at the same day: the timestamp is accurate to the second. It will be never the same. Maybe it's possible by passing only /mnt as source and using the rsync options to include / exclude what you want in your backup. I'll test that. PS thank you for your donation! Ah. that didn't occur to me.....I think I got it working using the exclude option like you suggested....did a few dry runs to test and it is now backing up the data. Will take about a 30 hours so I'll know tomorrow. I'm wondering what will happen when I do another backup next week to the data. Should I see another folder with the new date along with the most recent incremental copy of the data inside that folder? This is a screen grab of my root on the back up drive along with my change to the script. It looks like the data is being placed in the .01-12-22 folder in their correct sub folders. (I removed the H:M:S: variable so it only used the date) The folder name starts with a "." - is that normal? backup_jobs=( # source # destination "/mnt/user" "/mnt/disks/backup3" Excludes - rsync_options=( --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP" --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP" --exclude="Cache/" # exclude dirs with the name "Cache" --exclude="[Pp][Ll][Ee][Xx]/" --exclude="[Aa][Pp][Pp][Dd][Aa][Tt][Aa]/" --exclude="[Dd][Oo][Ww][Nn][Ll][Oo][Aa][Dd][Ss]/" --exclude="[Ss][Yy][Ss][Tt][Ee][Mm]/" backup drive.... Thanks for the help! Edited January 12, 2022 by bclinton Quote Link to comment
mgutt Posted January 12, 2022 Author Share Posted January 12, 2022 Yeah exclude works, but by that you need to add every new folder which is added under /mnt/user. Instead try this if you use /mnt/user as source: --include="data**" --include="backups**" --include="isos**" --include="domains**" --exclude="*" By that everything is excluded except the includes Or if you use /mnt as source: --include="disk2**" --include="user/" --include="user/backups**" --include="user/data**" --include="user/isos**" --include="user/domains**" --exclude="*" Note: Each parent dir needs a separate include as you can see for "user/". Note: You don't need the other exclude entries for TMP etc. Those are only examples if you like to match folder names case insensitive. 1 Quote Link to comment
mgutt Posted January 12, 2022 Author Share Posted January 12, 2022 28 minutes ago, bclinton said: The folder name starts with a "." - is that normal? Yes. This means the backup is still in creation. It will be renamed after it has been (successfully) finished. 1 Quote Link to comment
bclinton Posted January 12, 2022 Share Posted January 12, 2022 19 minutes ago, mgutt said: Yes. This means the backup is still in creation. It will be renamed after it has been (successfully) finished. Awesome. Thanks again! Quote Link to comment
schwarzzrawhcs Posted January 18, 2022 Share Posted January 18, 2022 On 7/7/2021 at 6:17 PM, mgutt said: a) enable unix extensions in the SMB settings of the Synology NAS or b) mount unraid on your Synology NASand execute the script on the Synology NAS Hi, love this script. But i ran into the the same problem as someone before when creating an Backup on a Synology NAS. Hardlinks wont get created and so there is alway a full backup. I searched for an "unix extensions" option in the Synology SMB settings but coudnt find one. So i added "unix extensions=yes" into the smb.conf under /etc/samba via Telnet and restarted the NAS, but without success... Does someone have any idea what i am missing? Thanks Quote Link to comment
mgutt Posted January 18, 2022 Author Share Posted January 18, 2022 5 hours ago, schwarzzrawhcs said: Does someone have any idea what i am missing? I'm not sure, but are unix extensions only available for SMB1? There seems to be a special option for SMB2, but it seems to be unavailable by now: https://www.reddit.com/r/archlinux/comments/fx33rc/posix_smb311_extensions_unsupported_with_current/ I would test to mount the Synology share manually and set version 1. Quote Link to comment
schwarzzrawhcs Posted January 18, 2022 Share Posted January 18, 2022 4 hours ago, mgutt said: I'm not sure, but are unix extensions only available for SMB1? There seems to be a special option for SMB2, but it seems to be unavailable by now: https://www.reddit.com/r/archlinux/comments/fx33rc/posix_smb311_extensions_unsupported_with_current/ I would test to mount the Synology share manually and set version 1. Ok thanks for the response. I did that by restricting the maximal SMB protocol to SMB1. But now i cant connect it as a remote share in Unraid. In windows i was able to install support for SMB1 and got it working, but not in Unraid even if NetBIOS is set to yes in SMB settings... Sorry i am a really new to this stuff, thanks for the help! Quote Link to comment
itsalljustdata Posted January 19, 2022 Share Posted January 19, 2022 Because I am going unRAID -> unRAID via SSH i'm getting the issue with the "hostfile_replace_entries" error (notwithstanding the workaround you suggested above). As such the setting of "last_backup" is not working properly (as the errors are coming out in the rsync listing. # obtain last backup if last_backup=$(rsync --dry-run --recursive --itemize-changes --exclude="*/*/" --include="[0-9]*/" --exclude="*" "$dst_path/" "$empty_dir" 2>&1); then last_backup=$(echo "$last_backup" | grep -v Operation | grep -oP "[0-9_/]*" | sort -r | head -n1) putting the "| grep -v Operation " in there cures this. Can you incorporate this into your next version please. Cheers! Quote Link to comment
itsalljustdata Posted January 19, 2022 Share Posted January 19, 2022 On 1/7/2022 at 8:42 AM, mgutt said: This happens because /root/.ssh is linked to the flash drive and the flash drive does not support hardlinks (as it uses FAT): ls -lah | grep .ssh lrwxrwxrwx 1 root root 21 Apr 7 2021 .ssh -> /boot/config/ssh/root/ It is produced by OpenSSH and is patched in various projects: https://github.com/termux/termux-packages/issues/2909 https://github.com/haikuports/haikuports/issues/6018 I don't know if it is patched in OpenSSH itself?! As Unraid 6.9.3 uses a relatively OpenSSH version, it does not seem so: ssh -V OpenSSH_8.4p1, OpenSSL 1.1.1j 16 Feb 2021 One solution would be to remove the link and copy the file from the usb flash drive: The downside of this solution is that any changes to /root/.ssh are not permanently saved in your unraid configuration and are lost on reboot. So you maybe should add an daily check as a script which copies the file back to your flash drive if the timestamp differs. For posterity here's what i've done to fix it... at the bottom of /boot/config/go i added rm /root/.ssh cp -R /boot/config/ssh/root/ /root/.ssh and then i've done a User Script (scheduled hourly) to sync the data back to the flash drive rsync -au /root/.ssh/ /boot/config/ssh/root/ Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.