Cpt. Chaz Posted December 17, 2020 Share Posted December 17, 2020 (edited) In my previous post found here, i outlined a multi-step process with 2 different scripts and 2 cron jobs for automating the unraid flash zip backup process using the User Scripts Plugin. Thanks to @nitewolfgtr and @sreknob i've refined it down to one simple "fill in the blanks" script. I've also added an optional UI notification. This script backs up a zipped copy of your unraid flash disk to a location of your choice, removes old backups after specified number of days, as well as the optional notification upon completion Due to the changes, i've marked the other OP outdated and now refer to this script. While i have tested (version 6.8.3 only) and currently run this on all 3 of my unraid servers with reasonable confidence, i stress for anyone that decides to try, MANUALLY BACKUP YOUR FLASH DRIVE FIRST BEFORE TRYING. if you are at all nervous or unfamiliar, please don't try this - i don't assume any liability here for data loss. Use at your own risk, this is still a work in progress. Here's the script, with explanation underneath currently for unraid version 6.8.3 AND 6.9.0-rc2 (version 6.9b30 see below): #!/bin/bash #### SECTION 1 ####------------------------------------------------------------------------------------------------------ #dir = WHATEVER FOLDER PATH YOU WANT TO SAVE TO dir="/insert/your/path/here" echo 'Executing native unraid backup script' /usr/local/emhttp/webGui/scripts/flash_backup #### SECTION 2 ####------------------------------------------------------------------------------------------------------ echo 'Remove symlink from emhttp' find /usr/local/emhttp/ -maxdepth 1 -name '*flash-backup-*.zip' -delete sleep 5 #### SECTION 3 ####------------------------------------------------------------------------------------------------------ if [ ! -d "$dir" ] ; then echo "making directory as it does not yet exist" # make the directory as it doesnt exist mkdir -vp "$dir" else echo "As $dir exists continuing." fi #### SECTION 4 ####------------------------------------------------------------------------------------------------------ echo 'Move Flash Zip Backup from Root to Backup Destination' mv /*-flash-backup-*.zip "$dir" sleep 5 #### SECTION 5 ####------------------------------------------------------------------------------------------------------ echo 'Deleting Old Backups' #ENTER NUMERIC VALUE OF DAYS AFTER "-MTIME +" find "$dir"* -mtime +10 -exec rm -rfv {} \; echo 'All Done' #### SECTION 6 ####------------------------------------------------------------------------------------------------------ #UNCOMMENT THE NEXT LINE TO ENABLE GUI NOTIFICATION UPON COMPLETION #/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Flash Zip Backup" -d "A copy of the YOURTOWERNAME unraid flash disk has been backed up" -i "normal" exit Here's the slightly modified script if you're running unraid version 6.9b30: Copy and paste the appropriate version above into a new blank script in the User Scripts Plugin, then proceed to edit per below. Some final changes on your end to implement the script. I broke the script down into sections for an easier explanation here: Section 1 - you need enter the path to where you want to save your zipped flash copy (local or remote). Section 5 - be sure to enter the number of days you want to keep old flash backups, default is 10 days. Section 6 - optional, to enable notifications, uncomment the bottom line and replace "YOURTOWERNAME" with your actual tower name (this is for notification only) And that's it, all there is to it. Just copy/paste the script into the user scripts plugin, set how often you want it to run, and you're done. If you have any questions or suggestions, let me know. Enjoy. EDIT 1: 12/21/20 - Merged two previous scripts into one script for aforementioned versions EDIT 2: 12/21/20 - Added quotes to $dir in Section 4 that could cause directories with spaces to get lost EDIT 3: 2/5/21 - Step by step instructions found on my youtube channel here Edit 4: 2/8/21 - Inserted new Section 3 with mkdir command to prevent errors if directory does not yet exist, bumped subsequent sections down. No user interaction required - Added quotes to primary directory path to help prevent syntax errors Edited February 8, 2021 by Cpt. Chaz improvements 7 4 Quote Link to comment
falconexe Posted December 17, 2020 Share Posted December 17, 2020 Nice work! I’ll check this out. Quote Link to comment
Cpt. Chaz Posted December 17, 2020 Author Share Posted December 17, 2020 13 hours ago, falconexe said: Nice work! I’ll check this out. Thanks! using unassigned devices, i mounted a folder from my google drive, and now have daily syncs of my flash's stored on google drive, super handy. Quote Link to comment
Mihle Posted February 5, 2021 Share Posted February 5, 2021 Just curious, what do Section 2 do? Quote Link to comment
Cpt. Chaz Posted February 5, 2021 Author Share Posted February 5, 2021 13 minutes ago, Mihle said: Just curious, what do Section 2 do? The native Unraid script creates a symlink in the /usr/local/emhttp/ directory each time it's ran. For the purposes of this script, the symlink is not needed and therefor removed. Quote Link to comment
Mihle Posted February 6, 2021 Share Posted February 6, 2021 (edited) So I did run this and this happened: I think it somehow deleted more things than it should have done EDIT: Reboot fixed it, so I think it caused a memory leak or what its called that made unRAID itself not able to run from memory anymore as it usually do. Had to turn it off by the power button. (I do have a manual flash backup already btw) This is what the script looks like for me right now: #!/bin/bash #### SECTION 1 ####------------------------------------------------------------------------------------------------------ #dir = WHATEVER FOLDER PATH YOU WANT TO SAVE TO dir=/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/ echo 'Executing native unraid backup script' /usr/local/emhttp/webGui/scripts/flash_backup #### SECTION 2 ####------------------------------------------------------------------------------------------------------ echo 'Remove symlink from emhttp' find /usr/local/emhttp/ -maxdepth 1 -name '*flash-backup-*.zip' -delete sleep 5 #### SECTION 3 ####------------------------------------------------------------------------------------------------------ echo 'Move Flash Zip Backup from Root to Backup Destination' mv /*-flash-backup-*.zip "$dir" sleep 5 #### SECTION 4 ####------------------------------------------------------------------------------------------------------ echo 'Deleting Old Backups' #ENTER NUMERIC VALUE OF DAYS AFTER "-MTIME +" find "$dir"* -mtime +90 -exec rm -rfv {} \; echo 'All Done' #### SECTION 5 ####------------------------------------------------------------------------------------------------------ #UNCOMMENT THE NEXT LINE TO ENABLE GUI NOTIFICATION UPON COMPLETION /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Flash Zip Backup" -d "A copy of the NAS unraid flash disk has been backed up" -i "normal" exit When I run it, it did say something about it could not find directory " " over and over and over I think? dont quite remember Did I write something in to the script wrongly? Edited February 6, 2021 by Mihle Quote Link to comment
Mihle Posted February 6, 2021 Share Posted February 6, 2021 (edited) Tested again, but I removed the / at the end of the directory, it says this: Then remove some other things then continues like this, probably until the NAS runs out of memory and crashes? (I run it manually) Is there something I have done that causes it? Edited February 6, 2021 by Mihle Quote Link to comment
Energen Posted February 6, 2021 Share Posted February 6, 2021 Does your backup directory (/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/) already exist? If not, it needs to be created before the script runs. I learned this making my Plex script that you cannot move a file to a destination that does not exist. Also, I'm not clear on whether it's actually needed or not, but it's good practice to enclose your backup path in quotes since it has spaces in the path: dir="/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/" The error message on the mv command indicates that one or both of these are the problem. It's probably also why the rm is failing as well. During the creation of my Plex script I had this rm command mis-formed and had the same errors as you, and the GUI was totally unresponsive due to it.. had to pull the power plug to reset the server since I couldn't restart it any other way... was not a good time! So I'd change your section 1 to this and see if it helps: #dir = WHATEVER FOLDER PATH YOU WANT TO SAVE TO dir="/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/" mkdir -p "$dir" The mkdir -p just ensures that if the path does not exist that it is created. 1 Quote Link to comment
Cpt. Chaz Posted February 6, 2021 Author Share Posted February 6, 2021 (edited) I’d have to agree with @Energen. The spaces in the directory path was the first thing that caught my eye. I think it’s time for a revision with a mkdir section too Edited February 6, 2021 by Cpt. Chaz Quote Link to comment
Mihle Posted February 6, 2021 Share Posted February 6, 2021 (edited) 1 hour ago, Energen said: Does your backup directory (/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/) already exist? Yes. Quote If not, it needs to be created before the script runs. I learned this making my Plex script that you cannot move a file to a destination that does not exist. Also, I'm not clear on whether it's actually needed or not, but it's good practice to enclose your backup path in quotes since it has spaces in the path: dir="/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/" I did not know that Quote The error message on the mv command indicates that one or both of these are the problem. It's probably also why the rm is failing as well. During the creation of my Plex script I had this rm command mis-formed and had the same errors as you, and the GUI was totally unresponsive due to it.. had to pull the power plug to reset the server since I couldn't restart it any other way... was not a good time! So I'd change your section 1 to this and see if it helps: #dir = WHATEVER FOLDER PATH YOU WANT TO SAVE TO dir="/mnt/user/NCloud/Mihle/files/Filer/Backup/NAS Flash Backup/" mkdir -p "$dir" The mkdir -p just ensures that if the path does not exist that it is created. Ah, ok, only adding the quotes seem to have fixed it! Well, It does not show up in Nextcloud as I ideally want, but that might be Nextcloud. At least it doesnt make the folder it is moved to uneditable for Nextcloud as the appdata backup do. Edited February 6, 2021 by Mihle Quote Link to comment
Mihle Posted February 6, 2021 Share Posted February 6, 2021 (edited) 19 hours ago, Mihle said: Well, It does not show up in Nextcloud as I ideally want, but that might be Nextcloud. At least it doesnt make the folder it is moved to uneditable and unsyncable for Nextcloud as the appdata backup plugin do. Probably Nextcloud fault, maybe it dont like putting files to its database without being through its GUI or its own code. Work around of having a own share just to store the backup and adding it as external storage worked, it then shows up in Nextcloud and syncs to other devices just fine. Not ideal but it works. Thanks. Edited February 7, 2021 by Mihle Quote Link to comment
Cpt. Chaz Posted February 8, 2021 Author Share Posted February 8, 2021 On 2/6/2021 at 8:18 AM, Cpt. Chaz said: I’d have to agree with @Energen. The spaces in the directory path was the first thing that caught my eye. I think it’s time for a revision with a mkdir section too done 1 Quote Link to comment
wgstarks Posted December 2, 2021 Share Posted December 2, 2021 Love this script. Working great for me. I'm backing up to another computer via SMB (UD mount). My only question is what happens if the network destination isn't mounted when the script runs? Quote Link to comment
Cpt. Chaz Posted December 3, 2021 Author Share Posted December 3, 2021 (edited) Thanks! That’s a good question… might be worth adding a check for that… I’ve got a few irons in the fire at the moment, but I’ll try to write something up for it over the coming weekend if schedule permits. Anyone is free to take a stab at it as well and I’ll add it. Edited December 3, 2021 by Cpt. Chaz Quote Link to comment
wgstarks Posted December 3, 2021 Share Posted December 3, 2021 Yeah. I know the UD plugin has warnings about SMB mounts failing. Quote Link to comment
L0rdRaiden Posted February 1, 2022 Share Posted February 1, 2022 (edited) Does someone has a version of this script that just copy the files without making a zip file? The next time the backup is done is ok if it deletes everything in that folder and put the new files. I'm doing backups with Kopia (highly recommended after trying many) and this would be wonderful to have a decent versioning without taking almost any additional space. With Kopia docker I don't have permissions to access to the boot folder so this is the only solution so far For example for each snapshot You can browse it individually and get any file, restore everything... it has encryption deduplication, compression, and it is way faster than duplicati. Edited February 1, 2022 by L0rdRaiden Quote Link to comment
wgstarks Posted February 1, 2022 Share Posted February 1, 2022 3 minutes ago, L0rdRaiden said: Does someone has a version of this script that just copy the files without making a zip file? The next time the backup is done is ok if it deletes everything in that folder and put the new files. I'm doing backups with Kopia (highly recommended after trying many) and this would be wonderful to have a decent versioning without taking almost any additional space. With Kopia docker I don't have permissions to access to the boot folder so this is the only solution so far The CA Backups app will do uncompressed backups to the location of your choice IIRC. Quote Link to comment
L0rdRaiden Posted February 1, 2022 Share Posted February 1, 2022 (edited) 9 minutes ago, wgstarks said: The CA Backups app will do uncompressed backups to the location of your choice IIRC. I know the problem with CA backup is that it can not only do flash backup, you have to do it together with appdata (docker) although I guess I could you a dummy folder and In any case if I backup the USB will CA remove all the files a backup the current status or will only replace o create files but won't delete older ones? Edited February 1, 2022 by L0rdRaiden Quote Link to comment
Cpt. Chaz Posted February 2, 2022 Author Share Posted February 2, 2022 6 hours ago, L0rdRaiden said: Does someone has a version of this script that just copy the files without making a zip file? The next time the backup is done is ok if it deletes everything in that folder and put the new files. This script is built on the native unraid script that includes the zip function. However, it would be easy enough to add an unzip function to my backup script for you to use. I wrote this fairly quickly and only tested once to make sure it worked. Looks like it did, but please use caution until you've tested and are certain. Also, this assumes you only have the one single zip file in $dir. the way it's written, it will unzip every zip file in $dir. Just copy and paste to the end of your script. Hope it helps. #### SECTION 7 ####------------------------------------------------------------------------------------------------------ echo 'Unzip Flash Backup' cd "$dir" unzip *.zip #UNCOMMENT THE NEXT LINE IF YOU WISH TO DELETE THE ORIGINAL ZIP FILE AFTER EXTRACTION #rm -rfv *.zip 1 Quote Link to comment
L0rdRaiden Posted February 2, 2022 Share Posted February 2, 2022 4 hours ago, Cpt. Chaz said: This script is built on the native unraid script that includes the zip function. However, it would be easy enough to add an unzip function to my backup script for you to use. I wrote this fairly quickly and only tested once to make sure it worked. Looks like it did, but please use caution until you've tested and are certain. Also, this assumes you only have the one single zip file in $dir. the way it's written, it will unzip every zip file in $dir. Just copy and paste to the end of your script. Hope it helps. #### SECTION 7 ####------------------------------------------------------------------------------------------------------ echo 'Unzip Flash Backup' cd "$dir" unzip *.zip #UNCOMMENT THE NEXT LINE IF YOU WISH TO DELETE THE ORIGINAL ZIP FILE AFTER EXTRACTION #rm -rfv *.zip Thanks a lot, I would also need to delete all the content of the folder before the unzip, so the old files won't appear anymore and the unzip or the copy of the file starts always in a clean folder Quote Link to comment
Cpt. Chaz Posted February 2, 2022 Author Share Posted February 2, 2022 No problem. Does section 5 not do this for you? 1 Quote Link to comment
L0rdRaiden Posted February 2, 2022 Share Posted February 2, 2022 (edited) 59 minutes ago, Cpt. Chaz said: No problem. Does section 5 not do this for you? Yes you are right, so I used my advanced skills in scripting programming 😁 and swapped section 4 and 5, so the files are deleted (mtime +0) and then the zip file is moved and unzipped. I won't delete the original zip (section 6) file but I will exclude that file in the backup system. Now looks perfect for my use case, thanks a lot (EDIT: see next posts, still doesn't work) #!/bin/bash #### SECTION 1 ####------------------------------------------------------------------------------------------------------ #dir = WHATEVER FOLDER PATH YOU WANT TO SAVE TO dir="/mnt/user/Backup/UnraidFlash/" echo 'Executing native unraid backup script' /usr/local/emhttp/webGui/scripts/flash_backup #### SECTION 2 ####------------------------------------------------------------------------------------------------------ echo 'Remove symlink from emhttp' find /usr/local/emhttp/ -maxdepth 1 -name '*flash-backup-*.zip' -delete sleep 5 #### SECTION 3 ####------------------------------------------------------------------------------------------------------ if [ ! -d "$dir" ] ; then echo "making directory as it does not yet exist" # make the directory as it doesnt exist mkdir -vp "$dir" else echo "As $dir exists continuing." fi #### SECTION 4 ####------------------------------------------------------------------------------------------------------ echo 'Deleting Old Backups' #ENTER NUMERIC VALUE OF DAYS AFTER "-MTIME +" find "$dir"* -mtime +0 -exec rm -rfv {} \; echo 'All Done' sleep 5 #### SECTION 5 ####------------------------------------------------------------------------------------------------------ echo 'Move Flash Zip Backup from Root to Backup Destination' mv /*-flash-backup-*.zip "$dir" sleep 5 #### SECTION 6 ####------------------------------------------------------------------------------------------------------ echo 'Unzip Flash Backup' cd "$dir" unzip *.zip #UNCOMMENT THE NEXT LINE IF YOU WISH TO DELETE THE ORIGINAL ZIP FILE AFTER EXTRACTION #rm -rfv *.zip #### SECTION 7 ####------------------------------------------------------------------------------------------------------ #UNCOMMENT THE NEXT LINE TO ENABLE GUI NOTIFICATION UPON COMPLETION /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Flash Zip Backup" -d "A copy of the Unraid flash disk has been backed up" -i "normal" exit Edited February 2, 2022 by L0rdRaiden Quote Link to comment
L0rdRaiden Posted February 2, 2022 Share Posted February 2, 2022 (edited) @Cpt. Chaz well I am not as good as I though For some reason Section 4 for doesn't delete previous zip files but deletes uncompressed files Section 6 fails to unzip the file because I guess there are more than 1 file that matches, right? If we can fix section 4 I guess section 6 will work, it should be easy but I am afraid to experiment with the rm command. Maybe replacing section 4 with this will fix everything? cd "$dir" rm -rfv echo 'All files deleted' Edited February 2, 2022 by L0rdRaiden Quote Link to comment
Cpt. Chaz Posted February 2, 2022 Author Share Posted February 2, 2022 you're on the right track! however the rm syntax requires a target. try this instead rm -rfv "$dir"* echo 'All files deleted' if you want to confirm this works before trying on "mission critical" files, take everything out of $dir. Then cd into the directory, and run this touch test1.txt test2.txt mkdir testfolder this just simply puts in a couple basic files and a directory. If the commands i gave you for section 4 work correctly, it should remove these test files, and the rest of the flow should result in the latest unzipped backup copy of your flash drive. 1 Quote Link to comment
L0rdRaiden Posted February 2, 2022 Share Posted February 2, 2022 46 minutes ago, Cpt. Chaz said: you're on the right track! however the rm syntax requires a target. try this instead rm -rfv "$dir"* echo 'All files deleted' if you want to confirm this works before trying on "mission critical" files, take everything out of $dir. Then cd into the directory, and run this touch test1.txt test2.txt mkdir testfolder this just simply puts in a couple basic files and a directory. If the commands i gave you for section 4 work correctly, it should remove these test files, and the rest of the flow should result in the latest unzipped backup copy of your flash drive. Thanks, it works perfectly I have learnt a lot with you Together with this Script (the first one in my life) and Kopia I have tune up my backup system #!/bin/bash /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "OneDrive backup started" -d "Synchronizing Kopia backup to OneDrive" -i "normal" echo 'OneDrive backup started' rclone sync /mnt/user/Backup/Kopia OneDrive:"/Backup/Kopia" /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "OneDrive backup finished" -d "Synchronization completed" -i "normal" echo 'One Drive backup completed' 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.