tonsofguns Posted February 16, 2021 Share Posted February 16, 2021 (edited) I just lost 7 days of progress on a self hosted game server and this will be a life saver. I do have 2 questions. 1. I created a share for the backup itself. The source points to my server folder, but then in the backup folder is has the entire path instead of just the single folder (/mnt/user/appdata/valheim/.config/unity3d/IronGate/Valheim/worlds). So every folder is backed up in the specified end folder. Is this normal or did I do something wrong? If it's normal then no big deal since there are no additional files. Just a bunch of folders. Was just making sure I had it right. 2. If I change the dropdown on UserScript plugin to schedule daily, is that all I need to do or is there another section in the script to change? Edited February 16, 2021 by tonsofguns Quote Link to comment
mgutt Posted February 16, 2021 Author Share Posted February 16, 2021 3 hours ago, tonsofguns said: Is this normal or did I do something wrong? Yes it's normal. If I would use only the last dir "valheim", a second backup of the path "/mnt/user/valheim" would write in the same folder. But usually my script replaces /mnt/user against /Shares. Does it not happen to you? Please post your set source paths 1:1. Maybe a whitespace in front of the slash? Quote is that all I need Yes 1 Quote Link to comment
tonsofguns Posted February 16, 2021 Share Posted February 16, 2021 6 hours ago, mgutt said: Yes it's normal. If I would use only the last dir "valheim", a second backup of the path "/mnt/user/valheim" would write in the same folder. But usually my script replaces /mnt/user against /Shares. Does it not happen to you? Please post your set source paths 1:1. Maybe a whitespace in front of the slash? source_paths=( "/mnt/user/appdata/valheim/.config/unity3d/IronGate/Valheim/worlds" ) backup_path="/mnt/user/valheimbackup" Quote Link to comment
cgram23 Posted February 17, 2021 Share Posted February 17, 2021 On 11/5/2020 at 9:43 AM, mgutt said: Nothing wrong with that. The script should work with all paths (and not only "shares"). This would cause a huge problem if a user backups different paths with the same last subdir name. Example: /mnt/user/Moritz/Notizen /mnt/user/Max/Notizen Both would target "/backup/Notizen". I know its "ugly" having super long paths, but how could we solve this? Maybe an optional setting like "force last subdir name"? The permission problem is not related to rsync. It's only related to the "mv" command. The "mv" renames (or "moves") the backup folder from the hidden ".20201105" to "20201105" if the backup was successful: mv "${backup_path}/.${new_backup}" "${backup_path}/${new_backup}" It's a really basic linux command so I wonder why it does not work for you. How did you mount "/mnt/hawi/192.168.178.101_disk17/Backup" and what could be the reason why read & write is allowed, but not renaming? Please manually repeat the command through the WebTerminal: mv '/mnt/hawi/192.168.178.101_disk17/Backup/disk17/Notizen/.20201105_130528' '/mnt/hawi/192.168.178.101_disk17/Backup/disk17/Notizen/20201105_130528' If it works now, then something in this path locked the directory or a file inside of it. Maybe an index service on the external location or similar? First, thanks for creating this. It's exactly what I've been trying to accomplish. With regards to the mv permissions issue, it seems the log file is locking the .hidden directory: # lsof +D .20210216_191026/ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tee 10992 root 4w REG 0,152 42261 1376475 .20210216_191026/backup.log Any idea how to prevent this? Quote Link to comment
bclinton Posted February 17, 2021 Share Posted February 17, 2021 On 2/15/2021 at 4:48 PM, mgutt said: Yes, but the next backup isn't incremental. Thank you for the help. The next disk I will use I will format it in BTRFS. Thanks again for the great script! Quote Link to comment
AndreasNe Posted March 8, 2021 Share Posted March 8, 2021 Hi, how can i run the Backup to a external Share like a Diskstation? I have a SMB Share for it, but i run in some problems 🙂 Script location: /tmp/user.scripts/tmpScripts/Unraid Backup/script Note that closing this window will abort the execution of this script Create backup of /mnt/user/Musik/Mischmasch/Led Backup path has been set to /mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led Create full backup 20210308_172541 sending incremental file list Led/ Led/Controller-1/ Led/Controller-1/00_program1.led Number of files: 3 (reg: 1, dir: 2) Number of created files: 3 (reg: 1, dir: 2) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 455,680 bytes Total transferred file size: 455,680 bytes Literal data: 455,680 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 455,971 Total bytes received: 47 sent 455,971 bytes received 47 bytes 912,036.00 bytes/sec total size is 455,680 speedup is 1.00 Try #1 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #2 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #3 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #4 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #5 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #6 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #7 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #8 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #9 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #10 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #11 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #12 to make backup visible mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied Try #13 to make backup visible Quote Link to comment
mgutt Posted March 9, 2021 Author Share Posted March 9, 2021 12 hours ago, AndreasNe said: Try #13 to make backup visible Was this the last try and finally successful or did it finally return a "Backup failed" error? If it returned an error, are you able to execute this command successful through the WebTerminal? mv "/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541" "/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541" 1 Quote Link to comment
AndreasNe Posted March 9, 2021 Share Posted March 9, 2021 9 minutes ago, mgutt said: 12 hours ago, AndreasNe said: Try #13 to make backup visible Was this the last try and finally successful or did it finally return a "Backup failed" error? I have cut the LOG here. At the End it returned "Backup failed" error. I will try the command today. Thank you. Quote Link to comment
toasti Posted March 14, 2021 Share Posted March 14, 2021 It is possible that the script had some problems with big initial backups? I started yesterday to backup my 4 TB of Data to an external drive and it stopped after 1,5 TB. Now I started again and I am excited to see what will happen. But very cool script, really! I had a look how the hard links and inodes work, very clever! Quote Link to comment
mgutt Posted March 14, 2021 Author Share Posted March 14, 2021 5 minutes ago, toasti said: it stopped after 1,5 TB And why? Logs? Quote Link to comment
toasti Posted March 14, 2021 Share Posted March 14, 2021 The log just stopped - no errors. Quote Link to comment
mgutt Posted March 14, 2021 Author Share Posted March 14, 2021 Then the script itself was killed. If only resync is killed, the final cleanup of the script still occurs. This is not a normal behaviour. Server restart or execution in a terminal which was closed can be excluded? Quote Link to comment
toasti Posted March 14, 2021 Share Posted March 14, 2021 I presume the problem for just a big job was that I ran the script manually in the web gui user script plugin. At the moment it's still running. When done I will schedule an automatic job. Quote Link to comment
mgutt Posted March 14, 2021 Author Share Posted March 14, 2021 Yes, you need to execute it "in background" or it will be killed by closing the window. Quote Link to comment
toasti Posted March 14, 2021 Share Posted March 14, 2021 The Window was open, anytime. But anyway, I presume this is the problem. Quote Link to comment
toasti Posted March 14, 2021 Share Posted March 14, 2021 By the way... is there any chance to use this script locally on a Windows PC to backup the internal disks to an external drive? Quote Link to comment
mgutt Posted March 14, 2021 Author Share Posted March 14, 2021 2 hours ago, toasti said: is there any chance to use this script locally on a Windows PC Nope. This script is written in Bash/Shell = Linux. I add in my Windows Clients a new user names "Unraid" and by right-clicking on a folder I share this folder to this user read-only. Now I mount my Windows Client through Unassigned Disks and use my script to backup this Windows Client Quote Link to comment
toasti Posted March 14, 2021 Share Posted March 14, 2021 (edited) Good idea! 🙂 But how to automate / trigger this from the Windows Client to make backup now? It must be so easy that my wife can do this. Maybe it's possible to use the "inbuilt" Ubuntu in W10 and mount the local disks and run it from there? https://thedatafrog.com/en/articles/backup-rsync-windows-wsl/ Edited March 14, 2021 by toasti Quote Link to comment
mgutt Posted March 15, 2021 Author Share Posted March 15, 2021 @hawihoney @cgram23 @AndreasNe I think I found the problem with the renaming. Because I forward the terminal output to a log file which is in the same subdirectory as the backup directory, it is not possible to rename the backup directory, because the log file is still open for writing. I solved this in version 0.5. Quote Link to comment
mgutt Posted March 15, 2021 Author Share Posted March 15, 2021 Version 0.4 released # - timeout for backup renaming removed (as it did not solve the renaming problem) # - changed log file location to solve renaming problem # - skip permission errors if permission_error_treshold is under 20% (default) # - optional: skip "host is down" error # - optional: skip "wrong source path" error Hopefully moving the logs out of the backup directory into a separate "logs" directory will solve the renaming problem. In addition some soft errors can be skipped: skip permission errors The backup is valid if 20% or less of the files are not copied because of permission errors. Can be changed through the permission_error_treshold setting. skip "Host is down (112)" error The backup is invalid, but does not return an error, if rsync returns this message. This happens if the source direcory is an SMB server which is mounted, but offline. skip "No such file or directory (2)" error The backup is invalid, but does not return an error, if rsync returns this message. This happens if the source direcory is an SMB server which is offline (or if the source path is simply wrong). Quote Link to comment
mgutt Posted March 15, 2021 Author Share Posted March 15, 2021 Version 0.5 released # - preserve logs directory on clean up # - ignore logs directory on last backup check Sorry, v0.4 compared the new backup with the logs directory instead of the last backup, so the new backup was a full backup instead of incremental. Quote Link to comment
mgutt Posted March 15, 2021 Author Share Posted March 15, 2021 10 hours ago, toasti said: But how to automate / trigger this from the Windows Client to make backup now? There are many ways to realize this. But at first the explanation, why I don't create backups through my client. My server has a share "backups" and no client is able to write into this folder. This is important because: a) No access means no access for Ransomware b) If a client changes the file backups/20190101/document.txt, which is old, it will overwrite this file in all backups, because rsync used hardlinks to save storage space and hardlinks are only different file names for the same file. So it's more secure to execute the script on the server to fetch the files from the client (read-only) and not pushing them to the server. Now the question how to start the script through the client: a) Schedule One idea would be to execute it every x hours, which creates multiple backups per day. As they are incremental, they won't consume much space. To start the script directly or for example 10 minutes after the client was powered on, we could add a "ping client check" before executing the script. This means the script starts every x hours and waits for the client as long as needed. As my script is atomic, it won't re-execute as long its waiting. This method does not need any interaction by the client. This is how I solved it. b) Ping / send request to unraid server A script could monitor incoming traffic on port X and if it was found it starts the backup script. The traffic could be created by a simple startbackup.bat file which will be placed on the client's desktop. c) Create file or update file on client The script permanently checks if a file on the client is updated / created and if, it starts the backup. Quote Link to comment
mgutt Posted March 15, 2021 Author Share Posted March 15, 2021 Version 0.6 released # - File count check repaired Dumb bug. Missed one exclamation mark in a check. Quote Link to comment
toasti Posted March 15, 2021 Share Posted March 15, 2021 Aahhhh, changed 5 min ago and startet backup On 3/14/2021 at 10:42 PM, mgutt said: Now I mount my Windows Client through Unassigned Disks and use my script to backup this Windows Client That works Thanks for letting me know the possible ways to trigger a backup. The question is what would be the easiest and fastest to implement? Quote Link to comment
mgutt Posted March 16, 2021 Author Share Posted March 16, 2021 18 hours ago, toasti said: The question is what would be the easiest and fastest to implement? Executing every 3 hours, is the easiest solution: Another solution is to search for this line in my script: if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT; And add this after it: if tcpdump -i eth0 -c1 port 8133; then echo "Traffic detected!" fi Now set a custom cron to execute the script every 5 minutes: */5 * * * * Now the script is executed every 5 minutes, but only one session stays open and monitors traffic on port 8133 (you can change this port of course). Now you only need to open the link "http://tower:8133" in your browser, which returns an error, because the page does not exist, but it will execute the backup script as it detected this connection attempt: Opening a website, which does not return anything is strange? Ok, then create a startbackup.bat file on your windows desktop and use a text editor with the following content: curl http://tower:8133 Screenshot: Double-click the file and it tries to open this link. After some seconds the window automatically closes. The script on the server detected the traffic and starts the backup. And there is even more crazy stuff possible, like sending an answer ^^ By that it would be even possible to output the rsync logs to the client, but this would need some extra work. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.