bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 Thank you. I also thought about the hardlink limit. However, it's at 1023 and I only made it to 20 backups. The use case should be that someone who knows very little about computers can use the disk (family photos) without my help. That's why I didn't want to use a unix file system. Since I don't know where the person will use the disk, I also didn't want to use a system that only works with additional software under Windows. What else could depress the read/write speed so much that the disk sort of stands still? Could another standard job that is started in parallel cause the problem? Certain files should not be it, right? Is there any way I can turn on a debug mode? many greetings bonbonJaeger Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 1 hour ago, bonbonJaeger said: --compress This option is useless as you are copying locally. 1 hour ago, bonbonJaeger said: Does anyone have any idea what is causing the script to crash? What is written to the logs? How do you know that the script is crashed? Did you try to kill rsync by "pkill -xc rsync"? 1 hour ago, bonbonJaeger said: If I then force the system to reboot with "reboot", the parity check is started every time. I had a similar problem in the past. It seems Unraid is not able to restart if a rsync process is still running. Because of that I created a stop file with the following content: Unraid executes the stop file automatically if the system reboots / is shutdown. Quote Link to comment
bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 Thanks. 1 hour ago, mgutt said: 2 hours ago, bonbonJaeger said: --compress This option is useless as you are copying locally. I have removed the switch. 1 hour ago, mgutt said: 2 hours ago, bonbonJaeger said: Does anyone have any idea what is causing the script to crash? What is written to the logs? How do you know that the script is crashed? Did you try to kill rsync by "pkill -xc rsync"? I thought it crashed because nothing happened for hours. I tried to kill the processes with kill -9 1 hour ago, mgutt said: I had a similar problem in the past. It seems Unraid is not able to restart if a rsync process is still running. Because of that I created a stop file with the following content: I have created the script. Do I need to make it executable? If so how? A chmod +x shows no effect. I will run the script again and then post the logs here. Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 1 hour ago, bonbonJaeger said: I have created the script. Do I need to make it executable? Don't you use the user scripts plugin? Quote Link to comment
bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 14 minutes ago, mgutt said: 1 hour ago, bonbonJaeger said: I have created the script. Do I need to make it executable? Don't you use the user scripts plugin? Your screenshot showed the config editor and a different path than the one for the user scripts. That's why I didn't get the idea. Should I create the script via the UserScripts and then copy it to /boot/config? Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 7 minutes ago, bonbonJaeger said: Should I create the script via the UserScripts and then copy it to /boot/config? The Plugin has its own GUI, no need to create script files: Quote Link to comment
bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 Yes, thanks, I use the user scripts. But the screenshot of your stop script pointed to /boot/config. What do I have to do to run the user script on reboot? Just copy it to /boot/config? Or do I need to run it manually before the reboot? Quote Link to comment
bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 it happens again. The syslog shows: The unassigned devices - 0 free and 0 used space - no read or write speed: I had a window open with tail to view the output of the rsync script. Is there anything else I can check? I would keep running this now to see if it recovers. Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 20 minutes ago, bonbonJaeger said: it happens again While the backup is running? Should be "normal". As I said. Its a timeout of the UD Plugin and not an error. 21 minutes ago, bonbonJaeger said: I had a window open with tail to view the output of the rsync script. Does it still copy files or is it frozen? Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 36 minutes ago, bonbonJaeger said: But the screenshot of your stop script pointed to /boot/config. Ah ok. Simply create the file: touch /boot/config/stop Should be sufficient. It gets the same chmod as the other files, so it should work: For Copy & Paste: #!/bin/bash # ------------------------------------------------- # Kill all rsync processes # ------------------------------------------------- pkill -x rsync Quote Link to comment
bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 2 hours ago, mgutt said: While the backup is running? Should be "normal". As I said. Its a timeout of the UD Plugin and not an error. Does it still copy files or is it frozen? I can't tell if it's copying files. Because in the main window for the disk everything is set to 0. The rsync-log has no new entries. But the syslog has many error messages. Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 27 minutes ago, bonbonJaeger said: But the syslog has many error messages. Yes, this is normal as you access the WebGUI which trys to reload the unassigned devices usage every 10 seconds or so (which fails because of the timeout). 27 minutes ago, bonbonJaeger said: The rsync-log has no new entries. Ok, so it's dead. At first you should kill rsync: (x = exact match, c = count killed processes) pkill -xc rsync Then add "--timeout=10" to your rsync command. Rsync does never timeout without this flag. But finally it does not solve your problem. Maybe you should try to copy the folder manually with the "cp" command. And I would try to start a diagnosis with a vendor tool. As an example if it's an WD drive, the Data Lifeguard Diagnostic Tool. Quote Link to comment
bonbonJaeger Posted August 6, 2021 Share Posted August 6, 2021 1 hour ago, mgutt said: At first you should kill rsync: (x = exact match, c = count killed processes) pkill -xc rsync that doesn't work. I will reboot. Unfortunately, the reboot does not work because the disk does not respond. I now have to force the restart. But this also means a new parity check. Tomorrow I will check the disk with the diagnostic tool. Quote Link to comment
mgutt Posted August 6, 2021 Author Share Posted August 6, 2021 2 hours ago, bonbonJaeger said: that doesn't work. I will reboot Then the target disk is completely blocking rsync. Maybe because of the massive amount of sub shells calling "df"... I will ask the dev of UD what he thinks about that. Instead of rebooting: Wouldn't it be possible to simply disconnect the drive? Quote Link to comment
bonbonJaeger Posted August 7, 2021 Share Posted August 7, 2021 So far, removing the disk from unraid has not been detected. The same error messages were always displayed. I have downloaded the dashboard from WD. Unfortunately, only a SMART test is offered here. I could have also started in unraid. By the way, the dashboard shows that everything is good with the disk. I remembered that backups via a virtual machine with the software HardlinkBackup runs without errors. Could there be a problem with the USB port? For now, the advanced SMART test is running. Quote Link to comment
bonbonJaeger Posted August 8, 2021 Share Posted August 8, 2021 According to SMART, the disk is fine. So the mystery is not solved yet. Quote Link to comment
ullibelgie Posted August 16, 2021 Share Posted August 16, 2021 On 3/17/2021 at 8:22 AM, mgutt said: Add a new local user through the settings with a strong password and name it "backup". After that right mouse click on "documents" > share with... Select the new user from the drop-down. It should be "read only". Repeat this step for all dirs you want to share. Do not share the root username dir. It contains the hidden "appdata" dir and a massive amount of other files which aren't needed. After that you can mount this "smb server" through the Unassigend Disks Plugin. Hello - I want to follow this concept, but from an Ubuntu client (instead of Windows) However how do I do this ? Create additional user... should be ok created additional samba-user - should be also ok However finally I cannot mount the ubuntu ext4 disk in unassigned devices plug in It says permission denied - does it NEED write access ? Because when I grant full write access there is no problem.... Quote Link to comment
Shantarius Posted August 19, 2021 Share Posted August 19, 2021 Hi, if i execute the script i become the folloing error: root@Avalon:/boot/config/scripts# sh backup_rsync.sh backup_rsync.sh: line 74: syntax error near unexpected token `>' backup_rsync.sh: line 74: ` exec &> >(tee "${backup_path}/logs/${new_backup}.log")' root@Avalon:/boot/config/scripts# How can i solve this? Thank you! 1 Quote Link to comment
Braulio Dias Ribeiro Posted September 2, 2021 Share Posted September 2, 2021 @mguttplease, can you help me? I'm using your script on Unraid 6.9.2, but it's copying all folders again. The backup size always increases. I just modified the path: source_paths=( "/mnt/user/Test1" ) backup_path="/mnt/disks/Extern" Quote Link to comment
mgutt Posted September 2, 2021 Author Share Posted September 2, 2021 1 hour ago, Braulio said: The backup size always increases. Each backup is a full backup which uses hardlinks to already existing files in previous backups. So if you create an archive from one backup, it can not take advantage of the hardlinks. Instead it copies all files. The only solution is to create archives of multiple backups like one archive per week. But the archive type must support hardlinks preserving. Rar supports it for NTFS: https://documentation.help/WinRAR/HELPSwOH.htm But I do not suggest to use NTFS. NTFS has an extremely bad hardlink performance and it is not possible to backup all files that could be present on a Linux disk. For example you can't save files with a question mark in the filename and many others which are supported in Linux. Use ext4 or BTRFS. There is a free windows driver for BTRFS available. Quote Link to comment
Braulio Dias Ribeiro Posted September 2, 2021 Share Posted September 2, 2021 3 hours ago, mgutt said: Each backup is a full backup which uses hardlinks to already existing files in previous backups. So if you create an archive from one backup, it can not take advantage of the hardlinks. Instead it copies all files. The only solution is to create archives of multiple backups like one archive per week. But the archive type must support hardlinks preserving. Rar supports it for NTFS: https://documentation.help/WinRAR/HELPSwOH.htm But I do not suggest to use NTFS. NTFS has an extremely bad hardlink performance and it is not possible to backup all files that could be present on a Linux disk. For example you can't save files with a question mark in the filename and many others which are supported in Linux. Use ext4 or BTRFS. There is a free windows driver for BTRFS available. I thought I had only one folder with all the files (incremental) and each new folder (backup) pointed to the file location. Thanks for the explanations. Your script turned out really good. Congratulations Quote Link to comment
FFrank Posted September 8, 2021 Share Posted September 8, 2021 (edited) Hi, I am new to UnRaid. But not totally to Linux - I had a Open Media Vault Box since last week and changed the setup to UNRAID. All is working great an I am very pleased. Now I need a backup solution and found this script. This seems to be great. Sadly I do not find a explanation how to set it up...I read the whole thread but did not find anything...so please point me in the right direction. Thanks. Best. F. Edited September 8, 2021 by FFrank Quote Link to comment
FFrank Posted September 8, 2021 Share Posted September 8, 2021 OK, I tried to get it done...But I failed 😞 I am not sure what triggers the script. Is it only on schedule or on connection of the drive? You mention both in the first post: ******* Use the User Scripts Plugin (Unraid Apps) to execute it by schedule Use the Unassigned Devices Plugin (Unraid Apps) to execute it after mounting a USB drive ******* I found two places to enter the script. One in the Unassigned Devices Plugin and one in the User Scripts Plugin? How/where is it supposed to be? It would be perfect if I could just plug in the drive and after getting a message(mail) that the backup is done I just unplug the drive. I really want to learn more - so just point me to the right place where I find the info. I will investigate further - hope I will manage later the day.... Thanks. F. Quote Link to comment
trurl Posted September 8, 2021 Share Posted September 8, 2021 48 minutes ago, FFrank said: just plug in the drive Then you want to use an Unassigned Devices script. Take a look at the UD support thread, there are examples in the first page. You can go directly to the support thread for any of your plugins by clicking its Support Link on the Plugins page. Quote Link to comment
FFrank Posted September 8, 2021 Share Posted September 8, 2021 7 minutes ago, trurl said: Then you want to use an Unassigned Devices script. Take a look at the UD support thread, there are examples in the first page. You can go directly to the support thread for any of your plugins by clicking its Support Link on the Plugins page. Great, thanks for the information. But still it would be nice to know how this plugin is supposed to work. Is the drive attached the whole time? Is it always mounted? How do I have to set it up? Thanks so much for more information. It seams to be a great way to backup. But I want to understand it before fiddling arround Best. F. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.