Squid Posted February 26, 2017 Author Share Posted February 26, 2017 Awesome plug man, has saved me quite a few times from myself accidentally deleting stuff from appdata. Couple of questions: is stopping dockers really necessary? Never seen the 'file open error' when copying appdata manually; how did you find out which files need to be backed up for vms? (it's not just the xml templates); as an extension to 1st - what logic is applied on the docker restart? Every time backup job finishes, it also starts dnsmasq after starting the dockers (and other services?), so the dns docker fails to start up since port 53 is occupied (my issue thread) You have the option to keep any docker running during the backup. Your mileage may vary. If errors result by doing so the plugin will report it.The files for the VMs are the xml templates and by request the nvme romsDockers that the plugin stops at the start are the ones which are restarted when it finishesSent from my LG-D852 using Tapatalk Quote Link to comment
tuxbass Posted February 26, 2017 Share Posted February 26, 2017 (edited) So there's no apparent reason why dnsmasq is started? Note it doesn't happen when the physical box is rebooted. Edited February 26, 2017 by tuxbass Quote Link to comment
Squid Posted February 26, 2017 Author Share Posted February 26, 2017 So there's no apparent reason why dnsmasq is started? Note it doesn't happen when the physical box is rebooted.If it's showing as running when the backup starts then I stop it and then restart it. If it's not showing as running when the backup starts but the plugin restarts it then I'd have to give you a script to help me identify what's going onSent from my LG-D852 using Tapatalk Quote Link to comment
tuxbass Posted February 26, 2017 Share Posted February 26, 2017 11 hours ago, Squid said: If it's showing as running when the backup starts then I stop it and then restart it. If it's not showing as running when the backup starts but the plugin restarts it then I'd have to give you a script to help me identify what's going on Sent from my LG-D852 using Tapatalk Hmm, just started the job manually and it didn't reproduce; yet every Monday (when it's scheduled to run) the dns fails to start since dnsmasq had been started. Might have jumped to a conclusion here. So if the backup app detects that dnsmasq process is running, it'll restart it, and only then? Quote Link to comment
Squid Posted February 26, 2017 Author Share Posted February 26, 2017 Hmm, just started the job manually and it didn't reproduce; yet every Monday (when it's scheduled to run) the dns fails to start since dnsmasq had been started. Might have jumped to a conclusion here. So if the backup app detects that dnsmasq process is running, it'll restart it, and only then?Correct. Sent from my LG-D852 using Tapatalk Quote Link to comment
DazedAndConfused Posted March 16, 2017 Share Posted March 16, 2017 Whenever I do USB Backup, it starts copying files to my rclone disk and then the plugin just appears to stop doing anything. Is there a way to see a verbose output of what its currently doing? All my log says right now is "Backing up USB Drive". I started this at 7:38 and no other files have been changed since around then. Quote Link to comment
Squid Posted March 16, 2017 Author Share Posted March 16, 2017 Not sure... This is what the USB backup actually is: mkdir -p $destination # makes the path if it doesn't exist ...Creates a file called DISK_ASSIGNMENTS.txt in /boot/config cp /boot/* $destination -R -v # the -v is superfluous as the verbose output is actually ignored mv $destination/config/super.dat $destination/config/super.dat.CA_BACKUP very simple. Something to do with rclone, but I can't help diagnose that... Easy test is to set a separate destination for USB to the array. If that works its a given that its an rclone issue Quote Link to comment
DazedAndConfused Posted March 16, 2017 Share Posted March 16, 2017 What about the "Error in File IO (code 11)" that I get when just copying the Appdata folder. It happens on linuxserverio dockers as far as I can see. Quote Link to comment
Squid Posted March 16, 2017 Author Share Posted March 16, 2017 47 minutes ago, DazedAndConfused said: What about the "Error in File IO (code 11)" that I get when just copying the Appdata folder. It happens on linuxserverio dockers as far as I can see. Its an rsync error. Is it also going to an rclone destination? If so, then best guess rclone does not support symlinks and/or hardlinks, in which case it is not suitable for a destination since most apps utilize them. (Plex as an example makes HUGE use of them). Quote Link to comment
DazedAndConfused Posted March 16, 2017 Share Posted March 16, 2017 That may be the case, I'll try a regular unassigned disk. Quote Link to comment
kizer Posted March 16, 2017 Share Posted March 16, 2017 This totally saved my Bacon. Dropped in a new Drive and did a rebuild to a larger Data disk, and the Plex Plugin started and and later I started my my Plex Docker. I totally forgot to disable/delete the Plex Plugin and went bonkers trying to figure out why my settings, libraries was all messed up and ended up nuking my image, nuking my install from the appdata folder and later realizing the plugin was running. Initial testing I actually labeled my server with the plugin to Tower-Plugin, but must of re-installed it and forgot to change it back to Tower-Plugin. 1. Rebuilt the Image 2. Copied over my Plex Database. So glad I redirected my Backup elswhere than just in the Cache folder 3. Totally remembered I made a backup with this. Yahoooooowiieeeeeeeeeeeeeeeeee 4. Restarted everything and everybody was happy Man I can't tell you how much I'm grateful for this app and all the work you do @Squid I'll be throwing a little your way soon. Quote Link to comment
Squid Posted March 16, 2017 Author Share Posted March 16, 2017 This totally saved my Bacon. Dropped in a new Drive and did a rebuild to a larger Data disk, and the Plex Plugin started and and later I started my my Plex Docker. I totally forgot to disable/delete the Plex Plugin [emoji33] and went bonkers trying to figure out why my settings, libraries was all messed up and ended up nuking my image, nuking my install from the appdata folder and later realizing the plugin was running. [emoji5] Initial testing I actually labeled my server with the plugin to Tower-Plugin, but must of re-installed it and forgot to change it back to Tower-Plugin. 1. Rebuilt the Image 2. Copied over my Database. So glad I redirected my Backup elswhere than just in the Cache folder 3. Totally remembered I made a backup with this. Yahoooooowiieeeeeeeeeeeeeeeeee 4. Restarted everything and everybody was happy Man I can't tell you how much I'm grateful for this app and all the work you do [mention=10290]Squid[/mention] I'll be throwing a little [emoji8] your way soon. Got the lovins Kizer. Ty. Yeah I never recommend the backup destination to go to the cache drive. Glad it all worked out for you. I just used it myself to upgrade my cache drive size. Edit. Should be noted that a drag and drop from the backup back to the source over the network may not always work due to hardlinks. I only state that a full restore will work... Sent from my LG-D852 using Tapatalk Quote Link to comment
DazedAndConfused Posted March 17, 2017 Share Posted March 17, 2017 (edited) Quick question, do the default rsync flags follow symlinks? I'm reading that rclone may be updated to handle syimlinks this weekend, but Ive also been seeing the dev say that we need to include the "-l" flag on rsync if thats what we're using to send data to our rclone mounted share. https://github.com/ncw/rclone/issues/1152 Edited March 18, 2017 by DazedAndConfused Quote Link to comment
Squid Posted March 18, 2017 Author Share Posted March 18, 2017 -avXHq -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -r, --recursive recurse into directories -l, --links copy symlinks as symlinks -p, --perms preserve permissions -t, --times preserve modification times -g, --group preserve group -o, --owner preserve owner (super-user only) --devices preserve device files (super-user only) --specials preserve special files -D same as --devices --specials -v, --verbose increase verbosity -X, --xattrs preserve extended attributes -H, --hard-links preserve hard links -q, --quiet suppress non-error messages So, yes Quote Link to comment
DazedAndConfused Posted March 18, 2017 Share Posted March 18, 2017 Hmm, it looks like the "-a" already covers -l if I'm understanding that chart correctly. Quote Link to comment
MickMorley Posted March 27, 2017 Share Posted March 27, 2017 (edited) Hi Squid, It looks like the CA Backup plugin is causing my system to freeze because it does not finish and thus does not restart the docker applications and web ui. Attached is the syslog from this morning. My CA Backup runs every Monday morning at 3AM. You can review the successful run from last week 03/20 @ 3AM. This morning it stopped after "Mar 27 03:40:01 freddie root: mover finished" command. This is not the first occurrence, before when my system was an i3, I would not be able to log in after a CA Backup and copy the log. I would always have to power down the server manually and start it backup. This is the first time it froze with my new i5 but I was actually able to SSH in for the first time and grab the log. edit (update): The unraid will not response to any shutdown, reboot, or powerdown command from SSH. Each command is written in the log but the system still stays in the hanging state. edit (update2): Added community appdata backup log for Squid Thank you for your help. System i5-6600 Asus H170-Pro 3 Gaming 24 MB RAM unRaid 6.3.2 (i think this is the version, it is the newest) syslog_2017-03-27.txt syslog_2017-03-27-7_after startup.txt appdata_backup_20170327.zip Edited March 28, 2017 by MickMorley Added community appdata backup log Quote Link to comment
Squid Posted March 27, 2017 Author Share Posted March 27, 2017 9 hours ago, MickMorley said: Hi Squid, It looks like the CA Backup plugin is causing my system to freeze because it does not finish and thus does not restart the docker applications and web ui. Attached is the syslog from this morning. My CA Backup runs every Monday morning at 3AM. You can review the successful run from last week 03/20 @ 3AM. This morning it stopped after "Mar 27 03:40:01 freddie root: mover finished" command. This is not the first occurrence, before when my system was an i3, I would not be able to log in after a CA Backup and copy the log. I would always have to power down the server manually and start it backup. This is the first time it froze with my new i5 but I was actually able to SSH in for the first time and grab the log. edit (update): The unraid will not response to any shutdown, reboot, or powerdown command from SSH. Each command is written in the log but the system still stays in the hanging state. Thank you for your help. System i5-6600 Asus H170-Pro 3 Gaming 24 MB RAM unRaid 6.3.2 (i think this is the version, it is the newest) syslog_2017-03-27.txt syslog_2017-03-27-7_after startup.txt It would be awesome if I could see the log from the backup cp /var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log /boot And then upload here Quote Link to comment
Holbert Posted March 28, 2017 Share Posted March 28, 2017 20 minutes ago, Squid said: It would be awesome if I could see the log from the backup cp /var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log /boot And then upload here I've actually had the same thing occur to me. WebGUI is completely unaccessible. Can SSH though. Have to do a hard reboot. My log is 38.7MB so a little too big for upload. Is there something in there that you'd like to look at in particular? Let me know if I can supply any other info. M/B: Gigabyte Technology Co., Ltd. - F2A88XM-D3H CPU: AMD Athlon™ X4 860K Quad Core @ 3700 Memory: 8 GB Unraid: 6.3.1 Settings: Start of Log: Stopping Deluge-VPN Stopping Dolphin Stopping Grafana Stopping Influxdb Stopping LS-Jackett Stopping LS-plexpy Stopping LS-Radarr Stopping openvpn-as Stopping Plex Stopping Sonarr Stopping untelegraf Backing up USB Flash Drive Backing up VM XML's Executing rsync: /usr/bin/rsync -avXHq --delete --exclude "docker.img" --log-file="/var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log" "/mnt/user/Dockers/" "/mnt/user/File Vault/DockerBackup/[email protected]" > /dev/null 2>&1 2017/03/27 04:02:10 [26210] building file list 2017/03/27 04:02:11 [26210] .d...pog..x ./ End of Log: 2017/03/27 04:47:06 [26210] sent 35,698,270,539 bytes received 2,585,578 bytes 13,239,701.88 bytes/sec 2017/03/27 04:47:06 [26210] total size is 35,675,301,506 speedup is 1.00 Restarting Deluge-VPN Restarting Dolphin Restarting Grafana Restarting Influxdb Restarting LS-Jackett Restarting LS-plexpy Restarting LS-Radarr Restarting openvpn-as Restarting Plex Restarting Sonarr Restarting untelegraf Backup/Restore Complete. Rsync Status: Success Deleting Dated Backup set: /mnt/user/File Vault/DockerBackup/[email protected] Deleting /mnt/user/File Vault/DockerBackup/[email protected] Quote Link to comment
Squid Posted March 28, 2017 Author Share Posted March 28, 2017 1 minute ago, Holbert said: I've actually had the same thing occur to me. WebGUI is completely unaccessible. Can SSH though. Have to do a hard reboot. My log is 38.7MB so a little too big for upload. Is there something in there that you'd like to look at in particular? Let me know if I can supply any other info. M/B: Gigabyte Technology Co., Ltd. - F2A88XM-D3H CPU: AMD Athlon™ X4 860K Quad Core @ 3700 Memory: 8 GB Unraid: 6.3.1 Settings: Start of Log: Stopping Deluge-VPN Stopping Dolphin Stopping Grafana Stopping Influxdb Stopping LS-Jackett Stopping LS-plexpy Stopping LS-Radarr Stopping openvpn-as Stopping Plex Stopping Sonarr Stopping untelegraf Backing up USB Flash Drive Backing up VM XML's Executing rsync: /usr/bin/rsync -avXHq --delete --exclude "docker.img" --log-file="/var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log" "/mnt/user/Dockers/" "/mnt/user/File Vault/DockerBackup/[email protected]" > /dev/null 2>&1 2017/03/27 04:02:10 [26210] building file list 2017/03/27 04:02:11 [26210] .d...pog..x ./ End of Log: 2017/03/27 04:47:06 [26210] sent 35,698,270,539 bytes received 2,585,578 bytes 13,239,701.88 bytes/sec 2017/03/27 04:47:06 [26210] total size is 35,675,301,506 speedup is 1.00 Restarting Deluge-VPN Restarting Dolphin Restarting Grafana Restarting Influxdb Restarting LS-Jackett Restarting LS-plexpy Restarting LS-Radarr Restarting openvpn-as Restarting Plex Restarting Sonarr Restarting untelegraf Backup/Restore Complete. Rsync Status: Success Deleting Dated Backup set: /mnt/user/File Vault/DockerBackup/[email protected] Deleting /mnt/user/File Vault/DockerBackup/[email protected] Your's finished. IE: nothing of interest there... @MickMorley's did not. But the system kept on chugging away. That's what's making me curious. I did just start a backup on my system, and invoked mover several times, and it finished no problems. 4 minutes ago, Holbert said: I've actually had the same thing occur to me. WebGUI is completely unaccessible. Install Fix Common Problems, pop it into trouble shooting mode, and try to replicate the issue. Then upload FCPsyslog_tail.txt and the last diagnostics generated Quote Link to comment
Holbert Posted March 28, 2017 Share Posted March 28, 2017 6 minutes ago, Squid said: Your's finished. IE: nothing of interest there... @MickMorley's did not. But the system kept on chugging away. That's what's making me curious. I did just start a backup on my system, and invoked mover several times, and it finished no problems. Install Fix Common Problems, pop it into trouble shooting mode, and try to replicate the issue. Then upload FCPsyslog_tail.txt and the last diagnostics generated Will do. I run it weekly on Sunday night/Monday Morning at 3AM, but I'll try it tonight and try to replicate the issue. It's happened once before about 5 or 6 weeks ago, so this is the second time. Quote Link to comment
MickMorley Posted March 28, 2017 Share Posted March 28, 2017 (edited) 1 hour ago, Squid said: It would be awesome if I could see the log from the backup cp /var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log /boot And then upload here Hi Squid - I edited my post and uploaded the appdata log above to keep it together. I checked out the log and it doesn't really correspond with the syslog. According to appdata log, it looks like it froze when backing up Emby. FYI - @Holbert, I started troubleshooting mode via fix common problems and couldn't get it to occur during a force backup. If these logs don't show enough data to warrant a cause then I will enable troubleshooting mode until it occurs again. @Squid, if you need troubleshooting logs from fix common problems to help find and fix this issue, let me know. Otherwise I may just remove Emby to see if it occurs again. Edited March 28, 2017 by MickMorley Quote Link to comment
Squid Posted March 28, 2017 Author Share Posted March 28, 2017 16 minutes ago, MickMorley said: Hi Squid - I edited my post and uploaded the appdata log above to keep it together. I checked out the log and it doesn't really correspond with the syslog. According to appdata log, it looks like it froze when backing up Emby. FYI - @Holbert, I started troubleshooting mode via fix common problems and couldn't get it to occur during a force backup. If these logs don't show enough data to warrant a cause then I will enable troubleshooting mode until it occurs again. @Squid, if you need troubleshooting logs from fix common problems to help find and fix this issue, let me know. Otherwise I may just remove Emby to see if it occurs again. Can you show me a screenshot of Settings - Docker Settings (Click Advanced in the top right) Quote Link to comment
MickMorley Posted March 28, 2017 Share Posted March 28, 2017 Can you show me a screenshot of Settings - Docker Settings (Click Advanced in the top right) Here you go, I had to screenshot it from my phone. Quote Link to comment
Squid Posted March 28, 2017 Author Share Posted March 28, 2017 Well that blew away the only theory I had... Here's what's going on: The rsync started and was carrying on no problems. Then, up in the middle of it it just decided to quit. This is probably because something somewhere killed off CA's backup script. Except all of this happened without any errors or notifications or anything being logged anywhere. There have been intermittent reports of mover crashing (also uses rsync), but AFAIK no one has ever adequately explained why or how. This is a built-in command to linux, and is not an add-on at all. But the messed up thing is that if you run it manually, it runs to completion. My suggestion on how to fix you're probably not going to like: Easter is coming... Take the server with you on Easter Sunday to your local church and get it blessed along with all of your food for the dinner.... It's possessed. IE: I have no clue what's going on, nor is there anything I can do about it. The cause is not within CA Appdata Backup. EDIT: Not quite true. I have a feeling that php is getting Bus Errors causing complete abnormal exits (may be a bug in php7) (does anything appear on the locally attached monitor?), but either way it's way out of my purvue. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.