Jump to content
Squid

[Plugin] CA Appdata Backup / Restore

374 posts in this topic Last Reply

Recommended Posts

Awesome plug man, has saved me quite a few times from myself accidentally deleting stuff from appdata.
 Couple of questions:
 
  1. is stopping dockers really necessary? Never seen the 'file open error' when copying appdata manually;
  2. how did you find out which files need to be backed up for vms? (it's not just the xml templates);
  3. as an extension to 1st - what logic is applied on the docker restart? Every time backup job finishes, it also starts dnsmasq after starting the dockers (and other services?), so the dns docker fails to start up since port 53 is occupied (my issue thread)

You have the option to keep any docker running during the backup. Your mileage may vary. If errors result by doing so the plugin will report it.

The files for the VMs are the xml templates and by request the nvme roms

Dockers that the plugin stops at the start are the ones which are restarted when it finishes

Sent from my LG-D852 using Tapatalk

Share this post


Link to post

So there's no apparent reason why dnsmasq is started? Note it doesn't happen when the physical box is rebooted.

Edited by tuxbass

Share this post


Link to post
So there's no apparent reason why dnsmasq is started? Note it doesn't happen when the physical box is rebooted.

If it's showing as running when the backup starts then I stop it and then restart it. If it's not showing as running when the backup starts but the plugin restarts it then I'd have to give you a script to help me identify what's going on

Sent from my LG-D852 using Tapatalk

Share this post


Link to post
11 hours ago, Squid said:


If it's showing as running when the backup starts then I stop it and then restart it. If it's not showing as running when the backup starts but the plugin restarts it then I'd have to give you a script to help me identify what's going on

Sent from my LG-D852 using Tapatalk
 

Hmm, just started the job manually and it didn't reproduce; yet every Monday (when it's scheduled to run) the dns fails to start since dnsmasq had been started. Might have jumped to a conclusion here. So if the backup app detects that dnsmasq process is running, it'll restart it, and only then?

Share this post


Link to post
Hmm, just started the job manually and it didn't reproduce; yet every Monday (when it's scheduled to run) the dns fails to start since dnsmasq had been started. Might have jumped to a conclusion here. So if the backup app detects that dnsmasq process is running, it'll restart it, and only then?

Correct.

Sent from my LG-D852 using Tapatalk

Share this post


Link to post

Whenever I do USB Backup, it starts copying files to my rclone disk and then the plugin just appears to stop doing anything. Is there a way to see a verbose output of what its currently doing? All my log says right now is "Backing up USB Drive". I started this at 7:38 and no other files have been changed since around then.

Share this post


Link to post

Not sure...  This is what the USB backup actually is: 

 

mkdir -p $destination   # makes the path if it doesn't exist

...Creates a file called DISK_ASSIGNMENTS.txt in /boot/config

cp /boot/* $destination -R -v    # the -v is superfluous as the verbose output is actually ignored

mv $destination/config/super.dat $destination/config/super.dat.CA_BACKUP

 

very simple.  Something to do with rclone, but I can't help diagnose that...  Easy test is to set a separate destination for USB to the array.  If that works its a given that its an rclone issue

 

 

Share this post


Link to post

What about the "Error in File IO (code 11)" that I get when just copying the Appdata folder. It happens on linuxserverio dockers as far as I can see.

Share this post


Link to post
47 minutes ago, DazedAndConfused said:

What about the "Error in File IO (code 11)" that I get when just copying the Appdata folder. It happens on linuxserverio dockers as far as I can see.

Its an rsync error.  Is it also going to an rclone destination?  If so, then best guess rclone does not support symlinks and/or hardlinks, in which case it is not suitable for a destination since most apps utilize them.  (Plex as an example makes HUGE use of them).  

Share this post


Link to post

This totally saved my Bacon.

 

Dropped in a new Drive and did a rebuild to a larger Data disk, and the Plex Plugin started and and later I started my my Plex Docker. I totally forgot to disable/delete the Plex Plugin :o and went bonkers trying to figure out why my settings, libraries was all messed up and ended up nuking my image, nuking my install from the appdata folder and later realizing the plugin was running. :$ Initial testing I actually labeled my server with the plugin to Tower-Plugin, but must of re-installed it and forgot to change it back to Tower-Plugin. 

 

1. Rebuilt the Image

2. Copied over my Plex Database. So glad I redirected my Backup elswhere than just in the Cache folder

3. Totally remembered I made a backup with this. Yahoooooowiieeeeeeeeeeeeeeeeee

4. Restarted everything and everybody was happy

 

Man I can't tell you how much I'm grateful for this app and all the work you do @Squid

I'll be throwing a little :x your way soon. 

Share this post


Link to post
This totally saved my Bacon.  

Dropped in a new Drive and did a rebuild to a larger Data disk, and the Plex Plugin started and and later I started my my Plex Docker. I totally forgot to disable/delete the Plex Plugin  and went bonkers trying to figure out why my settings, libraries was all messed up and ended up nuking my image, nuking my install from the appdata folder and later realizing the plugin was running.  Initial testing I actually labeled my server with the plugin to Tower-Plugin, but must of re-installed it and forgot to change it back to Tower-Plugin. 

 

1. Rebuilt the Image

2. Copied over my Database. So glad I redirected my Backup elswhere than just in the Cache folder

3. Totally remembered I made a backup with this. Yahoooooowiieeeeeeeeeeeeeeeeee

4. Restarted everything and everybody was happy

 

Man I can't tell you how much I'm grateful for this app and all the work you do [mention=10290]Squid[/mention]

I'll be throwing a little  your way soon. 

 

Got the lovins Kizer. Ty. Yeah I never recommend the backup destination to go to the cache drive.

 

Glad it all worked out for you. I just used it myself to upgrade my cache drive size.

 

Edit. Should be noted that a drag and drop from the backup back to the source over the network may not always work due to hardlinks. I only state that a full restore will work...

 

Sent from my LG-D852 using Tapatalk

 

 

 

 

 

Share this post


Link to post

Quick question, do the default rsync flags follow symlinks? I'm reading that rclone may be updated to handle syimlinks this weekend, but Ive also been seeing the dev say that we need to include the "-l" flag on rsync if thats what we're using to send data to our rclone mounted share.

 

https://github.com/ncw/rclone/issues/1152

 

Edited by DazedAndConfused

Share this post


Link to post

-avXHq

 

-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
-r, --recursive             recurse into directories
-l, --links                 copy symlinks as symlinks
-p, --perms                 preserve permissions
-t, --times                 preserve modification times
-g, --group                 preserve group
-o, --owner                 preserve owner (super-user only)
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -v, --verbose               increase verbosity
 -X, --xattrs                preserve extended attributes
 -H, --hard-links            preserve hard links
 -q, --quiet                 suppress non-error messages

So, yes

Share this post


Link to post

Correct

Share this post


Link to post

Hi Squid,

 

It looks like the CA Backup plugin is causing my system to freeze because it does not finish and thus does not restart the docker applications and web ui. Attached is the syslog from this morning. My CA Backup runs every Monday morning at 3AM.  You can review the successful run from last week 03/20 @ 3AM.  This morning it stopped after "Mar 27 03:40:01 freddie root: mover finished" command.  This is not the first occurrence, before when my system was an i3, I would not be able to log in after a CA Backup and copy the log.  I would always have to power down the server manually and start it backup.  This is the first time it froze with my new i5 but I was actually able to SSH in for the first time and grab the log.

 

edit (update): The unraid will not response to any shutdown, reboot, or powerdown command from SSH.  Each command is written in the log but the system still stays in the hanging state.

 

edit (update2): Added community appdata backup log for Squid

 

Thank you for your help.

 

System

i5-6600

Asus H170-Pro 3 Gaming

24 MB RAM

unRaid 6.3.2 (i think this is the version, it is the newest)

 

syslog_2017-03-27.txt

syslog_2017-03-27-7_after startup.txt

appdata_backup_20170327.zip

Edited by MickMorley
Added community appdata backup log

Share this post


Link to post
9 hours ago, MickMorley said:

Hi Squid,

 

It looks like the CA Backup plugin is causing my system to freeze because it does not finish and thus does not restart the docker applications and web ui. Attached is the syslog from this morning. My CA Backup runs every Monday morning at 3AM.  You can review the successful run from last week 03/20 @ 3AM.  This morning it stopped after "Mar 27 03:40:01 freddie root: mover finished" command.  This is not the first occurrence, before when my system was an i3, I would not be able to log in after a CA Backup and copy the log.  I would always have to power down the server manually and start it backup.  This is the first time it froze with my new i5 but I was actually able to SSH in for the first time and grab the log.

 

edit (update): The unraid will not response to any shutdown, reboot, or powerdown command from SSH.  Each command is written in the log but the system still stays in the hanging state.

 

Thank you for your help.

 

System

i5-6600

Asus H170-Pro 3 Gaming

24 MB RAM

unRaid 6.3.2 (i think this is the version, it is the newest)

 

syslog_2017-03-27.txt

syslog_2017-03-27-7_after startup.txt

It would be awesome if I could see the log from the backup

 

cp   /var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log   /boot

 

And then upload here

 

Share this post


Link to post
20 minutes ago, Squid said:

It would be awesome if I could see the log from the backup

 

cp   /var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log   /boot

 

And then upload here

 

 

I've actually had the same thing occur to me. WebGUI is completely unaccessible. Can SSH though. Have to do a hard reboot. My log is 38.7MB so a little too big for upload. Is there something in there that you'd like to look at in particular? Let me know if I can supply any other info.

 

M/B: Gigabyte Technology Co., Ltd. - F2A88XM-D3H
CPU: AMD Athlon™ X4 860K Quad Core @ 3700
Memory: 8 GB
Unraid:               6.3.1

 

Settings:

58d9ab219c96c_ScreenShot2017-03-27at7_14_12PM.thumb.png.9965959219db759ff8939dd317f75255.png

 

Start of Log:

Stopping Deluge-VPN
Stopping Dolphin
Stopping Grafana
Stopping Influxdb
Stopping LS-Jackett
Stopping LS-plexpy
Stopping LS-Radarr
Stopping openvpn-as
Stopping Plex
Stopping Sonarr
Stopping untelegraf
Backing up USB Flash Drive
Backing up VM XML's
Executing rsync: /usr/bin/rsync -avXHq --delete --exclude "docker.img"  --log-file="/var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log" "/mnt/user/Dockers/" "/mnt/user/File Vault/DockerBackup/2017-03-27@04.00" > /dev/null 2>&1
2017/03/27 04:02:10 [26210] building file list
2017/03/27 04:02:11 [26210] .d...pog..x ./

 

End of Log:

2017/03/27 04:47:06 [26210] sent 35,698,270,539 bytes  received 2,585,578 bytes  13,239,701.88 bytes/sec
2017/03/27 04:47:06 [26210] total size is 35,675,301,506  speedup is 1.00
Restarting Deluge-VPN
Restarting Dolphin
Restarting Grafana
Restarting Influxdb
Restarting LS-Jackett
Restarting LS-plexpy
Restarting LS-Radarr
Restarting openvpn-as
Restarting Plex
Restarting Sonarr
Restarting untelegraf
Backup/Restore Complete.  Rsync Status: Success
Deleting Dated Backup set: /mnt/user/File Vault/DockerBackup/2017-03-13@04.00
Deleting /mnt/user/File Vault/DockerBackup/2017-03-13@04.00

 

Share this post


Link to post
1 minute ago, Holbert said:

 

I've actually had the same thing occur to me. WebGUI is completely unaccessible. Can SSH though. Have to do a hard reboot. My log is 38.7MB so a little too big for upload. Is there something in there that you'd like to look at in particular? Let me know if I can supply any other info.

 

M/B: Gigabyte Technology Co., Ltd. - F2A88XM-D3H
CPU: AMD Athlon™ X4 860K Quad Core @ 3700
Memory: 8 GB
Unraid:               6.3.1

 

Settings:

58d9ab219c96c_ScreenShot2017-03-27at7_14_12PM.thumb.png.9965959219db759ff8939dd317f75255.png

 

Start of Log:


Stopping Deluge-VPN
Stopping Dolphin
Stopping Grafana
Stopping Influxdb
Stopping LS-Jackett
Stopping LS-plexpy
Stopping LS-Radarr
Stopping openvpn-as
Stopping Plex
Stopping Sonarr
Stopping untelegraf
Backing up USB Flash Drive
Backing up VM XML's
Executing rsync: /usr/bin/rsync -avXHq --delete --exclude "docker.img"  --log-file="/var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log" "/mnt/user/Dockers/" "/mnt/user/File Vault/DockerBackup/2017-03-27@04.00" > /dev/null 2>&1
2017/03/27 04:02:10 [26210] building file list
2017/03/27 04:02:11 [26210] .d...pog..x ./

 

End of Log:


2017/03/27 04:47:06 [26210] sent 35,698,270,539 bytes  received 2,585,578 bytes  13,239,701.88 bytes/sec
2017/03/27 04:47:06 [26210] total size is 35,675,301,506  speedup is 1.00
Restarting Deluge-VPN
Restarting Dolphin
Restarting Grafana
Restarting Influxdb
Restarting LS-Jackett
Restarting LS-plexpy
Restarting LS-Radarr
Restarting openvpn-as
Restarting Plex
Restarting Sonarr
Restarting untelegraf
Backup/Restore Complete.  Rsync Status: Success
Deleting Dated Backup set: /mnt/user/File Vault/DockerBackup/2017-03-13@04.00
Deleting /mnt/user/File Vault/DockerBackup/2017-03-13@04.00

 

Your's finished.  IE: nothing of interest there...  @MickMorley's did not.  But the system kept on chugging away.  That's what's making me curious.

 

I did just start a backup on my system, and invoked mover several times, and it finished no problems.

 

4 minutes ago, Holbert said:

I've actually had the same thing occur to me. WebGUI is completely unaccessible.

Install Fix Common Problems, pop it into trouble shooting mode, and try to replicate the issue.  Then upload FCPsyslog_tail.txt and the last diagnostics generated

Share this post


Link to post
6 minutes ago, Squid said:

Your's finished.  IE: nothing of interest there...  @MickMorley's did not.  But the system kept on chugging away.  That's what's making me curious.

 

I did just start a backup on my system, and invoked mover several times, and it finished no problems.

 

Install Fix Common Problems, pop it into trouble shooting mode, and try to replicate the issue.  Then upload FCPsyslog_tail.txt and the last diagnostics generated

 

Will do. I run it weekly on Sunday night/Monday Morning at 3AM, but I'll try it tonight and try to replicate the issue. It's happened once before about 5 or 6 weeks ago, so this is the second time.

Share this post


Link to post
1 hour ago, Squid said:

It would be awesome if I could see the log from the backup

 

cp   /var/lib/docker/unraid/ca.backup.datastore/appdata_backup.log   /boot

 

And then upload here

 

 

Hi Squid - I edited my post and uploaded the appdata log above to keep it together. I checked out the log and it doesn't really correspond with the syslog.  According to appdata log, it looks like it froze when backing up Emby.

 

FYI - @Holbert, I started troubleshooting mode via fix common problems and couldn't get it to occur during a force backup.  If these logs don't show enough data to warrant a cause then I will enable troubleshooting mode until it occurs again.

 

@Squid, if you need troubleshooting logs from fix common problems to help find and fix this issue, let me know. Otherwise I may just remove Emby to see if it occurs again.

Edited by MickMorley

Share this post


Link to post
16 minutes ago, MickMorley said:

 

Hi Squid - I edited my post and uploaded the appdata log above to keep it together. I checked out the log and it doesn't really correspond with the syslog.  According to appdata log, it looks like it froze when backing up Emby.

 

FYI - @Holbert, I started troubleshooting mode via fix common problems and couldn't get it to occur during a force backup.  If these logs don't show enough data to warrant a cause then I will enable troubleshooting mode until it occurs again.

 

@Squid, if you need troubleshooting logs from fix common problems to help find and fix this issue, let me know. Otherwise I may just remove Emby to see if it occurs again.

Can you show me a screenshot of Settings - Docker Settings (Click Advanced in the top right)

 

Share this post


Link to post
Can you show me a screenshot of Settings - Docker Settings (Click Advanced in the top right)
 


Here you go, I had to screenshot it from my phone. 691398b811f6878193f125b912cb9d9f.jpg

Share this post


Link to post

Well that blew away the only theory I had...

 

Here's what's going on:

 

The rsync started and was carrying on no problems.  Then, up in the middle of it it just decided to quit.

 

This is probably because something somewhere killed off CA's backup script.  Except all of this happened without any errors or notifications or anything being logged anywhere.

 

There have been intermittent reports of mover crashing (also uses rsync), but AFAIK no one has ever adequately explained why or how.  This is a built-in command to linux, and is not an add-on at all.  But the messed up thing is that if you run it manually, it runs to completion.

 

My suggestion on how to fix you're probably not going to like:  Easter is coming...  Take the server with you on Easter Sunday to your local church and get it blessed along with all of your food for the dinner....  It's possessed.  >:D

 

IE:  I have no clue what's going on, nor is there anything I can do about it.  The cause is not within CA Appdata Backup.

 

EDIT: Not quite true.  I have a feeling that php is getting Bus Errors causing complete abnormal exits (may be a bug in php7) (does anything appear on the locally attached monitor?), but either way it's way out of my purvue.  

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now