[Plugin] CA Appdata Backup / Restore v2


Squid

Recommended Posts

On 1/11/2020 at 1:22 AM, ConnectivIT said:

I think a number of people are having issues with how long backup/verify is taking (particularly users of Plex docker with hundreds of thousands or even millions of files)  I had an idea that might resolve this:

...

So in the settings you could specify something like:

 

online backup folders:

/mnt/user/appdata/plex/Library/Application Support/Plex Media Server/Metadata,

/mnt/user/appdata/plex/Library/Application Support/Plex Media Server/Media

...

Hello, 

You mention excluding these folders from the backup config in CA Backup / Restore. Will this reduce the time it takes to restore? I have been running my restore for the last 3 days, and so far I see no indication of where it is at in its progress. Is it supposed to take this long to restore the AppData folder? It's about 19GB in size... I still think it should be faster than that. 

The two file paths above, are these all files related to the media itself? or does it also include the "watched" markers data? I ask because I would like to know where I am at with my shows, but I do not need to back up all the show metadata, album and cover pictures etc... Plex can re-scrape this info, as long as it still has the actual information of each show and movie. Just not the larger image files. 

Is it normal for a restore to take this long? 

 

On 1/11/2020 at 1:22 AM, ConnectivIT said:

 

 

Link to comment
On 7/11/2020 at 9:34 PM, Furby8704 said:

update:

restore stops 20gb in out of 90gb tar file but plugins says restore was completed and completed succesfully.

 

 

How long did your restore take? Mine has been running for over 3 days now... 19GB in total size. 

 

Link to comment
6 hours ago, NLDer said:

How long did your restore take? Mine has been running for over 3 days now... 19GB in total size. 

 

 

my new build was unstable which was causing the issues.

but back up and restore was about and a little over an hour each once i took care of my system.

 

Link to comment

 

For some reason, the "delete old backups setting" is not working for me. I see the attempts in the syslog, but the folders are left behind:

Jul 19 03:04:04 UnRAID CA Backup/Restore: #######################
Jul 19 03:04:04 UnRAID CA Backup/Restore: appData Backup complete
Jul 19 03:04:04 UnRAID CA Backup/Restore: #######################
Jul 19 03:04:04 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 03:04:04 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 03:04:04 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 03:04:04 UnRAID CA Backup/Restore: Backup / Restore Completed

I tried running a manual backup a couple of times, but the same folders were still left behind: 

Jul 19 17:20:07 UnRAID CA Backup/Restore: #######################
Jul 19 17:20:07 UnRAID CA Backup/Restore: appData Backup complete
Jul 19 17:20:07 UnRAID CA Backup/Restore: #######################
Jul 19 17:20:07 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 17:20:07 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 17:20:07 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 17:20:07 UnRAID CA Backup/Restore: Backup / Restore Completed
...
Jul 19 17:53:44 UnRAID CA Backup/Restore: #######################
Jul 19 17:53:44 UnRAID CA Backup/Restore: appData Backup complete
Jul 19 17:53:44 UnRAID CA Backup/Restore: #######################
Jul 19 17:53:44 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 17:53:44 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 17:53:45 UnRAID CA Backup/Restore: Deleting /mnt/user/backups/unraid_appdata/[email protected]
Jul 19 17:53:45 UnRAID CA Backup/Restore: Backup / Restore Completed

This is the destination folder as of now:

root@UnRAID:~# ls -la /mnt/user/backups/unraid_appdata/
total 0
drwxrwxrwx 1 root   root  96 Jul 19 17:53 ./
drwxrwxrwx 1 nobody users 80 Jul  4 13:43 ../
drwxrwxrwx 1 root   root  38 Jul  4 12:44 2020-07-04\@12.42/
drwxrwxrwx 1 root   root  38 Jul  6 03:01 2020-07-06\@03.00/
drwxrwxrwx 1 root   root  38 Jul 12 03:02 2020-07-12\@03.00/
drwxrwxrwx 1 root   root  32 Jul 19 03:01 2020-07-19\@03.00/
drwxrwxrwx 1 root   root  32 Jul 19 17:17 2020-07-19\@17.15/
drwxrwxrwx 1 root   root  32 Jul 19 17:51 2020-07-19\@17.49/

 

Appreciate any suggestions.

Link to comment
  • 2 weeks later...
4 hours ago, Alex.b said:

I want to exclude video thumbnails from Plex in backup, is it the right path to exclude the cache folder ?


/mnt/user/appdata/plex/Library/Application\ Support/Plex\ Media\ Server/Cache/

 

I knocked out /mnt/user/appdata/Plex/Library/Application Support/Plex Media Server/Media/ and that seems to work fine. Once in a blue moon I'll remove it, let the backup take everything, then store the backup elsewhere. That way if I do need to recover, I can pull that snapshot and then the latest as an "incremental" from there.

Link to comment
On 7/28/2018 at 4:19 PM, CaptainTivo said:

2. I am concerned about this warning: 

 

I have a dedicated share for all backups from all my computers called Backup1, which is limited to a specific disk.  Within that share are a bunch of directories with different backup types:

 

Backup1\

   backups\

      PC1\

      PC2\

  Game backups\

  Pictures\

 

etc.  Can I simply make a dedicated folder for this backup?  i.e.:

 

Destination Share:   Backup1\backups\mariadb

 

The warning seems to imply that this could write over files/folders at the Backup1 directory  level.

 

Thanks.

 

I'm also curious about this.  I'd prefer the store my appdata backups at /mnt/user/backup/appdata_backup instead of a dedicated share

Link to comment
On 7/28/2018 at 4:19 PM, CaptainTivo said:

I have a dedicated share for all backups from all my computers called Backup1, which is limited to a specific disk.  Within that share are a bunch of directories with different backup types:

 

Backup1\

   backups\

      PC1\

      PC2\

  Game backups\

  Pictures\

 

etc.  Can I simply make a dedicated folder for this backup?  i.e.:

 

The warning seems to imply that this could write over files/folders at the Backup1 directory  level.

This is exactly what I do (and I suspect most people do). The warning is saying it could overwrite files in the destination path. If the destination path specifies a subfolder of a user share, it won't go up a level and overwrite other things.

Link to comment
53 minutes ago, trurl said:

This is exactly what I do (and I suspect most people do). The warning is saying it could overwrite files in the destination path. If the destination path specifies a subfolder of a user share, it won't go up a level and overwrite other things.

great because using having appdata and appdata_backup was really cramping my auto-complete game in the cli

Link to comment

Does anyone have a clear understanding of libvert.img and why it would need to be backed up.  Of course, including it in the backup is no big deal, but I also offsite my backups of appdata and flash to AWS S3.  I'm wondering if I need to do the same for libvert.img and how important it is.  Currently i'm doing a checksum to see the backed up libvert.img file has changed, and even though I have no VMs it seems the checksum is changing every 2-3 days, which then triggers a 1GB upload to aws, which seems not necessary.  Am I missing something?

 

I tried to search around but it doesn't seem there is a clear answer of what libvert.img (beyond used to run VM) and why/how it would change, especially when not running VMs.

 

Thank you!

Link to comment
1 minute ago, Squid said:

It contains the VM xml's etc.  If you have no VMs, then zero point in backing it up.

Thanks!  I eventually will run a Blue Iris VM, but I'm guessing that just backing up my Blue Iris settings from the VM would be sufficient instead of that massive file.  

 

Appreciate it!

Link to comment
  • 2 weeks later...

First let me say that this is a good plugin and that it has saved my ass a few times. but I have a question.

 

at the moment all my docker contenter information is stored in one place ' appdata folder '  that is stored in the cache, but I am about to move my Plex dockcker container out and onto a SSD. and this SSD with be connected via the Unassigned drive.

 

so my question is. will the ' CA Backup / Restore Appdata ' plugin see this new location and backup my Plex data ? 

Link to comment
50 minutes ago, chris_netsmart said:

First let me say that this is a good plugin and that it has saved my ass a few times. but I have a question.

 

at the moment all my docker contenter information is stored in one place ' appdata folder '  that is stored in the cache, but I am about to move my Plex dockcker container out and onto a SSD. and this SSD with be connected via the Unassigned drive.

 

so my question is. will the ' CA Backup / Restore Appdata ' plugin see this new location and backup my Plex data ? 

Unassigned Devices are not part of user shares. You could make the plugin backup a folder on an Unassigned Device, but then it wouldn't be set to backup appdata for other dockers that were in the appdata user share.

Link to comment
On 8/10/2020 at 4:04 PM, trurl said:

Unassigned Devices are not part of user shares. You could make the plugin backup a folder on an Unassigned Device, but then it wouldn't be set to backup appdata for other dockers that were in the appdata user share.

Thanks for the info.  I had a feeling that it was not designed to look at two difference locations.

 

What i will do is look at a script, and schedule and get it to copy the plex data into the appdata folder.

 

here is the script I am using

 

sudo docker stop PlexMediaServer                                               -  Stop Docker                   

cp -r  -p /mnt/disk4/* /mnt/disk8                              - Copy from Plexdata to un-appdata  daily 3am

sudo docker start PlexMediaServer                                            -  Start Docker    

             

Edited by chris_netsmart
Added my Script code
Link to comment

Yeah, utilizing separate appdata sources adds a whole other layer of complexity to the GUI and underlying script, and when push comes to shove this is simply ~1000 lines of code acting as a front end to a single tar command.

 

Multiple things are on the whiteboard for the plugin, namely individual storage of folders within the appdata, and multiple sources, but there is no ETA on that, and are considered to be part of the wish list.

Link to comment
  • 2 weeks later...
On 7/28/2020 at 3:10 PM, Flick said:

I knocked out /mnt/user/appdata/Plex/Library/Application Support/Plex Media Server/Media/ and that seems to work fine. Once in a blue moon I'll remove it, let the backup take everything, then store the backup elsewhere. That way if I do need to recover, I can pull that snapshot and then the latest as an "incremental" from there.

My compressed backup was 164GB before I excluded the Plex Media location. What I've done is exclude it and then add a Custom Start Script to run rsync on that location instead. It's static content, so I'm not too concerned about it syncing while the Docker container is running and rsync only has to copy the new/modified files.

Link to comment
22 hours ago, Simon said:

My compressed backup was 164GB before I excluded the Plex Media location. What I've done is exclude it and then add a Custom Start Script to run rsync on that location instead. It's static content, so I'm not too concerned about it syncing while the Docker container is running and rsync only has to copy the new/modified files.

Brilliant! I'm going to do this as well. Thanks!

Link to comment
On 8/22/2020 at 11:09 AM, Squid said:

How do you know?  What shows up if you open the backup set with WinRAR on a windows box?

I just wanted to let you know that I'm a dumb-dumb and figured it out. My backup location had cache enabled. So when it would backup, the cache would get full after Plex (~60GB) and stop. I made a new backup share with no cache enabled and it's working well now. 

 

Sorry to waste your time, and thank you very much for your help. 

 

P.S. I bought you a beer for your trouble ;)

Edited by G3TG0T
  • Like 1
Link to comment
  • Squid locked this topic
Guest
This topic is now closed to further replies.