[Plugin] CA Appdata Backup / Restore - Deprecated


Squid

Recommended Posts

What's the best strategy for keeping the backups of your USB?  By default they go onto your unRAID server but if your USB key is hosed then your server won't start so you would have to move the disk(s) to another server to find the backup, wouldn't you?

 

Therefore shouldn't you keep the USB backup on another PC or in the cloud or somewhere else? How about even a second USB key connected to your unRAID server?  That is cheap and would make getting back up a snap as you just move the key or even just change the boot order.

Edited by wayner
Link to comment

Ah, I see.  Thanks for the response.  What I'm looking to do is upload the USB backup to my Crashplan cloud account for exactly the reason that wayner mentions.

 

I could probably point Crashplan at the flash drive itself...but then I don't really get a date-consistent backup to easily restore the flash drive back to a point in time.  

 

I could also just leave the USB backup with the same destination as Appdata....but I don't really need/want to upload the whole Appdata backup to Crashplan.

Edited by alfi33
Link to comment
1 hour ago, Squid said:

, there's not particularly anything on the flash drive that's updated constantly that you'd ever need to worry about having to go back where last week's backup isn't identical to the one made 2 weeks ago.  Its why I never particularly worried about it.

What about when you upgrade versions, like from 6.31 to 6.32,  won't that be out of date?

Link to comment
1 hour ago, wayner said:

What about when you upgrade versions, like from 6.31 to 6.32,  won't that be out of date?

Yeah it would be.  3 files. in which case you upgrade again.  But a dated backup wouldn't make a difference anyways.  (BTW, on upgrades, the previous version is stored on the flash drive in the previous folder)

 

Its on a todo list.  Just not at the top.

Link to comment

How frequently have folks found the USB flash drives going bad?  If it happens every once in a while then why not to have a second USB flash drive that is a clone of the first (and have that functionality built into CA Backup) and have them set up 1 & 2 in the boot order?  That way you have redundancy for your system "drive" at a cost of less than $10.  Or is there some reason why this is a bad idea?

 

Link to comment
20 minutes ago, wayner said:

How frequently have folks found the USB flash drives going bad?  If it happens every once in a while then why not to have a second USB flash drive that is a clone of the first (and have that functionality built into CA Backup) and have them set up 1 & 2 in the boot order?  That way you have redundancy for your system "drive" at a cost of less than $10.  Or is there some reason why this is a bad idea?

 

You didn't include the cost of the unRAID license in your $10 cost. Each flash drive would need one.

Link to comment
18 minutes ago, wayner said:

How frequently have folks found the USB flash drives going bad?  If it happens every once in a while then why not to have a second USB flash drive that is a clone of the first (and have that functionality built into CA Backup) and have them set up 1 & 2 in the boot order?  That way you have redundancy for your system "drive" at a cost of less than $10.  Or is there some reason why this is a bad idea?

 

There can only be one flash drive labelled UNRAID, and that is the only one that will boot.

Link to comment
19 minutes ago, wayner said:

How frequently have folks found the USB flash drives going bad?  If it happens every once in a while then why not to have a second USB flash drive that is a clone of the first (and have that functionality built into CA Backup) and have them set up 1 & 2 in the boot order?  That way you have redundancy for your system "drive" at a cost of less than $10.  Or is there some reason why this is a bad idea?

 

Flash failure is rare.  Accidentally corrupting / deleting files is far more common.

 

You idea would actually work setting the second flash as a backup location with the following caveat:

 

It cannot be named UNRAID.  That would really mess up the system.  You would still have to reassign your drives (or rename the backup of the super.dat file if a failure occurred).  CA doesnot and will not ever backup the super.dat file without renaming it.  This is because there is a situation that can happen when you rearrange drives after a backup has been done, and a failure happens before the next backup.  This could potentially trash a data drive.  (There is also a text file named drive-assignments.txt that details the exact assignments as of the last backup)

Link to comment
2 minutes ago, wgstarks said:

You didn't include the cost of the unRAID license in your $10 cost. Each flash drive would need one.

The assumption is that if the backup takes over, it would get re-registered with the existing key file.  Only one registration is needed.

Link to comment
Just now, Squid said:

The assumption is that if the backup takes over, it would get re-registered with the existing key file.  Only one registration is needed.

It could get re-registered, but that process would result in a new key file and the original key which worked with the original flash would be blacklisted.

Link to comment
2 minutes ago, trurl said:

It could get re-registered, but that process would result in a new key file and the original key which worked with the original flash would be blacklisted.

And isn't some user interaction required? Can't remember for sure but I think I had to at least click a button last time. Wouldn't just be seamless boot for a second device.

Link to comment

Maybe this should be given some thought - if the "UNRAID" drive won't boot then try "UNRAID_BK" and allow it to run temporarily until the license is re-registered.

 

In reading some other threads on this it seems like the way to backup the USB is to backup /tower/flash to any other PC on your system or to the cloud.  You should even be able to do this on a Windows machine on a daily basis using SyncToy.  It seems to me that the least useful place to back this up would be on your unRAID server since it is tricky to get to it when you will need it to recover from a flash drive failure.

Link to comment

I already had that scenario. Acidentally delieted /boot. The procedure I had to take was prepareing a whole new flash drive with a fresh install of UnRaid, boot, install unassigned devices... just to copy of the data of the backup   And then recreate the flash drive with the recovered data. 

First thing I did after that procedure was to schedule a weekly incremental versioned offside backup to a friends unraid box via duplicati and sftp. That won't happen again. 

Link to comment
47 minutes ago, wgstarks said:

And isn't some user interaction required? Can't remember for sure but I think I had to at least click a button last time. Wouldn't just be seamless boot for a second device.

Yes, it requires user interaction, and there are limits to how frequently this can be done automatically. So automating this failover is probably not a good idea since often the original flash drive with its original key is still usable, though it might require fixing in some way.

 

As for where else to do the backup, I have mine backing up to an NTFS Unassigned Device that I can easily plug into my Windows computer. Another possible backup location is to an SMB share (using Unassigned Devices).

 

You could also use a FAT32 flash drive labelled UNRAID_BK (Unassigned Devices again), and that would be readable from Windows. So, there are solutions available with the existing functionality.

Link to comment

A totally failed flashdrive happens so seldom I doubt it's worth spending any time changing the status quo. As long as you have a relatively recent backup somewhere, it's trivial to get back up and running. Even if the backup is on an array disk, it's still not a huge deal, it just becomes a two step process of mounting the data drives using a trial key, finding the backup, and copying the files over.

 

Start to finish I think I'd spend more time figuring out a suitable candidate for a new flash drive than I would getting the array back up.

 

I've had 1 flash drive failure in the almost 10 years I've been running multiple unraid towers. The strategy of reading from the USB only at boot and to save and retrieve setting changes means there just isn't much wear.

Link to comment

I agree that it is trivial, even if you don't know the drive assignments, you can assign then all as data just to get at the backup. 

 

But the problem with the destination within the array is that if you don't understand what's going on as to how to get at the backup, then panic sets in.

 

Myself, I have the USB backup done to a SMB share (mounted via Unassigned Devices) hosted on one of my desktops.  No fuss, no muss.   The only time I've ever needed to get anything off of it is for when I've inadvertently deleted something that I need from the flash.

Link to comment
8 minutes ago, jonathanm said:

I've had 1 flash drive failure in the almost 10 years I've been running multiple unraid towers. The strategy of reading from the USB only at boot and to save and retrieve setting changes means there just isn't much wear.

Good to know - I only became worried about this as I recently had the second SD card fail for me on a RPi and I figured that SD cards and flash drives are pretty much the same thing.  But I guess the Pi is using the SD card all the time, unlike unRAID's use of the flash drive.

Link to comment
43 minutes ago, wayner said:

I guess the Pi is using the SD card all the time, unlike unRAID's use of the flash drive.

Yep. pi's don't have much ram, so they need to write to their media for anything intensive. No possibility of setting up a 2+GB RAM drive to run out of like unraid. :D

Link to comment
  • 2 weeks later...

I run kodi headless on my box and its being backed up with the appdata plugin.

The issue is that the .kodi/userdata/Thumbnails/* content is 213 megs and makes up over half of the whole backup itself.

This folder is not something I care to backup but I cant exclude this folder ".kodi/userdata/Thumbnails/" from what I can figure out because you can only exclude a top level directory per the Excluded Folder Browser and the Excluded Folders input field is not user editable..

Link to comment
7 hours ago, Squid said:

Manually edit the text field.

Sent from my LG-D852 using Tapatalk
 

 

Looks like the javascript on the page errored when I was selecting the exclude folder which locked it from being edited, clearing cache and trying again I can modify it directly.

So I'll just go this route, thanks

 

Link to comment

The past 2 times a schedule backup ran my whole server has halted to a stop after it finished. It happened for the first time last Sunday and again today.

 

Last week when it happened, I noticed it about 7 hours after the backup finished and I wasn't able to access SMB shares, the web GUI or see anything on the monitor I have connected but I was able to SSH in. Just getting to the login promt took probably 5 minutes and after waiting an hour for the diagnostics to be made I just force restarted it.

 

Today I managed to grab the diagnostics from the web GUI as it only had been a few minutes since the backup finished but after that nothing responded. Shutting it down via the button had it halt at force shutdown so I force restarted it again.

 

I haven't been able to see any errors that could have caused it in the logs myself but I only have a basic knowledge of this.

 

I'm backing up about 200 GB from the cache drive to the array into the user mount to a share that's only on the array, which includes a ton of small files from Plex's metadata library. It's been working fine for 2 or so months until  2'nd of June. Backups are only kept for 28 days to not go over 1 TB usage.

 

I can't remember if I was late to upgrade to 2017.06.12 and it happened after that update or if I updated it when it came out and ran fine the 2 weeks before.

unraid-diagnostics-20170709-0332.zip

Link to comment
10 hours ago, SelfSD said:

The past 2 times a schedule backup ran my whole server has halted to a stop after it finished. It happened for the first time last Sunday and again today.

 

Last week when it happened, I noticed it about 7 hours after the backup finished and I wasn't able to access SMB shares, the web GUI or see anything on the monitor I have connected but I was able to SSH in. Just getting to the login promt took probably 5 minutes and after waiting an hour for the diagnostics to be made I just force restarted it.

 

Today I managed to grab the diagnostics from the web GUI as it only had been a few minutes since the backup finished but after that nothing responded. Shutting it down via the button had it halt at force shutdown so I force restarted it again.

 

I haven't been able to see any errors that could have caused it in the logs myself but I only have a basic knowledge of this.

 

I'm backing up about 200 GB from the cache drive to the array into the user mount to a share that's only on the array, which includes a ton of small files from Plex's metadata library. It's been working fine for 2 or so months until  2'nd of June. Backups are only kept for 28 days to not go over 1 TB usage.

 

I can't remember if I was late to upgrade to 2017.06.12 and it happened after that update or if I updated it when it came out and ran fine the 2 weeks before.

unraid-diagnostics-20170709-0332.zip

The backup finished.  But then it went through and was trying to delete the old dated backup sets.  This is where it was when you grabbed the diagnostics:

rm -rf /mnt/user/backups/appdata_backup/[email protected]

 

Assuming that this backup set was ~200Gig all small files, this will take a while to complete.  However, the command itself will not impact anything else in unRaid.

 

There is nothing that I see in the diagnostics as to why your system was running at a snail's crawl.  (And your CPU is better than mine)

 

One or two users have issues with the standard bash commands used by the backup script (rsync & rm).  No specific cause has been found, and any issue with those commands is completely out of my control.

 

The best I can offer is to install the tips & tweaks plugin and try this:

 

 

  • Upvote 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.