Complete Disaster Recovery Process

[Hotsammysliz]

Hello all!

So I've been using Unraid now for about 3 years and have been loving it just recently upgraded to pro and about to setup another server remotely. I initially setup my 1st Unraid because i had a catastrophic server failure prior to. So now that I've got things all setup running well with Unraid I'm curious what the process would look like if I completely lost my server (i.e fire, flood, etc).

 

I do currently have my flash backup running (I think that's something I setup through a SpaceInvader guide). Then those backups along with all other important data on my arrays is being backed up through duplicati to a remote S3 bucket. So if I had to get all new hardware, in this situation just assume exact same for drive and cache sizes, what would that process exactly look like? Should I be backing up the flash to something not related to my array and if so what and where do you folks do that with? How would I access and actually restore my duplicati backups if the machine I am backing up is gone? should I be making regular exports of my duplicati jobs as well and if so is there a way to schedule that.

 

I've been burnt once so I am backing things up but I've never fully thought about the whole process to restore if i happen to lose everything at once. Hope these questions all make sense. Also not sure where this topic should be exactly since it's not really support or anything so please let me know if this should be moved or deleted and posted somewhere else. Thanks!

[trurl]

Unraid Connect plugin will backup flash to Unraid Cloud. 

 

[TimTheSettler]

On 1/2/2024 at 1:36 PM, Hotsammysliz said:

How would I access and actually restore my duplicati backups if the machine I am backing up is gone?

 

Are you running duplicati as a docker app?  You should have all your docker apps backed up regularly.

 

On 1/2/2024 at 1:36 PM, Hotsammysliz said:

I'm curious what the process would look like if I completely lost my server (i.e fire, flood, etc).

 

The server is gone so what do you do?  The following list is a very basic process.  Maybe someone else here has a better list.

• Buy new hardware including new hard drives.
• Restore the flash backup to a new USB (re-assign the license key to the new USB).
• Boot to unRAID using the restored flash backup.
• Assign the new hard drives to the array.
• Install the docker apps you used to have but then replace the config folders with your backups.
• Using the restored duplicati docker app retrieve your data.

[Hotsammysliz]

8 hours ago, TimTheSettler said:

 

Are you running duplicati as a docker app?  You should have all your docker apps backed up regularly.

 

 Thanks for the info!

 

So duplicati is running as a docker app on the unRAID server and I am backing up my docker apps, as in the appdata folder, through duplicati to a remote location. Is there a different way I should be backing up my docker apps?

 

8 hours ago, TimTheSettler said:

 

The server is gone so what do you do?  The following list is a very basic process.  Maybe someone else here has a better list.

• Buy new hardware including new hard drives.
• Restore the flash backup to a new USB (re-assign the license key to the new USB).
• Boot to unRAID using the restored flash backup.
• Assign the new hard drives to the array.
• Install the docker apps you used to have but then replace the config folders with your backups.
• Using the restored duplicati docker app retrieve your data.

I think the part that confuses me a little in this process is the last 2 steps and maybe that's where my process needs to change.

 

If I'm backing up all my data with duplicati how can I restore that duplicati data for itself before installing duplicati? Hopefully that question makes sense.

 

Thanks again for the help.

[TimTheSettler]

On 1/15/2024 at 9:10 AM, Hotsammysliz said:

If I'm backing up all my data with duplicati how can I restore that duplicati data for itself before installing duplicati? Hopefully that question makes sense.

 

Yes, I can see how this part seems confusing.  What you need to do is back up the docker config somewhere else.  The appdata directory (where the docker config is stored) is on my cache.  I then use the Appdata Backup plugin (from Robin Kluth) to back up the config to the array.  That config is then synchronized to another server using syncthing.  If I lose the cache I have the backup on the array.  If I lose the array then I have the docker still on the cache.  If you lose both the cache and array (the server is fried, burnt, or stolen) then you have the backup somewhere else.  You don't need to use syncthing (or the plugin), you can back up the cache to a flash drive.  Did I miss anything?

[Hotsammysliz]

1 hour ago, TimTheSettler said:

 

Yes, I can see how this part seems confusing.  What you need to do is back up the docker config somewhere else.  The appdata directory (where the docker config is stored) is on my cache.  I then use the Appdata Backup plugin (from Robin Kluth) to back up the config to the array.  That config is then synchronized to another server using syncthing.  If I lose the cache I have the backup on the array.  If I lose the array then I have the docker still on the cache.  If you lose both the cache and array (the server is fried, burnt, or stolen) then you have the backup somewhere else.  You don't need to use syncthing (or the plugin), you can back up the cache to a flash drive.  Did I miss anything?

 Yes that makes! Thanks for clearing that up. So essentially I want to have a different backup method outside of duplicati that I can rely on to restore duplicati (along with the other docker apps) appdata config and then once that's been restored proceed with duplicati restoration. Thanks again for the help. I think I know how to move forward now!

[TimTheSettler]

On 2/10/2024 at 2:09 PM, Hotsammysliz said:

So essentially I want to have a different backup method outside of duplicati that I can rely on to restore duplicati (along with the other docker apps) appdata config and then once that's been restored proceed with duplicati restoration.

 

Yup!

[ConnerVT]

I have a keyring of flash drives.  Each is loaded with bootable software needed to recover each of the major devices (two Unraid servers, my firewall, a few PCs) I have at home.  Each has a tag on it, identifying to which it goes.  I also have a ISO files for each saved on the external USB HDs I swap out to keep an off-site backup copy.  (There's a few troubleshooting tools flash drives on the keyring as well).

 

An important part of the process is to test and make sure that the restoration of your backup actually works (and you know how to do it).  Too many folks just click on the "Make Recovery Key" button, burn a flash drive, then toss it in the desk drawer.  You need to try it out, otherwise it is just security theater.  Makes you feel good, but may not be helpful in a crisis.

[TimTheSettler]

12 hours ago, ConnerVT said:

I have a keyring of flash drives.

 

I'm imagining this and this

[ConnerVT]

The first picture isn't that far off from reality.  I used to have them scattered all over.  Untagged.  You are likely in a stressed situation when you need one of them, and many don't give you much info as to what is on the drive when you pop them into a running system and look at the contents.  Organization, labeling and a small inventory text file I keep really helps.

[Hotsammysliz]

On 2/15/2024 at 6:55 AM, ConnerVT said:

I have a keyring of flash drives.  Each is loaded with bootable software needed to recover each of the major devices (two Unraid servers, my firewall, a few PCs) I have at home.  Each has a tag on it, identifying to which it goes.  I also have a ISO files for each saved on the external USB HDs I swap out to keep an off-site backup copy.  (There's a few troubleshooting tools flash drives on the keyring as well).

 

An important part of the process is to test and make sure that the restoration of your backup actually works (and you know how to do it).  Too many folks just click on the "Make Recovery Key" button, burn a flash drive, then toss it in the desk drawer.  You need to try it out, otherwise it is just security theater.  Makes you feel good, but may not be helpful in a crisis.

I realize this is very late reply but I've been going through and testing my processes as well as completely rebuilding my offsite backup server so things are moving albeit slowly. But I was curious specifically about what you have loaded on your flash drives as bootable software. I'd love to just hear what your specific process looks like just for more ideas. Thanks for the help!

[ConnerVT]

I don't have as many as I did years ago, when every computer needed files/drivers/etc from many different sources.  Right now (off the top of my head as I am not at home) I believe it is:

• Windows Recovery USBs (daily driver desktop and one made from a recently required laptop)
• Linux Mint - Live ISO and installer
• Hiren's BootCD - Tons of freeware utilities
• Macrium Reflect Rescue bootable USB - I use Macrium to back up my Windows machines to my Unraid server.  I also keep one in the box with my off-site external USB HD (which I swap out with the one attached to my server - contains backups of the Windows machines).
• OPNsense installer - for my router/firewall
• Configuration files - Lives in the USB slot of my server.  A script copies the saved configurations from things such as my firewall, managed switches, wireless AP, Unraid flash drives and diagnostic files.