Looking for some advice, and critique, on my plans detailed below for a NAS appliance VM to be run alongside unRAID to store important data on. Have recently lost a large number of important photos, and need to re-evaluate my backup strategy. (See below for current setup)

 

Apologies for the wall of text, its broken into section so hopefully easier to follow. tl;dr at the end.

 

Background / Current setup:

 

Have been using unRAID since 4.x, with the main goal of storing media (shows, movies, music etc).

Currently running 5.0.5 as a guest under ESXI on a Microserver, with additional VMs as follows,

- debian VM running sickbeard/sabnzbd/logitechmediaserver etc.

- Win7 for some windows only services (ComicRack, Ubiquity server, Doxie scanner receiver)

- various test or dev VMs

 

unRAID is currently only storing replaceable media (bluray collection, music collection, acquired shows etc), serving to openelec clients, squeezeboxes etc.

All important documents, records, photos are currently stored on my desktop PC, and backed up via Crashplan online.

 

The Catalyst:

A few months ago I suffered corruption in a significant quantity of photos. Important photos (new baby, emotional attachment, usual story).

 

Not sure what the root cause was, but unfortunately due to the large size of photo's being dumped on my PC (as expected with a new baby), coupled with a limited upload rate, it appears that the backup of those photos to Crashplan had yet to happen prior to the corruption. So unfortunately the corrupted photos where the ones backed up and the issue was only noticed later.

 

There is plenty of things I could've done better (such as use Crashplan to backup immediately to the unRAID box for starters - or use SyncBack to my unRAID box) - however I want to set something up properly this time to ensure it will never happen again.

 

 

The Plan:

I've outgrown the Microserver in any case, so I intend to start a fresh with proper hardware - this has already been purchased (see specs below).

 

I intend to move all my important data off my desktop (and other users laptops), and onto the NAS.

Access to our documents will then be over network shares etc. I'm happy with not keeping local copies of documents on our PC's as they will generally always be used within the home network.

 

I am not entirely comfortable with utilising unRAID itself for the storage of this data. It's fine for my media, but I feel there are too many shortcoming for my critical data.

I also want to leverage on fs snapshot capabilities so that family members can recover previous document versions without having to resort to digging through Crashplan archives.

 

Therefore the plan is to utilise a VM, and pass through two harddrives mirrored, to store data, ensure integrity, and automatically backup offsite (Crashplan)

 

Requirements:

 

NAS Appliance:

- Protection against silent corruption

- Online-scrubbing

- Snapshots (passed through cifs to Windows for "Previous Versions" capability)

- Fairly easy samba set up, multi user etc.

- Ability to run Crashplan, BTSync

 

Other VMs:

- Sickbeard, sabnzbd, logitechmedia server, etc

- linux dev box

- tftp server

- Win7 services

- Win7 sandbox (will RDP into)

 

Hardware:

 

- Limetech D-316M case (whilst the HDD qty is a bit low compared to a Norco case, the qty is outstanding)

- Xeon E3-1230v3, Supermicro X10SL7-F, 32GB ECC memory, M1015 card

- HDD (NAS): passthru Onboard LSI-2308, flashed IT : 2 x 4TB WD Red

- HDD (unRAID): passthru M1015 card : 2 x 4TB WD Green (initial setup, then increase with current Microserver HDDS of 2 x 4TB + 3TB)

- HDD (others): 1TB 7200rpm for VM datastores. 1TB 5400rpm (spare disk to be put to use somehow)

 

VM configuration:

Whilst I'm currently utilising ESXi and am comfortable with it, like others on here I'm concerned with its future direction. After some initial reading I'm planning to go with KVM.

 

I've not yet decided whether to run unRAID 6 as the host, with KVM for the other VMs - or use another distro such as CentOS and run unRAID as a guest. Appreciate peoples opinion here.

 

I probably won't initially utilise Docker (if unRAID is host), as I'll probably migrate the existing VM over, or at least copy my existing configuration. May play around with docker afterwards. (I'm concerned with the inability to auto-update the odd python apps - sickbeard etc - though I may have this completely wrong).

 

NAS appliance:

Whilst not technically unRAID specific, I figure the experience floating around this forum would probably lead me to the best solution for the decision of what to use as the NAS appliance.

 

The requirement for snapshots, crc checks etc likely leads to a ZFS or BTRFS filesystem. I've not used these filesystems in the past - and my bit of digging doesn't seem to favour ZFS due to its high RAM requirements (this may not be an issue in any case?).

 

Should I go for a stock standard, pre-made appliance such as FreeNAS - that has proven support for ZFS, snapshots etc.

Or would I be better to roll my own and use BTRFS?

What would be a recommended distro for something like this?

I'm not shy of getting my hands dirty in bash, but the preference would be a fairly simple to use GUI to a) quickly check the health of the system and b) painlessly set up file shares etc

 

Considering that I'll be using a separate VM for 'service'-type tasks, I'll want my NAS appliance to be very lean (no need for plug-ins etc) - the only additional applications running would be Crashplan (to backup these files), and BTSync to sync a few directories on local PC's (Adobe Lightroom database for example).

 

Appreciate any advice - even if it is to pick my entire plan to pieces.

 

tl;dr

 

Want a VM NAS appliance with ZFS/BTRFS capabilities to store important docs.

Options:

Host: unRAID 6 as host with KVM vs CentOS with KVM (unRAID as guest)?

NAS Appliance guest: FreeNAS vs roll your own with BTRFS?

Just a couple of thoughts ...

 

First, ANY cloud-based backup is going to be relatively slow -- you should never depend on it as the first level backup if you're adding data at a fairly high rate (e.g. daily baby pictures).  Cloud-based backups are great for "off site" backups, but you should also have another local backup of your data.

 

Second, do NOT consider UnRAID, or any other NAS, as a backup UNLESS it's truly that -- i.e. the data is also stored on another system.    It's a nice, fault-tolerant server to provide highly reliable access to your data ... but the data on your server still needs to be backed up.    This is true whether you use UnRAID or any other NAS solution.

 

As for how to host it -- personally, I prefer a dedicated server for UnRAID, with another box that runs all apps, including any VM's I want to run.    But with v6 it's reasonable to run apps in Docker containers on the UnRAID box, as the integrity of the basic NAS is far better than with prior versions.    As the hypervisor supports matures, it'll also be a reasonable platform for your VMs as well.

 

WHICH system you elect to use isn't as important as ensuring you fix your backup shortfall that resulted in your recent data loss.    Remember that ALL important data should always be stored in at LEAST 2 places ... never delete it from the source until after that is true (e.g. if you'd still had your lost photos on an SD card you'd have been okay).

 

Cheers for that - and fully appreciate those points.

I guess my first post focussed entirely on the technical aspects, and didnt highlight my backup strategy.

 

I intend to run Crashplan on the VM, and initially back it up to the unRAID box, this should provide at least a second copy within fairly quick timeframe. Understood it is on the same hardware, and won't protect against physical /loss etc, but at least the OS's are segregated and it will protect against an errant program (or user) causing damage to the files. This should be sufficient prior to the online Crashplan completing.

 

Have also implemented rolling SD cards for photos, so that a card will be only wiped once the online backup is complete.

 

 

I use unraid v6 with docker.

My desktops use Crashplan and backup to my unraid box... great for version control etc

My important data (some docs and photos mainly) are stored directly on the server and Crashplan(docker) also backs these up to a remote server.

One pc backs up up important docs to the remote crashplan server directly and to the unraid crashplan.

 

My TV/Videos are just on unraid, served to Plex (docker) and to xbmc

My music (mostly itunes) is on my local mac and this is Carbon Cloned onto the unraid. iTunes became sluggish if it was using a share.

 

If my server gets stolen, I will lose my films and music, not the end of the world, but my pictures are safe.

 

I looked at using an alternative server technology but having lost data in the past when I didnt fully understand what I was doing is a problem for me!