Migration from FreeNAS -- Hardware config and Migration Strategy

[doctor15]

Wasn't sure if I should post this in the hardware sub-forum or general since its a bit of both..  Anyways I have a bunch of noobie questions.

 

I currently run a media server on FreeNAS and have finally given in that it is just not the best product for home/media use

 

For hardware, I have a Dell T20 w/ Pentium G3220 and 8gb ECC RAM.

HDs are:

  1x60gb SSD - used for jails, plugged into 2 port PCI-X controller card

  2x3TB in Raid-Z1 (ZFS Mirrored) - used for media, plugged into motherboard

  2x1TB in Raid-Z1 (ZFS Mirrored) used for documents, plugged into motherboard

 

I also have a 3TB USB hd that I plan to use for the migration, then add to the unraid machine once complete (as internal)

 

Uses: SMB file sharing, MythTV Backend (using networked tuners), Torrent server, and random Ubuntu experiments.

 

My plan is to use unraid6 with KVM, so that I can use KVM for Mythbuntu and whatever random servers I want to play with, and Docker for torrents and other apps.

 

1)  Is my hardware is sufficient for KVM with such use cases?

 

2) I have read the wiki, but don't see much detail on idea storage configurations. My understanding so far is I would be best creating 1 single array for documents and media?  Are there really no issues mixing my 3TB and 1TB drives?

 

3) Is there a way to use the SSD for the KVM boxes and Docker applications?  Do I make it a separate drive or part of the array?

 

4) Will having drives split between the motherboard and controller card cause any issues?

 

5) Do people find the cache drive option very beneficial or essential?  I was thinking of skipping that.

 

6) How would you suggest I do the data  migration?  I have two USB sticks so I don't need to do all the drives at the same time.  I was thinking I would initially unplug the 1TB drives and start with just the 3TB.  Should I start with one of the 3TB drives as parity, or both as data drives, then add the 3rd party drive once I'm done migrating?  Keep in mind I need to the 3rd drive until I'm done migrating from FreeNAS.  I'm also probably going to stick with 3x3TB for a short time so that I can try out the free version and make sure I'm satisfied before mixing in the 1xTBs.

 

7) Any tips or complications on migrating the MythTV backend?  It is currently running in a FreeBSD jail on the SSD. It should be as simple as backup the database while running freenas, then restoring once I have Mythbuntu up on KVM.  I'm concerned though because I currently have it on the SSD, and want to run mythbuntu off the SSD, so if I miss anything I'll have to restore from backup to access the original FreeBSD instance.

 

Thanks for reading this long thread!  I'm looking forward to finally getting away from FreeBSD and am excited about all the features of Unraid.

[trurl]

Wasn't sure if I should post this in the hardware sub-forum or general since its a bit of both..  Anyways I have a bunch of noobie questions.

 

I currently run a media server on FreeNAS and have finally given in that it is just not the best product for home/media use

 

For hardware, I have a Dell T20 w/ Pentium G3220 and 8gb ECC RAM.

HDs are:

  1x60gb SSD - used for jails, plugged into 2 port PCI-X controller card

  2x3TB in Raid-Z1 (ZFS Mirrored) - used for media, plugged into motherboard

  2x1TB in Raid-Z1 (ZFS Mirrored) used for documents, plugged into motherboard

 

I also have a 3TB USB hd that I plan to use for the migration, then add to the unraid machine once complete (as internal)

 

Uses: SMB file sharing, MythTV Backend (using networked tuners), Torrent server, and random Ubuntu experiments.

 

My plan is to use unraid6 with KVM, so that I can use KVM for Mythbuntu and whatever random servers I want to play with, and Docker for torrents and other apps.

 

1)  Is my hardware is sufficient for KVM with such use cases?

Your processor does not support VT-d which will limit your ability to directly access hardware from a VM. Not necessarily a concern depending on your use case.

2) I have read the wiki, but don't see much detail on idea storage configurations. My understanding so far is I would be best creating 1 single array for documents and media?  Are there really no issues mixing my 3TB and 1TB drives?

Mixing drive sizes is one of the greatest feature of unRAID.

3) Is there a way to use the SSD for the KVM boxes and Docker applications?  Do I make it a separate drive or part of the array?

Many people use SSDs as the cache drive, and put their apps, dockers, and VMs there. This allows the array drives to spin down. If you make it separate from the array or cache, then unRAID will not manage it for you.

 

4) Will having drives split between the motherboard and controller card cause any issues?

Not as long as the hardware is supported. Depending on the hardware there may be some performance differences.

 

5) Do people find the cache drive option very beneficial or essential?  I was thinking of skipping that.

I don't use the cache drive for caching, but do use it for dockers and VMs as mentioned in my answer to 3

 

6) How would you suggest I do the data  migration?  I have two USB sticks so I don't need to do all the drives at the same time.  I was thinking I would initially unplug the 1TB drives and start with just the 3TB.  Should I start with one of the 3TB drives as parity, or both as data drives, then add the 3rd party drive once I'm done migrating?  Keep in mind I need to the 3rd drive until I'm done migrating from FreeNAS.  I'm also probably going to stick with 3x3TB for a short time so that I can try out the free version and make sure I'm satisfied before mixing in the 1xTBs.

Some people advocate waiting until you get the initial data load done before adding parity. Parity will slow down writing to the array. Compared to preclearing drives though it is not really a big time suck. You should consider whether you have backups of the data. In fact you should have a backup strategy in any case.

 

7) Any tips or complications on migrating the MythTV backend?  It is currently running in a FreeBSD jail on the SSD. It should be as simple as backup the database while running freenas, then restoring once I have Mythbuntu up on KVM.  I'm concerned though because I currently have it on the SSD, and want to run mythbuntu off the SSD, so if I miss anything I'll have to restore from backup to access the original FreeBSD instance.

 

Thanks for reading this long thread!  I'm looking forward to finally getting away from FreeBSD and am excited about all the features of Unraid.

I have setup a mythbuntu VM that I plan to use with HDHomerun but have not really had time to do anything with it yet. dmacias (the author of the Virtual Machine Management plugin) has been using mythbuntu I believe.

[doctor15]

Awesome, this is what I was hoping to hear!

 

Your processor does not support VT-d which will limit your ability to directly access hardware from a VM. Not necessarily a concern depending on your use case.

 

Good catch.  I only need access to storage, so I'm thinking this should be fine.  I don't need to pass though my storage controller, do I? What are the common use cases I might need VT-d for?

 

Many people use SSDs as the cache drive, and put their apps, dockers, and VMs there. This allows the array drives to spin down. If you make it separate from the array or cache, then unRAID will not manage it for you.

 

I'm a bit confused by this.  So I tell unraid this is a cache drive, but don't use it for caching?  How do I specify what gets stored there?

 

Some people advocate waiting until you get the initial data load done before adding parity. Parity will slow down writing to the array. Compared to preclearing drives though it is not really a big time suck.

 

Sounds good.  What is preclearing?  Is there a good introductory guide that I should check out?  The user manual didn't have much info.

[trurl]

Awesome, this is what I was hoping to hear!

 

Your processor does not support VT-d which will limit your ability to directly access hardware from a VM. Not necessarily a concern depending on your use case.

 

Good catch.  I only need access to storage, so I'm thinking this should be fine.  I don't need to pass though my storage controller, do I? What are the common use cases I might need VT-d for?

Video output is the big one. Accessing unRAID storage from a VM is not a problem.

 

Many people use SSDs as the cache drive, and put their apps, dockers, and VMs there. This allows the array drives to spin down. If you make it separate from the array or cache, then unRAID will not manage it for you.

 

I'm a bit confused by this.  So I tell unraid this is a cache drive, but don't use it for caching?  How do I specify what gets stored there?

You create a user share to put this stuff on and tell unRAID it will only use the cache drive.

 

Some people advocate waiting until you get the initial data load done before adding parity. Parity will slow down writing to the array. Compared to preclearing drives though it is not really a big time suck.

 

Sounds good.  What is preclearing?  Is there a good introductory guide that I should check out?  The user manual didn't have much info.

Preclear script

 

When you add a new drive to a parity protected array, it must be zeroed in order to match existing parity. unRAID will do this for you but it will be offline until it finishes. The preclear script will allow you to zero the drive before you assign it to the array, so it already matches parity and unRAID doesn't have to zero it. Preclear also tests the drive to make sure it can all be reliably written and read. All the reading and writing done by preclear will take many hours (about 10 hours per TB on my system), but the system will not be offline unlike if you make unRAID zero the drive. Many people preclear more than once just for the additional testing. Making sure your drives are reliable is important because unRAID can only rebuild a disk if parity and all other drives are good.

[Brucey7]

I am interested in why you were thinking of moving from FreeNAS to unRAID?  I am thinking of exploring doing the opposite.  Without a clear path to dual parity, I'm getting very concerned about the future of unRAID.

 

The single biggest benefit of FreeNAS is ZFS, not only is dual & triple parity available, the file system is self-healing.  I have run it for many years on Thecus boxes without a hiccup and it's lightning quick.

[doctor15]

I am interested in why you were thinking of moving from FreeNAS to unRAID?

 

FreeNAS is great if you are looking for a simple NAS appliance and are obsessed with mitigating the risk of data failure.  IMO, it is geared much more towards small business and "protect my data at all costs" type market, at the expense of user friendly customization and budget friendliness.  For example, you can't plug in a USB drive to back up to, because "you should be backing up to another NAS".  The user community also reflects this market and isnt as willing to help out with typical home environment situations.  Also, keep in mind expanding in array is a bit of a problem, so you need to plan in advance exactly what storage size you will need.

 

I'm looking for an all in one media server that can store my media with some sort of fail safe, and allow me to run whatever arbitrary services I want to experiment with.  Doing anything like this on FreeNAS is difficult, because visualization of FreeNAS itself is strongly discouraged so all services must be in a FreeBSD jail.  While technically this allows many options.. I found it very difficult to support or find answers when I had problems.  I did get MythTV running stable but every other service was riddled with problems, and even standard "supported" plugins have many unexplained issues.

 

While Unriad does not have as great of data protection as FreeNAS, it is made to be a media box and the user community and features reflect that.  Working in a Linux environment and having KVM I found it much easier to get everything working, and like that I can expand the array at any time.  My critical data is backed up to the cloud, and while I would like to do my best to protect my media.. if I have 2 hard drives fail and lose some of the data, I'll live to see another day.

[doctor15]

Just reporting back.. I'm mostly up and running on Unraid.  I found it a bit confusing to get started.. as there is lots of important information spread among the forums missing from the documentation, but once I got going I'm finding everything relatively straight forward.  I'm particularly impressed with how simple it was to get Mythbuntu setup and running using the KVM plugin.

 

 

Many people use SSDs as the cache drive, and put their apps, dockers, and VMs there. This allows the array drives to spin down. If you make it separate from the array or cache, then unRAID will not manage it for you.

 

Some clarification on this again.. What is the advantage of using the cache drive over keeping my SSD outside of the array?  Right now I have an XFS formatted cache drive.. but the more I read I wonder if I should have a BTRFS formatted drive outside of the array using SNAP.

[limetech]

doctor15 - some very interesting insights!  Thank you for posting.  Maybe I can provide some clarification...

 

In unRaid there are three basic storage organizations:

 

1. The unRaid array - this is a set of individual devices protected by a single block-level parity device.  The non-parity devices are called Data devices or Data disks.  Data devices have their own file system and may be individually exported on the network, e.g, as a share called "disk1" or "disk2".  They also can be individually spun up or down, and as long as no devices are "disabled" then only those devices which are being accessed need to be spun up.

 

2. The cache pool - this is one or more devices organized as a btrfs "raid1" pool.  There's lots of information out there on btrfs vs. zfs.  No doubt zfs is a more mature file system, but the linux community appears highly motivated (especially lately) to make this file system absolutely robust, and most would say it's destined to be the file system of choice for linux moving forward.

 

Like data disks, the cache disk (single device pool) or cache pool can be exported on the network.  At this time we export "all or nothing" but there are plans to let you create subvolumes and export those individually as well.

 

The cache disk/pool also supports a unique feature: we are able to "cache" creation of new objects there, and then later move them off cache storage and onto the array.  The main purpose for doing this is to speed up write performance when you need it: at the time new files are being written to the server.

 

3. Ad hoc devices - these are devices not in the array or pool.  Sometimes they are referred to as "snap" devices (shared non-array partition).  Officially we don't support the use of snap devices but people do make use of them.  Eventually we will formalize this storage type though, especially for use by virtual machines.