Looking into unRAID and new server build

[rmilyard]

I have used FreeNAS for few years now but thinking about switching over to unRAID.  I have few reasons but 2 are the fact can add to array and docker support.  Just seems better for running media programs.  My current FreeNAS server is running on 11.1-U1.

 

CPU: Intel(R) Xeon(R) CPU E31275 @ 3.40GHz

MEMORY: 32gb DDR2 ECC

STORAGE: (12) 3tb Hard Drives, (4) 2tb Hard Drives, (2) 120gb SSDs (jails mirrored)

CASE: Silverstone Tek 3U 16-Bay 3.5-Inch Hot-Swap Rackmount Storage Server Chassis Cases RM316

PSU: FSP Group 700W PMBus V1.2 ATX Power Supply Single 2U Size 80 Plus Platinum Certified for Rack Mount Case (FSP700-80WEPB)

USB Boot: (2) SanDisk Ultra Fix 32gb in mirrored config

 

I use server for all our Home Media storage: Movies, TV Shows, Music and Photos.  I don't normally storage data per se on this server.  I do have QNAP NAS I use for that most of the time.  The server runs in jail on the SSDs Plex Server, Plexpy, Sabnzbd, Transmission, Sickrage and CouchPotato to make server pretty automated for house as I can.  I also have a HDHOMERUN Extended on network show can watch/record some Live TV.  As of late with more and more 4K content showing up been having some issues with playback on some devices.  Our 2 main clients are Shield TVs.  The wife and kids so like to access Plex outside the house which is one reason I switched from Kodi to Plex last year.

 

So I am looking for some hardware advice for new server thinking about.  Also what would be BEST way to setup unRAID on this.  I was thinking 2 Parity drives and using SSD or 2 for Dockers etc.

 

Looking at moving to 2011 Socket motherboard with (2) E5-2670 CPUs and 128gb DDR3 ECC ram. But only to all suggestions here.

I would reuse all drives, case and PSU.  Thinking maybe getting (2) Samsung 950 EVO 500gb SSDs for this.

 

Please chime in.  I am looking purchasing parts ASAP so really could use input.

 

[tdallen]

Hello -

 

The hardware requirements for unRAID are considerably less than Freenas.  Your E3-1275 with 32GB ECC RAM would actually be considered a fairly large unRAID server, suitable for basic NAS plus a number of Dockers.  It should also be capable of transcoding several 1080p Plex streams if needed.  It could easily run the applications you list above.

 

You can certainly go to a 2011 board and dual E5's with 128GB of RAM - but honestly that's a monster of a server for unRAID.  There's a use case for it and people are doing it, but they're generally into serious virtualization and I didn't see that in your requirements.  For basic NAS and Dockers that's extreme overkill at the expense of an oversize motherboard, high power consumption, heat, noise, etc.

 

4K video is a different topic, though.  While a pair of dual E5's can probably transcode a 4K stream or two, I think the future of 4K video isn't brute strength transcoding like 1080p was.  Rather, 4K video is't going to be about having lots of hardware, it's going to be about having the *right* hardware.  It is going to be about hardware assisted transcoding via the GPU or in the case of Kaby Lake and newer CPUs, their native support for H.265 HEVC.  The software support for all this is a new and evolving space though - there's no clear answers yet.

 

I'd give some thought to your transition strategy.  I know you said you used a QNAP for storage but there's a lot of drives in the Freenas server.  I'd go with dual parity if you are going to run them in unRAID, but I'm not clear on your approach to storage.  You're also going to want some time to setup all those Dockers, etc.  You can't reuse the case and drives, etc. without taking apart the old system  - and it seems like you'd want more of a transition strategy?

 

 

[rmilyard]

The plan was to tear down the FreeNAS server and reuse what I can.  However maybe hardware then I should just reuse it for unRAID build.  We have a pretty large family here so storage space is needed for Movies and TV shows which is why all the drives.  I have a 4K projector Home Theater with Dolby Atmos 7.2.4.  So a lot of our movies tend to be Blu Ray sized and now more 4K so need a lot of space for files.  

 

I was planning on replacing the 120gb SSDs with Samsung EVO 950 500gb anyways so maybe I should get those.  Maybe the best thing would be update some parts in current server ie SSDs and switch to unRAID?  I would like to save $$.  I could even maybe replace the 2TB drives with 3TB to get more storage.

 

Knowing these and if just get new SSDs and replace few drives what would be best way to setup?

 

All the spinning drives in one array with 2 parity drives?  How do I force Dockers etc to be on SSD?  Should one 500gb be Dockers etc and other for cache? Should I keep the other (2) 120gb in system and maybe use the 500gb and (2) 120gb for cache?

 

 

[tdallen]

I think your current server is a good starting point for unRAID.  Rather than a big bang upgrade I'd look to be focused/targeted on future needs like 4K.

 

The cache drive/pool in unRAID serves two main purposes.  First, it is the defacto application drive for unRAID and therefore the place where Dockers are installed by default.  Second, you can optionally do high speed caching of writes to the storage array (a process called mover later moves the files to the parity protected array).  Virtually everyone running unRAID 6+ uses the cache drive for Dockers, while a smaller subset turn on write-caching.  There are two components to Docker storage - where the Docker image is located and where Docker data goes.  Placing them in a "user share" with "Use Cache" set to "Only" will keep them on the cache drive.

 

The cache drive/pool needs to be sized for what you'll do with it.  120GB is small but adequate to start if you only plan to use it with Dockers.  250GB or 500GB gives you a lot more room to work with.  If you only plan to install one cache drive you can use XFS as the file system, but if you want redundancy you should use BTRFS so you can configure a cache pool - BTRFS RAID-1 is the default.  Cache pools get awkward with mixed size drives, though.

 

Finally, be aware of the heavy I/O associated with downloader programs.  SSDs have a maximum life expressed in TBW - terabytes written.  Unpacking downloaded files churns through the TBW and while SSDs make excellent cache drives you may want to invest in a Pro model using MLC rather than the TLC flash memory in consumer models.

 

If you buy new drives 8TB is definitely the sweet spot (people are shucking WD Easystores for very reasonable $$) - but keep in mind that your parity drives need to be as large as, or larger than any of your data drives.

 

An unRAID server only has one array (unlike Freenas), so yes - one or two parity drives and the rest data drives.  Dual parity starts to make sense as you increase the number of drives, it can protect you from two simultaneous drive failures.

 

Lastly, remember that your Freenas drives can't be read or imported directly into unRAID.  unRAID formats drives as they are added to the array.  Therefore, make sure everything is backed up before you start and have a transition strategy.

Edited February 13, 2018 by tdallen

[statecowboy]

If you're looking for build advice, you may have a look at this subreddit also.  I personally went with the E5-2680v2 build.  I have 64 GB Ram and I run VMs.  I have 2 SSDs in my unraid server.  One serves cache needs (speeds up downloading/processing for PVR programs), and the other is used to store and run my dockers and VMs.

 

 

 

[Hoopster]

@rmilyard tdallen has given you some excellent advice as he always does.  I concur with his opinion that for what you want to do in unRAID, there is no need to go to a dual-Xeon 2670 setup unless you want to run a lot of VMs and consolidate several desktop machines onto one server. Your current hardware is more than sufficient for running an unRAID NAS, several dockers, Plex transcoding for several 1080p streams and even a couple of lightweight VMs. 

 

Since your current CPU has integrated graphics, it can be used for hardware transcoding in Plex which greatly reduces the load on the CPU as it utilized Quick Sync Video support in the iGPU.  You don't say which version it is (v3, V4, v5, etc.) but all the most recent versions support 1080p without problem.  If you want 4K HEVC 10-bit transcoding you really want the v6 (Kaby Lake).

 

See the specs of my main server in my sig.  It is similar to yours.  I run 10 dockers, 2 VMs and Plex hardware transcoding on the server.  I also record OTA TV through HDHomeRun/Plex onto the array.

 

I have shucked several of the 8TB Easystore WD Red/White drives from their enclosures.  I personally prefer fewer large drives in my array over more smaller capacity drives.  The 8TB Easy stores at Best Buy have gone back up to $200, but, they very frequently are on sale for $150 (around Christmas) or $160 (as recently as last week).  If you want to spend some money now, do it on adding storage capacity to your array and get started with unRAID on your current hardware.  You can assess additional needs later, but, what you have now is more than enough for what you currently wish to do with unRAID

[rmilyard]

15 minutes ago, Hoopster said:

@rmilyard tdallen has given you some excellent advice as he always does.  I concur with his opinion that for what you want to do in unRAID, there is no need to go to a dual-Xeon 2670 setup unless you want to run a lot of VMs and consolidate several desktop machines onto one server. Your current hardware is more than sufficient for running an unRAID NAS, several dockers, Plex transcoding for several 1080p streams and even a couple of lightweight VMs. 

 

Since your current CPU has integrated graphics, it can be used for hardware transcoding in Plex which greatly reduces the load on the CPU as it utilized Quick Sync Video support in the iGPU.  You don't say which version it is (v3, V4, v5, etc.) but all the most recent versions support 1080p without problem.  If you want 4K HEVC 10-bit transcoding you really want the v6 (Kaby Lake).

 

See the specs of my main server in my sig.  It is similar to yours.  I run 10 dockers, 2 VMs and Plex hardware transcoding on the server.  I also record OTA TV through HDHomeRun/Plex onto the array.

 

I have shucked several of the 8TB Easystore WD Red/White drives from their enclosures.  I personally prefer fewer large drives in my array over more smaller capacity drives.  The 8TB Easy stores at Best Buy have gone back up to $200, but, they very frequently are on sale for $150 (around Christmas) or $160 (as recently as last week).  If you want to spend some money now, do it on adding storage capacity to your array and get started with unRAID on your current hardware.  You can assess additional needs later, but, what you have now is more than enough for what you currently wish to do with unRAID

 

I believe the CPU is version 1.  I have an Intel S1200BTL motherboard.

[Hoopster]

On 2/13/2018 at 12:50 PM, rmilyard said:

 

I believe the CPU is version 1.  I have an Intel S1200BTL motherboard.

 

OK, it appears that is a socket 1155 motherboard and it supports v1 and v2 E3-1275 Xeon CPUs.  Even the v1 supports QSV, however, it does not meet the officially-stated minimum requirements (v2 - Sandy Bridge) for Plex hardware-accelerated transcoding which are:

Linux system requirements

Hardware-Accelerated Streaming on Linux requires:

• 64-bit Ubuntu (16.04 or later) or 64-bit Fedora (26 or later) distributions. (Other distributions may be capable, but are not officially supported.)
• A recent Intel CPU meeting these requirements:
  • 2nd-generation Intel Core (Sandy Bridge, 2011) or newer
  • Supports Intel Quick Sync Video (Not sure? Look up your processor)
• Plex Media Server 1.9.3 or later
• Plex Pass subscription

Nevertheless, if you have a Plex Pass, there is no harm in trying it. 

 

Even if you can't take advantage of hardware transcoding, your CPU has an 8348 passmark score which will get you 2-3 simultaneous 1080p Plex transocdes in addition to the required unRAID/NAS overhead.

 

By the way, user signatures do not display automatically, you need to enable them.  If you have not already done so, I suggest you enable other user sigs.  Most put their system specs in their signature which can be helpful to new users as you can see what hardware others are using for unRAID.

 

To enable sigs, click on the down arrow by your user name at the top of the page, choose Account Settings > Signature > and enable View Signatures

 

EDIT:  Both your motherboard and CPU support VT-x and VT-d, so, if you wish, you can run a VM or two to try it out.

Edited March 14, 2018 by Hoopster

[rmilyard]

Thanks for all the great advice guys!  So sigs was off so that’s fixed now. 

 

Anyways I would like to update server. It seems the motherboard and cpu I have are limiting. When playing direct play and maybe a stream or 2 to iPhones outside the house get issues. My test rig with i5 4590 handles it much better. Just little ITX build I use for testing stuff. 

 

So would going to Xeon E3-1275v6 (Kaby Lake?) help my preformace much?  I know would need invest $ since would need motherboard, cpu and ram.  Should I keep server grade and use ECC ram?  Not cheap in DDR4. 

 

Fogot to say I have 2 LSI SAS cards would move and dual port 10gbe card. 

[Hoopster]

35 minutes ago, rmilyard said:

Anyways I would like to update server. It seems the motherboard and cpu I have are limiting. When playing direct play and maybe a stream or 2 to iPhones outside the house get issues. My test rig with i5 4590 handles it much better. Just little ITX build I use for testing stuff. 

 

So would going to Xeon E3-1275v6 (Kaby Lake?) help my preformace much?  I know would need invest $ since would need motherboard, cpu and ram.  Should I keep server grade and use ECC ram?  Not cheap in DDR4.

 

Both of my servers are small Mini-ITX-based systems.  It looks like we have similar CPUs.  I have the E3-1245 V5 in my main server and an i5 4590 in the backup.  The backup CPU used to be my main server CPU and it handled remote Plex transcodes just fine.

 

The E3-1245 V6 will get you 4K 10-bit HEVC support (my v5 only does 4K 8-bit in hardware).  Your Plex/streaming/transcode performance will definitely be better with the E3 V6.  For basic NAS functions and most dockers you won't see a difference as they are not CPU intensive.

 

As you have noted, DDR4 RAM is terribly expensive right now.  16GB should get you started.  Fortunately I got my 32 GB ECC DDR4 in 2016 for $178; now it would run me more than $400.  ECC is not a requirement for unRAID.  My backup server has non-ECC RAM.  I have ECC in the main server because it is a server motherboard, it is on 24x7 and I have adopted the "better safe than sorry" philosphy.  Many here will tell you ECC is not needed, and, technically, it is not.  It is just my preference in systems that support it.  unRAID does not have the ECC RAM and capacity requirements that FreeNAS recommends.  You choose the amount of RAM you need based on how you intend to use the server, not based on size of storage array.

 

On the server motherboard side, if you want ECC RAM look at Supermicro and ASRock Rack offerings.  There are many in Mini-ITX, micro-ATX and full ATX boards that support socket 1151 (Skylake - v5 and Kaby Lake - V6) Xeons and series 200 chipsets.  Of course, you'll want micro-ATX or ATX since you need PCIe slots for your controllers.

Edited February 13, 2018 by Hoopster

[rmilyard]

Well I ended up buying the e3-1275 v5 and Asus server motherboard. Also 32gb ecc ram and 2 Samsung evo 850 500gb SSDs. I wanted to rebuild server always so went ahead and pulled the trigger. 

 

So copying data off the FreeNAS server now. Will be few days. Ouch. So when parts come should be ready to start setting up unRAID. 

 

So with these new SSD drives I am guessing I should use one for cache and other for dockers?  If so is there a guide for setting up dockers for SSD?  I am not sure how you setup unRAID to know where stuff should be. Also thought read the appdata should be on SSD?  If so if it dies then would lose dockers and configs?  Guessing way to back that up. 

 

Also forgot I have an PCIE to M.2 Adapter with Samsung EVO 850 500gb NVMe think will add to server.  If did what should this be?  Dockers?  Cache?

Edited February 14, 2018 by rmilyard
Added more info

[tdallen]

You can create a cache pool with redundancy if you want using BTRFS - RAID-1 is the default.  Or, you can mount one individually via the Unassigned Devices plugin.  You'll install Dockers via Community Applications , which also provides the ability to backup appdata on a regular basis.

[rmilyard]

1 minute ago, tdallen said:

You can create a cache pool with redundancy if you want using BTRFS - RAID-1 is the default.  Or, you can mount one individually via the Unassigned Devices plugin.  You'll install Dockers via Community Applications , which also provides the ability to backup appdata on a regular basis.

 

Thanks. I have been messing if CA. Great stuff!  Just haven’t been able to figure out how make sure dockers, appdata and docker.img live in SSD. 

[wayner]

On 2/13/2018 at 9:26 AM, tdallen said:

4K video is a different topic, though.  While a pair of dual E5's can probably transcode a 4K stream or two, I think the future of 4K video isn't brute strength transcoding like 1080p was.  Rather, 4K video is't going to be about having lots of hardware, it's going to be about having the *right* hardware.  It is going to be about hardware assisted transcoding via the GPU or in the case of Kaby Lake and newer CPUs, their native support for H.265 HEVC.  The software support for all this is a new and evolving space though - there's no clear answers yet.

If you were to buy a new CPU today with a thought towards being able to transcode 4K content then what CPU would you recommend?  A Coffee Lake CPU like the i7-8700 or i7-8700K?  A Skylake 7800X?  Or does Skylake have the GPU capabilities?

[tdallen]

2 hours ago, rmilyard said:

Just haven’t been able to figure out how make sure dockers, appdata and docker.img live in SSD. 

In unRAID the settings for each user share determine where it is stored.  The setting Use Cache: Only or Prefer will make sure the data in the share stays on the cache drive.  Turn on Help in the UI for more information.

 

2 hours ago, wayner said:

If you were to buy a new CPU today with a thought towards being able to transcode 4K content then what CPU would you recommend?  A Coffee Lake CPU like the i7-8700 or i7-8700K?  A Skylake 7800X?  Or does Skylake have the GPU capabilities?

Personally I would go Kaby Lake or Coffee Lake in that situation, probably Coffee Lake.  But the reason I’m waiting on the sidelines is that I’d either like Coffee Lake with ECC or Skylake X with HEVC support...  The good news is that AMD is making Intel push their product plans faster but the bad news is that Intel’s products now seem rushed...  

 

[wayner]

Thanks - I wasn't sure if Skylake had HEVC support and from what  you are saying it does not.

[Hoopster]

1 hour ago, wayner said:

Thanks - I wasn't sure if Skylake had HEVC support and from what  you are saying it does not.

 

There are Skylake, Skylake X, and Skylake SP CPUs

 

Just plain Skylake (like my Xeon E3-1245 v5) has 8-bit HEVC support as do the desktop chips (i7/i5/i3, etc.) with integrated graphics/QSV support.  However, they have only partial 10-bit HEVC support.  For full 10-bit HEVC support you need Kaby Lake (desktop or  E3-12x5 v6 Xeons) or Coffee Lake (desktop only, no Xeons)

 

Skylake X (i9 extreme desktop chips) and Skylake SP (Xeon server chips) both lack integrated graphics/QSV support and take the brute force approach to HEVC.  In other words, they are not nearly as efficient or as fast in encoding/decoding 10-bit HEVC as the Kaby Lake or Coffee Lake chips that support QSV.

Edited February 14, 2018 by Hoopster

[wayner]

Thanks Hoopster.  Is the 10 bit support needed for HDR content?

[Hoopster]

2 hours ago, wayner said:

Thanks Hoopster.  Is the 10 bit support needed for HDR content?

 

Yes, HDR and Wide Color Gamut (WCG) support are part of the Main 10 HEVC profile.  It is generally agreed that the majority of 4K HEVC content will be delivered as 10-bit to include HDR/WCG

 

 

Edited February 14, 2018 by Hoopster

[rmilyard]

On ‎2‎/‎14‎/‎2018 at 5:37 AM, rmilyard said:

Well I ended up buying the e3-1275 v5 and Asus server motherboard. Also 32gb ecc ram and 2 Samsung evo 850 500gb SSDs. I wanted to rebuild server always so went ahead and pulled the trigger. 

 

So copying data off the FreeNAS server now. Will be few days. Ouch. So when parts come should be ready to start setting up unRAID. 

 

So with these new SSD drives I am guessing I should use one for cache and other for dockers?  If so is there a guide for setting up dockers for SSD?  I am not sure how you setup unRAID to know where stuff should be. Also thought read the appdata should be on SSD?  If so if it dies then would lose dockers and configs?  Guessing way to back that up. 

 

Also forgot I have an PCIE to M.2 Adapter with Samsung EVO 850 500gb NVMe think will add to server.  If did what should this be?  Dockers?  Cache?

 

All parts should be here tomorrow sometime.  Supemicro X11SAT-F-O, E3-1275v6, 32gb DDR4 ECC 2400, Samsung 860 EVO 500gb NVMe, (2) Samsung 860 EVO 500gb SSDs.  

 

I have (12) 3tb drives 3tb drives and (4) 2tb drives I can use in system.  So need to figure out best way I should setup.  I guess use 2 of the 3tb for parity drives and all the rest in the array.  Just not sure how should use the rest of these new SSDs.  Also pulling from old server I have (2) PNY 120gb SSDs I could use or just try to sell them.

[rmilyard]

Well all parts are here. Can’t put server in case since still copying files off the FreeNAS. Guessing will be done tomorrow. 

 

However see above post. How do you guys suggest I setup unraid with all these drives?

[hawihoney]

On 14.2.2018 at 1:37 PM, rmilyard said:

Also forgot I have an PCIE to M.2 Adapter with Samsung EVO 850 500gb NVMe think will add to server.

 

What type of adapter is this and does it work in your build?

 

Thanks.

 

[devros]

On 2/16/2018 at 11:56 AM, rmilyard said:

 

All parts should be here tomorrow sometime.  Supemicro X11SAT-F-O, E3-1275v6, 32gb DDR4 ECC 2400, Samsung 860 EVO 500gb NVMe, (2) Samsung 860 EVO 500gb SSDs.  

 

I have (12) 3tb drives 3tb drives and (4) 2tb drives I can use in system.  So need to figure out best way I should setup.  I guess use 2 of the 3tb for parity drives and all the rest in the array.  Just not sure how should use the rest of these new SSDs.  Also pulling from old server I have (2) PNY 120gb SSDs I could use or just try to sell them.

I see you went with the X11SAT-F and the E3-1275v6.  I have the same motherboard and the E3-1245v6 chip.  Were you ever able to get hardware transcoding to work with plex?  I don't seem to have a /dev/dri/renderD128 and I can't find anything obvious in the BIOS to enable it.

[rmilyard]

1 hour ago, devros said:

I see you went with the X11SAT-F and the E3-1275v6.  I have the same motherboard and the E3-1245v6 chip.  Were you ever able to get hardware transcoding to work with plex?  I don't seem to have a /dev/dri/renderD128 and I can't find anything obvious in the BIOS to enable it.

 

Yes I did get it working with some help.

 

See post:

 

 

[devros]

Thanks, I changed the primary video to iGFX in the BIOS and it works now!