Suggestion for server grade build


Recommended Posts

Hi -

 

have been using/running unRAID for about 8 years. I started with a basic build (only upgrade I have done over the period is add more RAM and hard drives), please see attachment for details.

 

However, now with dockers and VMs running, I feel like its time to upgrade my machine with server grade components. LimeTech has a good server hardware recommendation in their wiki page, but feels like its outdated. I would like to build a system that will last few years without a dire need for an upgrade.

 

I would like to keep using my Cooler Master Centurion 590 case. But, these are my requirements:

  • run the server as a development machine, where I can deploy/test code, run CI/CD tools etc. Use Plex to stream/transcode MKV movies.
  • run 10 - 15 dockers, mostly MariaDB, Plex, custom docker images with programs running on JVM
  • run 1 - 2 Ubuntu server or desktop VM. Using device pass through with peripherals.
  • Xeon (is i7 enough?) processors with multiple cores/hyper threads. I am thinking more cores the better, because I will be running multiple docker containers.
  • ECC RAM, is 32 GB enough or do I need 64 GB? Do I really need ECC ram?
  • Motherboard that can support about 8 SATA drives, I don't want to add SATA controllers. Currently, I have 5 hard drives, and one SSD drive as cache. In the near future I see myself adding one more SSD to create a BTRFS pool.
  • I am not into gaming, therefore I probably don't need a fancy graphics card. But, a graphics card is needed to run Ubuntu Desktop VM as a primary development machine.
  • I would like the budget to be in between $1500 - $2000.

 

I am thinking, the budget should be good enough to get a decent system, including a new power supply. But, I have been wrong before. Could you please give me a list of hardware, similar to the LimeTech server recommendation, that is still current and compatible with unRAID.

 

Also, please do let me know if I am out of my mind, or if what I am thinking is not be possible.

 

Thanks,

Sagun

 

 

Screen Shot 2018-03-23 at 11.51.32 PM.png

Link to comment

On a Xeon E3 platform, I built a few machines using a Gigabyte X170-Ws-Ecc workstation board.

It had 8 Sata3 ports, Plenty of expansion slots and of course ecc memory.

Typically you would run a couple of 500gb ssd's for the cache, a 3tb red drive for parity, which left up to 5 more drives for the array.

They fit nicely into an Antec nine hundred two case, and you can add a 5 drive hotswap bay to this case for the array drives. (The parity lives inside the case)

I usually fit the SSD's in a 2.5" 2 drive hotswap bay as well.

Alternatively if you don't want to throw money around, everything will fit into a Coolermaster silencio 550 case.

(Although I still use the ssd hotswap bay's.)

 

Typically I might run a windows 10 VM so that the machine acts as a normal workstation.

In the background It acts as a file server and it might run a number of dockers such as the letsencrypt nginx docker, nextcloud servers and what ever else fits the bill.

It is preferable to use Xeon processors ending in a"5" eg E3-1245.  They have sound processors built in and simplify getting sound to a windows vm workstation.

The sound is delivered via the hdmi connector, So if you want to get the sound to a quality speaker system I use those HDMI audio extractor modules.  You can get them on ebay for not very much.  They have HDMI inputs, a hdmi output to the screen and separate audio outputs to the speakers.

 

For a VM needing graphics pass through I have been using asus radeon r5-230 video cards. They are cheap but do what you need for general computing.

I also added a PCIe USB card to the system and passed it through to the VM.

This allowed Hot plugging USB devices such as USB backup drives into the windows VM without the need for a restart.

 

Another machine I built ran 2 operating system VM's with separate keyboards mice and screens.  You need to use unidentical hardware so I used an ASUS gt-720 graphics card as the second device. The graphics on this device works fine but I had issues with sound being out of sync when playing video. You also need to use unidentical keyboards and mice unless they are on the passed through USB card.

 

The down side to this board is it has been discontinued.

 

It also had an issue where there is a multithreading bug in the XEON processors which would cause the system to lock up.

It was subtle. The machine might run for a couple of weeks then stop.

The fix was a firmware upgrade.  All machines run rock solid after the upgrade.

The downside to the upgrade from an Unraid perspective was that functionality was lost in a couple of the PCIE slots.

So if you needed the slots to run an extra PCIe card, you were out of luck.

 

So if you have read this far down, I'm not sure what the next board is either.

The above uses a C236 chipset.

I have a feeling that the C236 is end of life.

If it is, I'm not sure what else you could use a Xeon e3 on.

There is a C232 chipset, but they usually only have 6 Sata ports.  Not enough for a functional Unraid machine.

 

I'm currently looking at an ASUS P10 motherboard (forget the exact model, but it uses a C236 chipset and is a bit dearer than the X170 board.)

I don't want to go to E5 processors, because the cost blows out too much.

 

For my use, I prefer to look for the Xeon workstation boards rather than server boards.  If you are going to use the computer as a workstation, the features needed are on the board where a server board will probably be missing them.

 

I'd love to know who else is using e3 processors and their motherboard choice.

The ultimate board would be one which broke out into a separate IOMMU for each device and naturally plenty of devices to choose from.

This would allow you to map anything to any VM.

 

Memory...  Most of the machines I built ran fine with 16gb ram.  The exception being one which runs small business server as a vm and also a windows 10 VM and a number of dockers.  It runs 32gb ecc ram.

 

 

 

Edited by Jessie
more info
Link to comment

Thank you for taking the time to reply Jessie! I really appreciate it.

 

I like the Gigabyte motherboard you mentioned, but like you said, it has been discontinued. I really don't mind going for a E3 processor under $500, however, I don't want to build a system that might feel sluggish after couple of years.

 

I am not into custom builds (I usually prefer to buy computers from Apple, very little fuss). I am finding it very hard to coalesce a list of right components that will play along in harmony.

Link to comment

Hi,

 

I'm considering coming back to unRAID after using a Synology, and also searching for best possible setup. I want to use a 24 bay chassis. Xcase located in the UK has some nice chassis with expander and without expander backplane. No sure yet if I follow the expander backplane option. I'm also looked at a motherboard of Supermicro that are server graded boards. I'm considering the socket 1151 or the socket FCBGA 1167 platform.

 

Also using VM's, Docker, Plex, etc and Apple. I will follow your topic with interest!

 

 

 

 

Link to comment
17 hours ago, sakh1979 said:

I like the Gigabyte motherboard you mentioned, but like you said, it has been discontinued. I really don't mind going for a E3 processor under $500, however, I don't want to build a system that might feel sluggish after couple of years.

 

I am not into custom builds (I usually prefer to buy computers from Apple, very little fuss). I am finding it very hard to coalesce a list of right components that will play along in harmony.

The ASUS P10S WS looks like a good contender for an E3 processor.  The other parts I mentioned above work.

It would be nice to know if anyone has already used the board.

I would be concerned about whether the board can work around the xeon bug.

 

There were also issues with V6 processors in the x170 board.  Namely they will not boot until a firmware update is applied to the motherboard.

Not a problem except you have to insert a v5 processor temporarily onto the board so that it will boot to load the bios update.

Not sure whether this issue exists on the ASUS board.

 

If you go E5, the sky is the limit.  Sata ports usually start at 10. Higher numbers are common.

Unfortunately the sky is also the limit for the cost.  I doubt you would get anything useful for $2000. (At least not in Australia.)

Lower end E5's have a lower clock speed, so performance might be an issue even if you had plenty of threads.

 

 

Link to comment
1 hour ago, HellDiverUK said:

Consider a Ryzen or Ryzen Threadripper.  I've just moved over to Ryzen for my unRAID and I'm really quite happy, despite it only being a little R5 1600 (6 core, 12 thread at 3.2-3.6GHz). 

 

Did you do anything special in unRAID to get the Ryzen to work, or does it work out of the box? I was reluctant in considering Ryzen because of the fact its new, and the issues about unRAID freezing. But, from your experience sounds like its working fine.

 

Can you share the list of components you have used in your build?

Link to comment
1 hour ago, sakh1979 said:

 

Did you do anything special in unRAID to get the Ryzen to work, or does it work out of the box? I was reluctant in considering Ryzen because of the fact its new, and the issues about unRAID freezing. But, from your experience sounds like its working fine.

 

Can you share the list of components you have used in your build?

 

Initially I set C-states to Off in the BIOS, but that was before the addition of the Zenstates setting in the go file, which basically does the same thing.  It's the C6 power saving mode that gives the hangups in unRAID (and linux in general).

 

Most of the components are in my signature, but here is a full run-down:

 

Asus Strix X370-F Gaming motherboard,

Ryzen R5-1600 CPU with box cooler,

Corsair LPX-3200 16GB kit (2x8GB in white),

Matrox G690 8MB PCI-E video card (yes, 8MB!),

Fractal Define R5 case (white),

Corsair RM750 PSU,

Samsung 950 Pro 512GB NVMe SSD (cache),

1x ST4000DM000 Seagate 4TB Desktop HDD,

2x ST8000AS0002 Seagate 8TB Archive HDD,

1x ST1000NM0033 Seagate 1TB Constellation ES.3 HDD (for Emby DVR storage),

2x WD80EZZX Western Digital Red HDD (parity).

 

Cooling is via 2x140mm Corsair ML140 fans in the front, and 1x 140mm Noctua Redux fan in the rear, all running between 400-600rpm.

 

It idles around 71W at the wall, and with all disks spinning and CPU busy it uses around 135W.  More than my old i5 system, but it's much, much faster.

 

Edited by HellDiverUK
  • Upvote 1
Link to comment
1 hour ago, HellDiverUK said:

 

Initially I set C-states to Off in the BIOS, but that was before the addition of the Zenstates setting in the go file, which basically does the same thing.  It's the C6 power saving mode that gives the hangups in unRAID (and linux in general).

 

Most of the components are in my signature, but here is a full run-down:

 

Asus Strix X370-F Gaming motherboard,

Ryzen R5-1600 CPU with box cooler,

Corsair LPX-3200 16GB kit (2x8GB in white),

Matrox G690 8MB PCI-E video card (yes, 8MB!),

Fractal Define R5 case (white),

Corsair RM750 PSU,

Samsung 950 Pro 512GB NVMe SSD (cache),

1x ST4000DM000 Seagate 4TB Desktop HDD,

2x ST8000AS0002 Seagate 8TB Archive HDD,

1x ST1000NM0033 Seagate 1TB Constellation ES.3 HDD (for Emby DVR storage),

2x WD80EZZX Western Digital Red HDD (parity).

 

Cooling is via 2x140mm Corsair ML140 fans in the front, and 1x 140mm Noctua Redux fan in the rear, all running between 400-600rpm.

 

It idles around 71W at the wall, and with all disks spinning and CPU busy it uses around 135W.  More than my old i5 system, but it's much, much faster.

 

 

Thank you for the detailed list of components.

 

Is there is a reason you choose Ryzen R5 instead of R7? If I go the Ryzen R7 route will the Asus Strix X370-F motherboard handle it?

 

I like your PSU, I will need to upgrade mine as well. I'll probably end up using what you have.

Link to comment
30 minutes ago, sakh1979 said:

 

Thank you for the detailed list of components.

 

Is there is a reason you choose Ryzen R5 instead of R7? If I go the Ryzen R7 route will the Asus Strix X370-F motherboard handle it?

 

I like your PSU, I will need to upgrade mine as well. I'll probably end up using what you have.

 

The R5 was cheap, and 6 cores with 12 threads is plenty. R3 was cheaper, but only ones available at the time were the 4-core ones, which were slower than the i5 I had. 

 

The motherboard is great, I used it in a gaming rig for a while with a R7-1800X overclocked to 4GHz and it never skipped a beat.  I'd still be using it but I got a great deal on an i7-5960X that overclocks to 4.6GHz that I could plug in to my old X99 rig.

 

The RM750 is a nice unit, it runs semi-fanless, so up to about 400W the fan doesn't turn.  As the unRAID box only uses 130W peak, it's basically silent.  Again, it's left over from my gaming rig, which now houses an AX860i which I got when I was running a pair of GTX980Ti cards (they were a little too much for the RM750 along with the i7-6800K I had at the time).

Link to comment
17 hours ago, HellDiverUK said:

 

The R5 was cheap, and 6 cores with 12 threads is plenty. R3 was cheaper, but only ones available at the time were the 4-core ones, which were slower than the i5 I had. 

 

The motherboard is great, I used it in a gaming rig for a while with a R7-1800X overclocked to 4GHz and it never skipped a beat.  I'd still be using it but I got a great deal on an i7-5960X that overclocks to 4.6GHz that I could plug in to my old X99 rig.

 

The RM750 is a nice unit, it runs semi-fanless, so up to about 400W the fan doesn't turn.  As the unRAID box only uses 130W peak, it's basically silent.  Again, it's left over from my gaming rig, which now houses an AX860i which I got when I was running a pair of GTX980Ti cards (they were a little too much for the RM750 along with the i7-6800K I had at the time).

 

How difficult was it to setup a VM with peripheral attached (example monitor, keyboard, etc)?

 

I am thinking of upgrading to following components:

  • Ryzen 7 1700 (do you have a recommendation for an external GPU, a decent entry point GPU will do)
  • Asus Strix X370-F Gaming Motherboard (do you recommend another motherboard?)
  • Corsair LPX-3200 32GB (2x16GB in red probably)
  • Corsair RM750 PSU
Link to comment
11 hours ago, HellDiverUK said:

Correct, I'm not.  I've no need for anything other than the VNC connection to my VMs.

 

Understood. For some unexplained reason, I was being adamant on using IOMMU passthrough for a VM. Whereas, I can do exactly what I want with a remote connection to the VM (example using RDP or VNC).

 

Thanks for all your help!

Link to comment

I think, I have coalesced a list of components I want, can someone please verify they play in harmony with unRAID?

  • Intel Xeon E3-1245 V5 3.5GHz Quad-Core Processor
  • Supermicro MBD-X11SSM-F-O Micro ATX LGA1151 Motherboard
  • Kingston ValueRAM 32GB Kit (2x16GB) 2133MHz DDR4 ECC CL15 DIMM 2Rx8 Desktop Memory (KVR21E15D8K2/32)
  • Corsair - RM 750W 80+ Gold Certified Fully-Modular ATX Power Supply

Thank you!

Edited by sakh1979
Link to comment
1 hour ago, sakh1979 said:

I think, I have coalesced a list of components I want, can someone please verify they play in harmony with unRAID?

  • Intel Xeon E3-1245 V5 3.5GHz Quad-Core Processor
  • Supermicro MBD-X11SSM-F-O Micro ATX LGA1151 Motherboard
  • Kingston ValueRAM 32GB Kit (2x16GB) 2133MHz DDR4 ECC CL15 DIMM 2Rx8 Desktop Memory (KVR21E15D8K2/32)
  • Corsair - RM 750W 80+ Gold Certified Fully-Modular ATX Power Supply

Passes my theory-craft. Unknown factor on mobo, due to not seeing IOMMU groups for myself (that's what ACS override is for, right?), but I can't give you reason why it would not work.

Edited by Jcloud
Link to comment
On 3/28/2018 at 6:52 PM, Jcloud said:

Passes my theory-craft. Unknown factor on mobo, due to not seeing IOMMU groups for myself (that's what ACS override is for, right?), but I can't give you reason why it would not work.

 

Thanks for the information! I am seriously thinking about getting them soon.

Link to comment
19 minutes ago, sakh1979 said:
On 3/28/2018 at 4:52 PM, Jcloud said:

Passes my theory-craft. Unknown factor on mobo, due to not seeing IOMMU groups for myself (that's what ACS override is for, right?), but I can't give you reason why it would not work.

 

Thanks for the information! I am seriously thinking about getting them soon.

I'm assuming you did, I did not but just thought of it now, checked that the RAM is on the motherboard's QVL (which I wouldn't sweat directly as I REALLY trust Kingston) but that the memory is of correct specification for mobo (it looked right when I was reading the post).  But I'm also a paranoid-type, and hence asking/double-checking.

Link to comment
13 hours ago, Jcloud said:

I'm assuming you did, I did not but just thought of it now, checked that the RAM is on the motherboard's QVL (which I wouldn't sweat directly as I REALLY trust Kingston) but that the memory is of correct specification for mobo (it looked right when I was reading the post).  But I'm also a paranoid-type, and hence asking/double-checking.

 

You got me curious, I went to the SuperMicro's website and search for the motherboard. It just lists one RAM they tested with. The Kingston ValueRAM 32GB Kit isn't one of them.

 

I went to pcpartpicker.com and added all of my components, according to it, my parts should work together.

 

Can anyone else suggest other methods to verify that the RAM and motherboard will work together before I buy them?

Link to comment
22 minutes ago, sakh1979 said:

You got me curious, I went to the SuperMicro's website and search for the motherboard. It just lists one RAM they tested with. The Kingston ValueRAM 32GB Kit isn't one of them.

They can't test all available modules, but that Kingston kit will work for sure as it's the same model I'm using on one of mine X11SSM-F.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.