Final Bachelor Build - Xeon-D vSAN Cluster for Plex Redundancy

IamSpartacus

By IamSpartacus
in Unraid Compulsive Design

Recommended Posts

IamSpartacus Collaborator

IamSpartacus

Posted
Posted

I'm getting married next year and thus this is my last opportunity to spend a good chunk of money on my "toys" before I need wifey approval for everything (queue the tears and boos).  I've been saving up for a network overhaul for some time now (since before I proposed :thumb:)

 

So the plan is to build a 3-node VMware vSAN cluster for the purpose of having full redundancy of all my VMs and Docker appdata.  I want to be able to take a single server off-line while still keeping the Linux VM running my media dockers (mainly Plex) online as well as the rest of my VMs (Windows AD, DNS, etc.).  I also just want to generally play around with VMware clustering, HA, and vMotion for my own personal knowledge as I'd like to implement some of this at work next year.

 

** Please note that while I realize UnRAID is only a small part of this overall build, a lot of why I'm doing this is to add redundancy/performance to my docker data.  Also a lot of the hardware I'm using I believe will be of some interest to users on this forum.

 

 

So without further ado, this is the hardware that will be going into a new server rack that will sit in my home office (I will build out a full network diagram once I have everything in place and will update you all with that in a subsequent post).

 

Rack:  22U Linier Server Rack

 

ALT=""http://www.overclock.net/content/type/61/id/2741696/width/350/height/700[/img]

 

 

Equipment Going in the Rack (from bottom to top)

 

Cooling:  AC Infinity CLOUDPLATE T7 - [2U]

 

UPS:  CyberPower 2U 900w rack mountable UPS - [2U]

 

ESXi Node #1:  vSAN Storage Only Node + UnRAID VM - [4U]

 

 

ESXi Node #2 & #3 (Identical Nodes):  Hyperconverged Computing/Storage Nodes - [2U each]

 

 

10GbE Switch:  Dell X1052 Smart Managed Switch (48 x 1Gb, 4 x SFP+ ports) - [1U]

 

pfSense Box:  Router and Firewall - [1U]

 

24-Port Patch Panel:  Cable Matters Rackmount or Wallmount 24-Port Cat6 RJ45 Patch Panel - [1U]

 

 

I should add that I also have a backup UnRAID server in a second site that is connected to my network via a site-to-site VPN connection.  I plan to use mergefs to merge both sets of UnRAID shares together into single mount points that I will present to my Dockers in Linux.  This way if for whatever I reason I need to take my main UnRAID array offline or it goes down, my Plex server will remain online.  100Mbps should suffice most of the time for my media streaming needs even with the amount or remote users I have.

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted
Round 1 Photos:  Equipment Arriving

 

Cable Matters Rackmount or Wallmount 24-Port Cat6 RJ45 Patch Panel

 

ALT=""http://www.overclock.net/content/type/61/id/2745643/width/500/height/1000[/img]

 

 

32GB Samsung DDR4 2133 RDIMMs

 

ALT=""http://www.overclock.net/content/type/61/id/2745637/width/500/height/1000[/img]

 

 

Hitachi 400GB SLC Enterprise SSDs

 

ALT=""http://www.overclock.net/content/type/61/id/2745638/width/500/height/1000[/img]

 

 

ICY DOCK MB994SK-1B ToughArmor 4 x 2.5" SATA 6Gbps/SAS HDD/SSD Mobile Rack / Cage with key lock in 1 x 5.25" bays

 

ALT=""http://www.overclock.net/content/type/61/id/2745639/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2745640/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2745641/width/500/height/1000[/img]

 

 

360w Seasonic Gold PSUs

 

ALT=""http://www.overclock.net/content/type/61/id/2745642/width/500/height/1000[/img]

 

 

iStarUSA D Value D-214-MATX Black Steel 2U Rackmount Chassis'

- Pre-Installation

 

ALT=""http://www.overclock.net/content/type/61/id/2745645/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2745646/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2745647/width/500/height/1000[/img]

 

iStarUSA D Value D-214-MATX Black Steel 2U Rackmount Chassis'

- Internal 2.5" drive mounts removed.  ICY DOCK, PSU, and case fans installed.

 

ALT=""http://www.overclock.net/content/type/61/id/2745648/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2745650/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2745652/width/500/height/1000[/img]

 

 

Round 2 Photos:  More Equipment Arriving

 

iStarUSA M-4160-ATX 4U 16-Bay Trayless Chassis

 

ALT=""http://www.overclock.net/content/type/61/id/2746371/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746372/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746373/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746374/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746376/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746377/width/500/height/1000[/img]

 

 

CyberPower 2U 900w rack mountable UPS

 

ALT=""http://www.overclock.net/content/type/61/id/2746364/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746365/width/500/height/1000[/img]

 

 

CyberPower RMCARD303 OL Series Remote Management Card

 

ALT=""http://www.overclock.net/content/type/61/id/2746367/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746369/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2746370/width/500/height/1000[/img]

 

 

 

 

 

 

Round 5 Photos:  Xeon D MoBos + Other Goodies

 

SuperMicro X10SDV-7TP4F Xeon D-1537 Motherboard (x2)

 

ALT=""http://www.overclock.net/content/type/61/id/2751836/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2751837/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2751838/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2751839/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2751840/width/500/height/1000[/img]

 

Nebula SFP-H10GB-CU2M 10GB SFP+ to SFP+ Twinax DAC 2M Cables (10Gb Goodness :D)

 

ALT=""http://www.overclock.net/content/type/61/id/2751841/width/500/height/1000[/img]

 

SuperMicro SuperChassis SC505-203B 1U Rackmount Case (pfSense Firewall)

 

ALT=""http://www.overclock.net/content/type/61/id/2751842/width/500/height/1000[/img]

ALT=""http://www.overclock.net/content/type/61/id/2751843/width/500/height/1000[/img]

 

 

Link to comment
ninthwalker Explorer

ninthwalker

Posted
Posted

This is fun to watch, thanks for the pics and updates. Wish I had the money/time to build this out.

I currently have 2 esxi servers at home with unraid virtualized as well, was thinking of fooling around with vsan at some point. We use it at work now for some of our UCS hosts.

 

The one thing I don't understand for your build is how you are doing unRAID's shared storage between the ESXi hosts.

I'm assuming the 16bay chasis is for all of your unraid storage drives. How is that connected back to each of the nodes, or is it only on 1?

Thanks.

 

 

- Ninthwalker

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

The one thing I don't understand for your build is how you are doing unRAID's shared storage between the ESXi hosts.

I'm assuming the 16bay chasis is for all of your unraid storage drives. How is that connected back to each of the nodes?

Thanks.

 

- Ninthwalker

 

Hey ninthwalker.  I'm not completely sure what you're asking here. 

 

My UNRAID VM will be outside the vSAN cluster but still on one of the vSAN nodes (D-1518 MoBo in 4U iStarUSA case).  I believe I have two options here for connecting it to the vSAN cluster.  I can either use a vSwitch to connect it to the vSAN cluster by way of the local host it's connected to or I can use the spare SFP+ port in that server, assign it to the UnRAID VM, and connect it back to my network (I have one spare SFP+ port on my switch).  I haven't yet decided which way I'm going to do that as I still have much to configure (distributed switch, HA, vMotion, etc.) on the cluster before I get to that point.

Link to comment
ninthwalker Explorer

ninthwalker

Posted
Posted

 

Hey ninthwalker.  I'm not completely sure what you're asking here. 

 

My UNRAID VM will be outside the vSAN cluster but still on one of the vSAN nodes (D-1518 MoBo in 4U iStarUSA case).  I believe I have two options here for connecting it to the vSAN cluster.  I can either use a vSwitch to connect it to the vSAN cluster by way of the local host it's connected to or I can use the spare SFP+ port in that server, assign it to the UnRAID VM, and connect it back to my network (I have one spare SFP+ port on my switch).  I haven't yet decided which way I'm going to do that as I still have much to configure (distributed switch, HA, vMotion, etc.) on the cluster before I get to that point.

 

Oh, alright. I think I understand. Your unRAID will be virtualized but not HA. It's just on the one node, can't withstand a node failure.

What other VM's do you plan on running? Also, have you thought about virtualizing pfsense, or do you like the physical separation?

I've had mine virtualized for awhile and it runs really well imo. You can HA it with it's CARP feature too.

 

- Ninthwalker

 

 

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

 

Hey ninthwalker.  I'm not completely sure what you're asking here. 

 

My UNRAID VM will be outside the vSAN cluster but still on one of the vSAN nodes (D-1518 MoBo in 4U iStarUSA case).  I believe I have two options here for connecting it to the vSAN cluster.  I can either use a vSwitch to connect it to the vSAN cluster by way of the local host it's connected to or I can use the spare SFP+ port in that server, assign it to the UnRAID VM, and connect it back to my network (I have one spare SFP+ port on my switch).  I haven't yet decided which way I'm going to do that as I still have much to configure (distributed switch, HA, vMotion, etc.) on the cluster before I get to that point.

 

Oh, alright. I think I understand. Your unRAID will be virtualized but not HA. It's just on the one node, can't withstand a node failure.

What other VM's do you plan on running? Also, have you thought about virtualizing pfsense, or do you like the physical separation?

I've had mine virtualized for awhile and it runs really well imo. You can HA it with it's CARP feature too.

 

- Ninthwalker

 

Right, the UnRAID won't be redundant in the way the all my other VM's on the vSAN cluster will be.  However it is redundant in the sense that I have a secondary UnRAID server connected via VPN that can still server my vSAN cluster fast enough (100Mbps) for basic media streaming purposes.

 

Other than the main Linux server VM that will be running all my dockers I also have 2 Server 2012 AD servers (Active Directory, DNS, Group Policy, etc) and a whole lot of various Windows, Linux, Mac OSX VM's for testing purposes.  The Docker VM as well as my two Windows Server VM's are what I'd consider my "production" VM's while the others are much less important.

 

I have in the past (last time was about 2 years ago) run pfSense virtualized however I do prefer to keep my networking nodes physically isolated.  In the past if I had to take down the host running pfSense I'd lose my network connectivity which caused some complications especially when working remotely.  Could I go with a HA pfSense VM now with this hardware?  Sure, but I don't really see the need.  I'm not running any kind of business applications on my network where I just can't afford downtime.  For me it's more of a convenience thing and I'm a media junkie who doesn't like when my media is inaccessible (nor do my remote users ;) ).

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

I've spend the past few days configuring my vSAN cluster and I'm happy to say it is up and running.  I've yet to setup the Ubuntu Server VM that I will be using to run Plex and the rest of my other dockers so that is the next step.  I hope to get that up and running by the end of this weekend so that I can start doing some testing with regard to mergefs and meshing both my main and backup UnRAID array shares for some redundancy.

 

I'm also contemplating a better way to configure my virtual networking but I'm not sure anyone on this board is interested in hearing all about my vDS setups so I'll reserve those details for anyone who explicitly asks for them.

 

Much more to come :D.

Link to comment
jbrodriguez Rising Star

jbrodriguez

Posted
Posted

hey IamSpartacus, that's some amount of equipment you have there ! :)

 

I especially like the SuperMicro A1SRi-2558F you set up for pfsense.

 

I have a bit older A1SAM-2550F, but it's been rock solid for almost two years (running vyos)

 

You did the RIGHT thing, by doing this now, while still months away from requiring wife approval ! :)

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

hey IamSpartacus, that's some amount of equipment you have there ! :)

 

I especially like the SuperMicro A1SRi-2558F you set up for pfsense.

 

I have a bit older A1SAM-2550F, but it's been rock solid for almost two years (running vyos)

 

You did the RIGHT thing, by doing this now, while still months away from requiring wife approval ! :)

 

Yea the A1SRi-2558F is a fantastic option for a firewall/router solution.  I have a 100Mbps site-to-site OpenVPN connection between my apartment and my parents house and saturating that link only pushes the CPU to the 15-20% range so I've got a lot of headroom.

 

And yes, wife approval is right around the corner so I got this project out of the way just in time :D.

Link to comment
  • 4 weeks later...
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

I'm happy to say that my build is COMPLETE and I'm very satisfied with the finished product  8).

 

uMYOSO4.jpg

 

From top to bottom in the rack:

 

24 port Cat6 Patch Panel

Switch: Dell X1052 10Gb Switch

Firewall: pfSense 2.3.1 running on SuperMicro A1SRi-2558F

vSAN Node #1: Xeon-D 1537 / 64GB RAM / 400GB Hitachi HUSSL (cache) / 800GB Intel S3500

vSAN Node #2: Xeon-D 1537 / 64GB RAM / 400GB Hitachi HUSSL (cache) / 800GB Intel S3500

vSAN Node #3: Xeon D-1508 / 16GB RAM / 8 x 8TB Seagate SMR HDDs

vSAN Node #4: Xeon D-1518 / 32GB RAM / 400GB Hitachi HUSSL (cache) / 800GB Intel S3500 / IBM M1015 / 8 x 8TB Seagate SMR HDDs

CyberPower 900w UPS

 

 

You'll notice that my rack is sort of broken down into 3 main sections.

 

Networking (PP, Switch, Firewall)

Computing (vSAN Node 1,2)

Storage (vSAN Node 3,4).

 

Nodes 1 and 2 are my main computing boxes. They run all my high CPU intensive applications and the VM's on those servers fail-over to one another or get migrated during maintenance.

 

Node 3 is currently just a "slave" for Node 4. It's not currently contributing any storage to the vSAN datastore but it will once I can pickup a 4th Intel S3500 800GB drive. The only VM running on it is my bulk storage OS (UnRAID). The 8TB drives are spun down 95% of the time except during data replication from Node 4 or if Node 4 is offline for whatever reason. This server isn't using much power.

 

Node 4 is running my main storage array (outside the vSAN datastore of course). On top of the storage OS, I also have vCenter running on there.

 

 

Deciding to convert my offsite backup server (very underutilized) to an on site backup / HA for my media was a good decision. One of the top priorities for this build was to add redundancy / HA to my Plex Server and while running Plex in an Ubuntu VM on the vSAN datastore did that, the actual media became the failure point. Moving my backup server on site allowed me to pool NFS shares off both servers into single mount points for Plex to access (using MergerFS). Thus if Bulk Array #1 is offline for whatever reason (failure, maintenance, etc.) Plex will automatically read/serve the data off of Bulk Array #2.

 

At higher than average usage right now the rack is pulling 360w in total. When mostly idle it's under 250w.

 

Now that I've got my "production" services running for home, it's now time to build out my Windows testing/lab environment ;).

 

** Additional Close-Up pics of each sections **?

 

k9EtkMz.jpg

EBixYUa.jpg

VFewDO7.jpg

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

VERY nice build  :)

 

I really like the Linier cabinet !!    If you're going to use rack-mount equipment, you should definitely have a nice rack to install it in ... far too many folks just buy a rack-mount case and stick it on a table.

I agree.  I was very impressed with the build quality of this rack.  It wasn't cheap but I can see it lasting me a very long time.

 

Sent from my Nexus 5X using Tapatalk

 

 

Link to comment
bluepr0 Explorer

bluepr0

Posted
Posted

Loved to see the process, thanks for sharing it!

 

May I ask what's all this equipment for? (feel free to not answer, I'm just curious to see what you can do with so much power!). Also, are you going to use one of the VMs to work (if you happen to need one, of course) or you have a dedicated machine for you? (just wondering to see how serious people takes VMs to do "real" work)

 

Thanks a lot!

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

Loved to see the process, thanks for sharing it!

 

May I ask what's all this equipment for? (feel free to not answer, I'm just curious to see what you can do with so much power!). Also, are you going to use one of the VMs to work (if you happen to need one, of course) or you have a dedicated machine for you? (just wondering to see how serious people takes VMs to do "real" work)

 

Thanks a lot!

 

I have no reservations about answering your questions.

 

I have been saving for this home network overhaul for about a year and the main goal was to build out a nice lab environment for testing different configurations I may want to run at work (I'm a network admin/IT consultant). 

 

On top of that, I feel the best way to learn new technologies is to apply them to your own environments.  I'm a HEAVY Plex user and have a lot of family members/close friends who rely on my Plex server daily.  I often have 12-15 streams going per night.  So as you can imagine, downtime is not a good thing.  Not that I would let others dictate how I run/setup my network, as I said I'm a heavy user myself and do not have cable TV service so it affects me the most.  So in the process of spec'ing out the hardware I decided to focus not only on creating a great lab environment, but configuring some sort of high availability to both my Plex server itself and the underlying bulk media associated with it.

 

As for using a VM as my everyday workstation, I did strongly consider doing so.  However, after a lot of consideration, I decided it was still in my best interest to keep my standalone PC as my everyday workstation completely isolated from the vSAN cluster.  I didn't want to have a scenario in the future where the cluster had to be down for some reason and thus I can't remotely manage it.

 

 

 

 

Link to comment
ninthwalker Explorer

ninthwalker

Posted
Posted

EDIT:

I found and read your Reddit posts and pretty much all of my questions were already asked/answered there, so you can disregard this post. On a side note though, I really want to setup mergerfs like that for my own setup at home now too. Thanks for answering everyone's questions (and the same ones like 3x, haha).

 

If you can answer my last 'PS' question below, I don't think that one was asked yet. Thanks.

 

- Ninthwalker

 

======================================

 

If you have time I would be interested in hearing how you implemented mergerfs with your two unraid servers. Was it semi easy with just pointing plex's path's to the mergerfs NFS shares from each of your unraid nodes? Can mergerfs be readonly for that purpose, as I'm assuming you don't write through it and instead sync your 2 unraid servers? Also, how does mergerfs decide which drive to read from if the same data is residing on your multiple unraid NFS Shares?

 

I currently am able to have everything HA between my ESXi hosts, but I have quite a few Plex users which it sounds like you do too.

Keeping Plex up while I take down one unraid to do stuff with would be legit and eliminates the storage SPOF with that.

Also, from what I understand your Node 4 is your main unraid, node 3 is backup. What method do you currently use to sync your 2 unraid servers? Have you tested a failover of Plex yet? Say your primary node with Plex goes out and also the primary unraid. How seamless is the migration of Plex to the other node and mergerfs routing to the drives still available? Sorry for all the questions, they just kept coming as I typed this. Thanks!

 

PS: Did I read it correct that you have a 400GB SSD Cache and an 800GB SSD data for each vsan node? 50% Cache? Why so high? VMware recommends ~10% cache usually. Also with a FTT of 1, you are only giving yourself about 400Gb of vsan storage. I guess that's enough for the VM's you plan on running on it?

 

-Ninthwalker

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

PS: Did I read it correct that you have a 400GB SSD Cache and an 800GB SSD data for each vsan node? 50% Cache? Why so high? VMware recommends ~10% cache usually. Also with a FTT of 1, you are only giving yourself about 400Gb of vsan storage. I guess that's enough for the VM's you plan on running on it?

 

-Ninthwalker

 

I'm using those 400GB cache drives because I got an insane deal on them ($99 each).  They have 38PB of write endurance so I figured I'd give myself lots of room to add more capacity drives to each node in the future.

 

As for the FTT, since I currently have 3 nodes contributing storage of 800GB which gives me about 1.2TB ((800GB x 3) / 2) of usable storage that allows for a single node to fail and all my VMs to stay up.  I have extra cache drives so as soon as I find a good deal on a 4th S3500 800GB I'm going to add storage to my 4th node which will allow me to tolerate a failure while one node is in maintenance mode and give me added space on the vSAN datastore.

Link to comment
IamSpartacus Collaborator

IamSpartacus

Posted
  • Author
Posted

Can you post the price total so I can show my wife next time I have to *ask* for permission for a part?

 

See honey, compared to that guy's, this is nothing

Haha anything I can do to help!

 

Luckily I already had all the 8TB drives and about $2500 worth of other gear to offset some of the cost.  Spent about $8k in new gear so $5500 out of pocket.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing