I've done a bad bad thing…but don't tell the wife…I had the opportunity to splurge a little…so I did.

 

I have a few build questions regarding software and hardware.

 

From this forum and others, I have read and learnt much from everyone's feedback. Below are some of my outstanding questions I have not been able to confirm and so now I seek your input and hopefully the responses will help others in time when they have similar build questions.

 

Thanks in advance for your input! I (and others) will value your constructive feedback.

 

Before getting to the questions, here is what I have dedicated to the build thus far:

 

M'board: Supermicro X10SL7-F

Cpu: Intel Quad Core Xeon CPU E3-1240v3, LGA1150, 3.4GHz 8MB CACHE

Ram: Kingston 32GB 1600MHz DDR3 ECC Reg CL11 DIMM (Kit of 4x 8GB)

Hdd: 5x Western Digital Black WD4001FAEX 3.5" CAVIAR BLACK 4TB 7200RPM 64MB SATA III

Case: Lian Li Black PC-D8000 Full Tower Chassis

Case Backplanes: 3x Lian Li BP3SATA 3.5 HDD Hot Swap BackPlane SATA3 6G BPS

Psu: SeaSonic XP860W Version 2 80Plus Platinum

 

As you have $een, this will be my Christma$ and Birthday pre$ent for the next few years. So although I am wary of the total cost, it is not an immediate factor in this build. (Long term stability and flexibility are the balancing items to cost.)

 

Purpose:

Initially I was going to simply run UnRaid on a server but quickly saw the benefits virtualisation would offer me…one of them being stability which was core to my hardware choices.

I admit the main driver for UnRaid was for Plex, and was going to take advantage of the 3rd party addons etc, but have determined it best to keep UnRaid as it was designed for and to separate server tasks to appropriate virtual servers. Ultimately I want to utilise virtual servers for Plex and other services.

 

After some reading I thought ESXi (free) would be the way to go with UnRaid. So I compiled the above hardware with ESXi in mind. However after some further reading and consideration I am now contemplating whether XenServer might be a better alternative.

 

Software related:

1) With what seems a growing interest in XenServer with UnRaid, do you see a long term benefit in sticking with ESXi or would you take the plunge and wait a little longer to go with XenServer? (once the dust settles down in the XenServer threads)

2) Can I setup UnRaid first before deciding whether to go ESXi or XenServer (or another hypervisor)? In other words, is it easy to migrate an existing UnRaid server to one of the proposed hypervisors?

3) If after finding the chosen hypervisor is no longer appropriate, what would be required and what would be lost upon switching hypervisors?

4) Are there any hardware restrictions which would prevent me from running one or the other hypervisor?

 

Hardware related:

SSD:

5) How many and what sizes do you recommend? Eventually I will use one as cache for UnRaid, but apart from the initial virtual server start up times, do you feel a ssd drive is beneficial to a hypervisor? [ps. I intended to use a USB stick to start ESXi.]

Data Cables:

6) What cables will be required to connect the X10SL7-F onboard lsi controller ports to the backplanes? [This lsi controller will solely be for UnRaid.]

7) Is anyone with the D8000 case able to say how long the various cables required (including power cables) need to be? 

Case Fans/Cooling:

With so much choice out there for fans I am open to suggestions for quiet 120mm and 140mm fans.  I am contemplating using 4-pin power distribution kits for the fans…thoughts?  Any other cooling options worth considering? My man-cave is not the coolest of rooms, and has carpet floors.

Power cables:

What are my molex requirements to split the power out to what will hopefully be a full case of HDDs in due course.

HBA card:

Eventually I will need a HBA card (also dedicated to UnRaid) to max out the HDD space the D8000 provides, which will connect to one of the PCIe slots.At that point I will get more backplanes too for the HDD hot swapping.

 

Again, thank you for reaching this far into the post.

 

gwl

 

 

 

Sent from my iPad using Tapatalk

I've done a bad bad thing…but don't tell the wife…I had the opportunity to splurge a little…so I did.

 

I have a few build questions regarding software and hardware.

 

From this forum and others, I have read and learnt much from everyone's feedback. Below are some of my outstanding questions I have not been able to confirm and so now I seek your input and hopefully the responses will help others in time when they have similar build questions.

 

Thanks in advance for your input! I (and others) will value your constructive feedback.

 

Before getting to the questions, here is what I have dedicated to the build thus far:

 

M'board: Supermicro X10SL7-F

Cpu: Intel Quad Core Xeon CPU E3-1240v3, LGA1150, 3.4GHz 8MB CACHE

Ram: Kingston 32GB 1600MHz DDR3 ECC Reg CL11 DIMM (Kit of 4x 8GB)

Hdd: 5x Western Digital Black WD4001FAEX 3.5" CAVIAR BLACK 4TB 7200RPM 64MB SATA III

Case: Lian Li Black PC-D8000 Full Tower Chassis

Case Backplanes: 3x Lian Li BP3SATA 3.5 HDD Hot Swap BackPlane SATA3 6G BPS

Psu: SeaSonic XP860W Version 2 80Plus Platinum

Good looking list thought the Black drives are complete overkill and not really worth the extra cost in my opinion.  The one thing I would highly suggest you pick up an M1015 card once you want more than 8 drives.  The first 8 can and should go on the LSI controller, you will also likely need to flash the LSI so it will work with unRAID.

 

Purpose:

Initially I was going to simply run UnRaid on a server but quickly saw the benefits virtualisation would offer me…one of them being stability which was core to my hardware choices.

I admit the main driver for UnRaid was for Plex, and was going to take advantage of the 3rd party addons etc, but have determined it best to keep UnRaid as it was designed for and to separate server tasks to appropriate virtual servers. Ultimately I want to utilise virtual servers for Plex and other services.

I love my ESXi setup and I am still running version 4.1 or something like that.

 

After some reading I thought ESXi (free) would be the way to go with UnRaid. So I compiled the above hardware with ESXi in mind. However after some further reading and consideration I am now contemplating whether XenServer might be a better alternative.

I am sure you can get the other ones to work but ESXi is mostly dead simple to get going. Especially with the very good "Atlas" thread.  You can also look at my "Andormeda" ESXi build for an idea of what pieces and parts I am using.

 

Software related:

1) With what seems a growing interest in XenServer with UnRaid, do you see a long term benefit in sticking with ESXi or would you take the plunge and wait a little longer to go with XenServer? (once the dust settles down in the XenServer threads)

2) Can I setup UnRaid first before deciding whether to go ESXi or XenServer (or another hypervisor)? In other words, is it easy to migrate an existing UnRaid server to one of the proposed hypervisors?

3) If after finding the chosen hypervisor is no longer appropriate, what would be required and what would be lost upon switching hypervisors?

4) Are there any hardware restrictions which would prevent me from running one or the other hypervisor?

1) I would stick with ESXi.  I am not one that upgrades there version of ESXi unless I need to.  My approach has always been that it it works why bother f'ing with it.  Get ESXi (or whatever you choose) setup and then don't mess with it any more.  Auto updates on a server are the devils doing.

2) Yes, it is fairly straight forward to do.  Though are I mentioned above I would suggest getting an M1015 card to use to connect all the drives.  You will need this if you decide to go ESXi.

3) So far as I know there would be not a whole lot to do from an unRAID side (depending on how you set it up).  I use the plop method to boot unRAID under ESXi, some use a vmdk.  A different hypervisor would likely NOT be able to read/use that vmdk file so you would need to create a new one of those.

4) Can't answer that as I am only really familiar with ESXi.

 

Hardware related:

SSD:

5) How many and what sizes do you recommend? Eventually I will use one as cache for UnRaid, but apart from the initial virtual server start up times, do you feel a ssd drive is beneficial to a hypervisor? [ps. I intended to use a USB stick to start ESXi.]

Data Cables:

6) What cables will be required to connect the X10SL7-F onboard lsi controller ports to the backplanes? [This lsi controller will solely be for UnRaid.]

7) Is anyone with the D8000 case able to say how long the various cables required (including power cables) need to be? 

Case Fans/Cooling:

With so much choice out there for fans I am open to suggestions for quiet 120mm and 140mm fans.  I am contemplating using 4-pin power distribution kits for the fans…thoughts?  Any other cooling options worth considering? My man-cave is not the coolest of rooms, and has carpet floors.

Power cables:

What are my molex requirements to split the power out to what will hopefully be a full case of HDDs in due course.

HBA card:

Eventually I will need a HBA card (also dedicated to UnRaid) to max out the HDD space the D8000 provides, which will connect to one of the PCIe slots.At that point I will get more backplanes too for the HDD hot swapping.

5) You could use an SSD for the datastore.  The Atlas build uses one and I originally had one also.  I have since moved to using an Areca 1222 RAID card with battery backup unit.  I have 4 x 1TB Seagate drives hook up in a RAID5.  I have not tested the speed of this RAID but it is more than fast enough for my use and the longevity should be much better than that of an SSD.  I also have a little bit of a fallback should one of the drives in the RAID fail.  With a single SSD you are opening yourself up to losing your VM's

6) From the pictures it looks like 8 SATA ports are hooked up to the controller so just SATA cables should work.

7) Can't answer that.  My suggestion is to order a variety from Monoprice.com and just go from there.  If you have extras so be it.

8 ) If you need more molex, send an email off to seasonic.  They will either sell you more cables for your modular PSU or they will direct you to the correct place to purchase them.

1-4 no xp with virtualization here

 

5. SSD:

Not really necessary as cache since the NIC is the bottleneck. A fast HDD will be fine.

SSD maybe good for virtualization and/or plugins?

 

6. The mobo connectors look like usual SATA.

 

7. Nice case, but again no xp with it here

 

Cooling:

You won't be happy with the WD BLACK and the backplanes if you want the rig to run at a low noise level.

Look at the small holes in the backplanes - they are limiting the air flow. Unfortunately this is common to all backplanes...small holes in the PCB.

If possible use the backplanes with "green" drives or better avoid the backplanes.

I'm not even sure if unRAID supports hot swap.

If you plan to use the server for storage, use "green" drives. WD RED series or Seagate NAS drives.

 

Power distribution:

I suppose your PSU has modular cables. I took one of the power cords and carefully removed the connectors. Somebody posted a source in the US where they can be orderer - I had to improvise.

Then I placed them in the gaps on another cable that I planned to use because I found the distance between 2 connectors was too far. Depending on the cable lenght you can easily string together 8 drives with one wire.

In addition to that i used this forward breakout cables.

With them I need 2 connectors from the PSU to power 8 drives.

They help to tidy up the case. Unfortunately they're FSC spare parts and neither easy to get nor inexpensive.

Look for part number T26139-Y4023-V501

 

For HBA:

Check the controller list in the wiki.

You will probably want a 8-port PCIe x8 controller. Either an HBA or an SAS RAID controller that can be flashed to

IT mode. It's all in the list.

Pay attention to obtain forward breakout cables!

 

If your case will be fully populated one day, you will have to find another place than your man-cave for it.

Silent fans are good but you can't silence 20+ drives. 

Luckily they don't spin all at once all the time...

Before getting to the questions, here is what I have dedicated to the build thus far:

 

M'board: Supermicro X10SL7-F

Cpu: Intel Quad Core Xeon CPU E3-1240v3, LGA1150, 3.4GHz 8MB CACHE

Ram: Kingston 32GB 1600MHz DDR3 ECC Reg CL11 DIMM (Kit of 4x 8GB)

 

...that mobo only supports unregistered, unbuffered ECC RAM.

Good looking list thought the Black drives are complete overkill and not really worth the extra cost in my opinion.  The one thing I would highly suggest you pick up an M1015 card once you want more than 8 drives.  The first 8 can and should go on the LSI controller, you will also likely need to flash the LSI so it will work with unRAID.

Yes, I had the Reds down first but a fellow Aussie (BillyJ) identified (http://lime-technology.com/forum/index.php?topic=27800.msg273506#msg273506) some problems with the WD Reds for the onboard lsi controller, so to play it safe I opted for the Blacks instead for the time being. Over time, I expect I'll have a quite an assortment of HDDs. The m1015 card is on my watchlist for when storage requirements increase.

I love my ESXi setup and I am still running version 4.1 or something like that.

 

I am sure you can get the other ones to work but ESXi is mostly dead simple to get going. Especially with the very good "Atlas" thread.  You can also look at my "Andormeda" ESXi build for an idea of what pieces and parts I am using.

The Atlas thread was certainly one that helped me along some of my decisions. I've brought up your Andromeda thread to read through again.

 

Software related:

1) With what seems a growing interest in XenServer with UnRaid, do you see a long term benefit in sticking with ESXi or would you take the plunge and wait a little longer to go with XenServer? (once the dust settles down in the XenServer threads)

2) Can I setup UnRaid first before deciding whether to go ESXi or XenServer (or another hypervisor)? In other words, is it easy to migrate an existing UnRaid server to one of the proposed hypervisors?

3) If after finding the chosen hypervisor is no longer appropriate, what would be required and what would be lost upon switching hypervisors?

4) Are there any hardware restrictions which would prevent me from running one or the other hypervisor?

1) I would stick with ESXi.  I am not one that upgrades there version of ESXi unless I need to.  My approach has always been that it it works why bother f'ing with it.  Get ESXi (or whatever you choose) setup and then don't mess with it any more.  Auto updates on a server are the devils doing.

2) Yes, it is fairly straight forward to do.  Though are I mentioned above I would suggest getting an M1015 card to use to connect all the drives.  You will need this if you decide to go ESXi.

3) So far as I know there would be not a whole lot to do from an unRAID side (depending on how you set it up).  I use the plop method to boot unRAID under ESXi, some use a vmdk.  A different hypervisor would likely NOT be able to read/use that vmdk file so you would need to create a new one of those.

4) Can't answer that as I am only really familiar with ESXi.

Agreed, if it ain't broke...

For 2), as mentioned here (http://lime-technology.com/forum/index.php?topic=27600.msg243301#msg243301) I think the onboard LSI controller can be passed through to ESXi for use by UnRaid and then when I need more storage I could also pass through the M1015 card. If I learn things (read: make mistakes) along this build journey to help others, so be it.

For ESXi, I was going to use a vmdk to boot UnRaid, but I'll do a bit more reading on your 'plop' approach.

 

Hardware related:

SSD:

5) How many and what sizes do you recommend? Eventually I will use one as cache for UnRaid, but apart from the initial virtual server start up times, do you feel a ssd drive is beneficial to a hypervisor? [ps. I intended to use a USB stick to start ESXi.]

Data Cables:

6) What cables will be required to connect the X10SL7-F onboard lsi controller ports to the backplanes? [This lsi controller will solely be for UnRaid.]

7) Is anyone with the D8000 case able to say how long the various cables required (including power cables) need to be? 

Case Fans/Cooling:

With so much choice out there for fans I am open to suggestions for quiet 120mm and 140mm fans.  I am contemplating using 4-pin power distribution kits for the fans…thoughts?  Any other cooling options worth considering? My man-cave is not the coolest of rooms, and has carpet floors.

Power cables:

What are my molex requirements to split the power out to what will hopefully be a full case of HDDs in due course.

HBA card:

Eventually I will need a HBA card (also dedicated to UnRaid) to max out the HDD space the D8000 provides, which will connect to one of the PCIe slots.At that point I will get more backplanes too for the HDD hot swapping.

5) You could use an SSD for the datastore.  The Atlas build uses one and I originally had one also.  I have since moved to using an Areca 1222 RAID card with battery backup unit.  I have 4 x 1TB Seagate drives hook up in a RAID5.  I have not tested the speed of this RAID but it is more than fast enough for my use and the longevity should be much better than that of an SSD.  I also have a little bit of a fallback should one of the drives in the RAID fail.  With a single SSD you are opening yourself up to losing your VM's

6) From the pictures it looks like 8 SATA ports are hooked up to the controller so just SATA cables should work.

7) Can't answer that.  My suggestion is to order a variety from Monoprice.com and just go from there.  If you have extras so be it.

8 ) If you need more molex, send an email off to seasonic.  They will either sell you more cables for your modular PSU or they will direct you to the correct place to purchase them.

I've certainly still got some work to do here and it will take some time to properly piece together the hardware and virtual servers to get it stable, cool and quiet enough that I can live with. I will just slowly build it piece by piece, that way I can spread out the co$t a little to minimise any wifey concerns.

 

cheers,

gwl

Before getting to the questions, here is what I have dedicated to the build thus far:

 

M'board: Supermicro X10SL7-F

Cpu: Intel Quad Core Xeon CPU E3-1240v3, LGA1150, 3.4GHz 8MB CACHE

Ram: Kingston 32GB 1600MHz DDR3 ECC Reg CL11 DIMM (Kit of 4x 8GB)

 

...that mobo only supports unregistered, unbuffered ECC RAM.

 

Thanks for that!

I've been able to exchange the RAM with "Kingston 8GB KVR13E9-8I 1333MHz DDR3 ECC Unbuffered CL9 DIMM with Thermal Sensor, E3 Series Xeon CPU" modules.

 

cheers,

gwl

1-4 no xp with virtualization here

 

5. SSD:

Not really necessary as cache since the NIC is the bottleneck. A fast HDD will be fine.

SSD maybe good for virtualization and/or plugins?

 

6. The mobo connectors look like usual SATA.

 

7. Nice case, but again no xp with it here

Thanks.

I guess the cache drive for UnRaid is something I can experiment with. Although a cache drive is optional, I've a few spare hdds around to see whether ssd or fast rpm makes any noticeable difference to the overall system.

 

Cooling:

You won't be happy with the WD BLACK and the backplanes if you want the rig to run at a low noise level.

Look at the small holes in the backplanes - they are limiting the air flow. Unfortunately this is common to all backplanes...small holes in the PCB.

If possible use the backplanes with "green" drives or better avoid the backplanes.

I'm not even sure if unRAID supports hot swap.

If you plan to use the server for storage, use "green" drives. WD RED series or Seagate NAS drives.

To initially be stable and to get things up and running with minimal conflicts, I went with the WD Black's. BillyJ identified some potential problems with this board and the Reds. I may end up getting greens further down the track but for now these drives will see me through for a little while.

I'll have to wait and see what the air flow is like with the backplanes attached in the D8000. I only got a few for now. If they won't work nicely with UnRaid I can always use them later on with other drives as you suggested.

Cooling will become more important as more drives are added, so I will need to ensure I have enough ventilation, both inside the case and in the man-cave.

 

Power distribution:

I suppose your PSU has modular cables. I took one of the power cords and carefully removed the connectors. Somebody posted a source in the US where they can be orderer - I had to improvise.

Then I placed them in the gaps on another cable that I planned to use because I found the distance between 2 connectors was too far. Depending on the cable lenght you can easily string together 8 drives with one wire.

In addition to that i used this forward breakout cables.

With them I need 2 connectors from the PSU to power 8 drives.

They help to tidy up the case. Unfortunately they're FSC spare parts and neither easy to get nor inexpensive.

Look for part number T26139-Y4023-V501

Great info. I'll investigate this a little more. I think I've got myself confused over time because I had originally looked at getting a RocketRaid card and so looked at getting reverse breakout cables to connect to the backplanes, but since then I've changed my mind a few times and the cables all started looking like spaghetti to me.

 

For HBA:

Check the controller list in the wiki.

You will probably want a 8-port PCIe x8 controller. Either an HBA or an SAS RAID controller that can be flashed to

IT mode. It's all in the list.

Pay attention to obtain forward breakout cables!

Yep, it is on a list of future purchases when I need more storage capacity. I don't need it right now, and there is all likelihood for new controllers to be out by then.

 

If your case will be fully populated one day, you will have to find another place than your man-cave for it.

Silent fans are good but you can't silence 20+ drives. 

Luckily they don't spin all at once all the time...

This was going to be another reason for a ssd cache drive...to keep the noise down a little during the day. But I suspect noise/heat may only be a problem when under high workload.

Fans are relatively cheap so I guess this is one area where I may have to try and buy a few times before determining the right combination of 120mm and 140mm fans.

 

cheers,

gwl

But I suspect noise/heat may only be a problem when under high workload.

Indeed. You won't notice any heat problems until you do a preclear of some/all adjacent drives or a parity check.

In normal operation you'll be just fine but during the monthly parity check cooling will be problematic.

 

Fan noise is basically a matter of rpm.

If you have easy access to your case you could install fans that rev @3000 rpm on 12V.

Use a switch to run them on 5V @~1200 rpm during normal operation and on 12V during parity checks.

That should be possible but requires discipline. But it won't let you automatically schedule the parity check since you always have to switch the fans.

A skilled electrician could assemble a relay card and use the serial interface to do this based on drive temps.

That would be the ultimate solution. 

 

I have a WD black together with 2 green drives in an SNT 2131 backplane.

I need to keep rpm/noise down because the server is also sitting next to me.

The installed fan has a flow rate of 35,7 m3/h @1700rpm.

During parity check the temps of the drives in this backplane will rise up to 52°C.

I will have to exchange the WD black with a "green" drive.

 

I'm very pleased with fans from Noiseblocker.

I have a WD black together with 2 green drives in an SNT 2131 backplane.

During parity check the temps of the drives in this backplane will rise up to 52°C.

 

I will start off with one or two drives and see how the temp goes during preclear and parity check processes.

It may be an idea for me to spread the blacks out a bit so to not centralise the drive's heat. But your 52 degrees is within the Black's specified operating range (perhaps not the Green though?). One other reason I went with the Black drives was their 5 year warranty. If the air flow is severely restricted by my backplanes I'll look to remove them or space them out also.

 

I like the idea of a switch to control the fans, might need a few of them to eventually take all the fans.

 

I'll do some local price shopping for the fans and compare quality/cost to those offered online (such as Noiseblocker).

 

Some useful insights for me to further read up on.

 

cheers,

gwl

But your 52 degrees is within the Black's specified operating range

That is true but somewhere I saw a chart showing the failure rates over temp.

The lowest failure rate was at ~40°C and rising with higher temps.

40 - 45°C is OK for me but above 50°C I feel somewhat uncomfortable although the spec says 60°C.

 

WD greens and Seagate ST4000DM000 are also specified with 65°C respectively 60°C.

 

You will probably have 4 or 5 120mm fans.

I think one appropriate switch should easily handle that.

At least the 5V rail delivers enough current.

 

The lowest failure rate was at ~40°C and rising with higher temps.

40 - 45°C is OK for me but above 50°C I feel somewhat uncomfortable although the spec says 60°C.

 

Agreed. The general consensus here is try to keep your drives under 40C. Up to 45C is ok but not ideal. Above 45C you should look into better cooling. Above 50C you're asking for trouble.

I'm very pleased with fans from Noiseblocker.

 

They look impressive, especially the eLoops, but don't appear to be sold in the US.  Have you tried a Cougar?  If you look closely at the blades, they seem to have some of the same design improvements.  I'd be interested in hearing of or seeing comparative reviews.  My Cougars have been excellent, move a lot of air quietly, for very little extra cost.

I'm not even sure if unRAID supports hot swap.

 

I have read in some older posts that although not a true hot swap (by definition) in UnRaid, the array has to be stopped first before "swapping" the HDD...I guess the "hot" factor here is you don't need to shutdown the whole machine. The backplanes also make it easier/faster by not having to go into the case and messing with the cables. I will need to see how much of a temperature impact the backplanes will have before determining how best to use them. I have another Lian-Li case here that they may fit into, if I don't use them in the D8000.

 

cheers,

gwl