It's time to upgrade, requesting advise

[hawihoney]

I need some advise for an upgrade of my two Unraid servers. I do use these headless servers mainly as a NAS with some Dockers and a remote Windows 10 VM. Here's my current installation for one of both servers. Both use identical hardware:

Internal:
Case: LimeTechnology MD 1500/LL (15 bays)
Power: Corsair 760 Watt AX Series Modular 80+ Platinum
Motherboard: Supermicro X10SL7-F
CPU: Intel i3-4130T 2.90GHz
Cooler: Boxed cooler
Memory: 1x 8 GB Single-bit ECC
Controller1: LSI 9300-8i
Controller2: Adaptec 1430SA
Drives: 2x Parity, 13x Data, 1x Flash

External:
Case: 2x Sans Digital HDDRACK5
Power: Cooler Master B500
Cooler: Integrated
Drives: 6x Data, 2x Cache, 1x Unassigned Devices
- External drives directly connected to internal Controllers

Total: 
2x Parity (WD 60EFRX)
19x Data (100TB, Mix of 3TB-6TB Seagate ST3000DM001, WD 60EFRX, Toshiba HDWE160, ...)
2x Cache (Samsung 860 Pro 256GB building a cache pool)
1x Unassigned Devices

Running: 
1x Unraid, latest stable
1x VM (Windows 10, 2 of 4 cores assigned)
9x Docker (DuckDNS, Letsencrypt, MariaDB, Nextcloud, Plex, MakeMKV, MKVToolNix, NZBGet, TinyMediaManager)
- Critical data is rsynced between both servers every night
- Dockers DuckDNS, Letsencrypt, MariaDB, Nextcloud are running on one server only
- Dockers Plex, MakeMKV, MKVToolNix, NZBGet, TinyMediaManager are running on both servers
- VM only started when needed

Everything is working perfect, with 2 exceptions:

• 4K transcoding to external. Brings all cores to 100%. Doesn't make sense IMHO, but required for parents ;-) However, 99% is in-house/direct streaming to 8 devices of different kind. The kids turn gigabit cables to red colour.
• Windows 10 VM. Brings both assigned cores to 100%.

 

What I'm thinking about:

• 24 bay case with SAS2 backplane. Direct connection to drives is not required, Expander would be fine.
  Supermicro, Norco, Fantec? Couldn't find out if Fantec's backplane is SAS2, Norco seems to be complicated to build. What do you think? 
• Power
  no idea?
• Motherboard with 2x M.2 (for Cache Pool) and IPMI or alternative
• CPU with integrated graphics (QSV) and at least 12,000-14,000 passmark
  Intel i7-8700K?
• Memory
  32GB, ECC is not required. What do you think about 64GB?
• Controller
  no idea

 

Any suggestions are highly appreciated.

 

Honey

 

[Tybio]

Take a look at the E-2176G or E-2186G, they are Xeon's with ECC that are basically i7-8700s.  The other nice thing is Supermicro has a C246 motherboard with IPMI avaliable for them.  The cost of the CPUs is ~300-~380 and the MB is ~300.  So not much of a premium to get IPMI and ECC with the same power.  Actually the E-2176G with 80 TDW may even be better than a i7-8700k in terms of heat/power while still having the exact same QSV and Passmark score.

 

http://www.supermicro.com/products/motherboard/X11/X11SCA-F.cfm

 

The only down side is the Processors are not shipping just yet, it should be this month.

 

The board should also work with i7's, but SM is now allowed to state that due to Intel wanting people to stick to the roles Intel thinks make them the most money.  You could drop an i7-8700k in there with non-ECC ram, but it isn't "Officially supported" just "Tested".  You can see SMs hinting on this in the CPU section of the page above.

Edited October 17, 2018 by Tybio

[hawihoney]

Thanks for your answer.

 

This combination looks promising. What about Controller. What would you prefer?

 

[Tybio]

The two obvious choices are the AOC-SAS2LP-MV8 or any of the LSI cards.  Generally people don't like to reccomend the SAS2LP as "Some" users have issues (Mostly related to running VMs) with them.  It seems to be an ether-or situation...if you don't have problems at the start it is rock solid for years...or you have a problem nearly instantly and it never goes away.

 

The LSI cards are the choice of the forum, and you can get them off Ebay for cheap.  When I do my upgrade, I'll be moving from SAS2LP to LSI cards, just to be on the safe side.

[hawihoney]

For my two servers I did buy new LSI 9300-8i from eBay a year ago. These are SAS3 cards. The cards came from HongKong and did cost around EUR 120. A used SAS2LP-MV8 is more expensive. What's the difference between both? Can I use both cards with an Expander Backplane?

 

Thanks again.

 

[Tybio]

Depends on what you mean, a backplane like the Noroco chassis aren't "Expanders" they just take 4 ports from the card and break them out to 4 of the hot-swap bays.  So you need a 1:1 ratior of card ports to Hot Swap bays.  If you want an expander like the Intel one that can take 2 channels from one of these cards and break them out to a lot more drives, that's a different question.

 

I've used both cards in hot swap chassis (1:1 port ratio) and they both work fine, the only real issue with the SAS2LPs has been some users reporting issues when using VT-D (VM Passthrough).  The Marvel chip they are based on is the issue, while the LSI cards get strong reccomendations almost universally on the forums.  If I was trying to play it safe, I'd start with an LSI based card, you can find them all over ebay (Some have to be flashed to "IT" mode as their RAID mode is not great for unraid).

 

I'm not sure you can go wrong with either one, just know that there is a small risk the SAS2LP cards might cause issues.

[hawihoney]

I'm screwing my head into the manuals of Supermicro and others currently. If I got it right I can feed a single Supermicro connector (on an EL1 backplane) or two (on an EL2 backplane) with my existing LSI 9300-8i adapter (2 ports with 4 lanes each). For other backplanes as Supermicros TQ, Norcos or Fantec I need more SATA ports. I hate the cabeling in my 15bay server. I can't even imagine cabeling 24 hard drives.

 

 

[Tybio]

They are different connections, they use the mini-sas connectors so you get one cable per 4 hard drives...each port on the LSI or the SM cards supports 4 drives (so an 8 port card it is 2 ports/cables).  You can also get "break out cables" that have one connection at the card side and 4 standard SATA tails hanging off it.

 

To make things even more complex, you can do an "Expander" that takes the two cables from the LSI card and "expands" it.  (Not sure if the SM card supports this)

 

https://www.amazon.com/Intel-RAID-Expander-Card-RES2SV240/dp/B0042NLUVE

 

With that, you take 1 of the LSI cards, connect both ports of that card to the expander...then you get 4xmini sas or 16 drives, doubling the number you can service from one LSI card.  (You can do single-link rather than dual link and get 20 I believe, but they would not get full BW for all 20 drives that way).

 

 

[JorgeB]

28 minutes ago, hawihoney said:

If I got it right I can feed a single Supermicro connector (on an EL1 backplane) or two (on an EL2 backplane) with my existing LSI 9300-8i adapter (2 ports with 4 lanes each).

Correct, these backplanes have a builtin expander.

[Tybio]

I differ to Johnnie in all things unraid, I was bassing my comments off a Norco 24 bay chassis I had which didn't have a built in expander.

[Hoopster]

1 hour ago, Tybio said:

I differ to Johnnie in all things unraid

Do you differ with (do not agree) or defer to (acknowledge his opinion as more informed, even if it differs) Johnnie?  😀

 

Just busting your chops, I assume you meant the latter, but, it reads as the former.  We don't want Johnnie getting all defensive (😉)

[Tybio]

Both!  I have a different level of knowledge so I defer to hi!

[hawihoney]

Thanks a lot.

 

With the LSI 9300-8i card in an x8 or x16 slot of the Supermicro motherboard mentioned in the second post, I will get approx. 8GB/s (4 GB/s for each adapter port). With 24 drives in an EL1 backplane (one expander) this would result in reduced performance during a parity check or rebuild. With an EL2 backplane (configured as two expander, not failover) it would be more than needed (calculated with approx. 200-225MB/s for each harddisk).

 

I think I can live with an EL1. What do you guys think?

 

There's light at the end of the tunnel.

 

[JorgeB]

IIRC the dual expander model is for redundancy only, the single expander already supports dual link, and your speeds are not correct, assuming you'll be using SATA devices, you'll get 2400MB/s with a single miniSAS link, of those around 2200MB/s are usable due to protocol overhead, double that for dual link.

[hawihoney]

Regarding Dual Link on BP-SAS2-846-EL1: Are you sure?

 

According to the BP-SAS2-846EL backplane manual page 3-1 and 3-3:

Quote

SAS2-846EL1 backplanes have a single-port expander that accesses all hard drives and supports cascading.

[...]

SAS2-846EL2 backplanes have dual-port expanders that access all the hard drives. These dual-port expanders support cascading, failover, and multipath

[...]

If multiple HBAs are used to achieve failover protection and load balancing, MPIO software must be installed and correctly configured to perform the load balancing and failover tasks.

For me it seems that the EL1 backplane does not allow Dual Link. Because the last statement, quoted above, is mentioned in a generic section (EL not EL1/EL2).

 

However I would be glad if someone can clarify that for me.

 

Many thanks in advance.

 

[JorgeB]

5 minutes ago, hawihoney said:

Regarding Dual Link on BP-SAS2-846-EL1: Are you sure?

Yes.

 

 

[JorgeB]

6 minutes ago, hawihoney said:

These dual-port expanders support cascading, failover, and multipath

Also failover and multipath are for redundancy, multipath not the same as dual link.