Jump to content
Adrian

Upgrade Time

29 posts in this topic Last Reply

Recommended Posts

So I'm looking at upgrading my unraid server. It's currently running on a Gigabyte Z77X-UD3H with an i3-3245 CPU in a Norco with 19 drives. I'm using 2x AOC-SAS2LP-MV8  plus the onboard SATA ports. On an off it's given me various issues and while it's still chugging along I'm just expecting any day now for it to take a crapper. I was going to upgrade it a while back, but then got distracted with other things. I'm trying to work on this again. I've been reading through various posts and also researching other sites regarding which MB to get and controllers. This is what I've put together so far:

 

Supermicro X11SAT-F

i7-6700 3.4 GHz

2x Supermicro Certified MEM-DR480L-HL01-UN21 Hynix Server Memory - 8GB DDR4-2133 2Rx8 Non-ECC UDIMM

LSI 9201-16i

LSI 9207-8i

 

or

 

Supermicro X11SSM-F-O (cheaper than the workstation motherboard)

Xeon E3-1230 V5 3.4 GHz

Do I need ECC memory?

 

I got to this choice as it's supports the i7, has IPMI, 4 PCI-E 3.0 in case in the future I want to upgrade the controllers.

 

For the controllers, I wanted to get away from the Marvel based controllers and saw several posts regarding these specific controllers being recommended. I couldn't find any 9201-8i (I'm getting them on Amazon), so hopefully the 9207-8i is good.

 

Looking for thoughts, recommendations, warnings, etc... regarding this build. Anything else I should look at? Alternatives?

 

Thank you,

Adrian

Edited by Adrian

Share this post


Link to post

Apart from my server having taken the crapper yesterday, and not only fearing it, I am like you looking at a new build. 

 

Have also looked a lot at the X11SSM-F. I would definitely go for the ECC RAM. Where I am, it is like 10-20% more. If everything else is server level, I wouldn't skimp on getting that extra level of error prevention.

 

Have you considered a V6 Xeon? I also looked at V5 but I've just noticed that fx an E3-1245 v6 is actually marginally cheaper than the E3-1245 v5. Go figure.

  • Like 1

Share this post


Link to post

Hi -

 

Looks like good choices.  I like ECC in an always-on fault tolerant server, so I'd go with your second choice unless you really expect to need the third controller.  The 9207-8i should be fine though I believe it will need flashing to IT mode.  That said, you should be aware of the Sky Lake / Kaby Lake hyper-threading issue:

 

 

 

  • Like 1

Share this post


Link to post
14 minutes ago, tdallen said:

Hi -

 

Looks like good choices.  I like ECC in an always-on fault tolerant server, so I'd go with your second choice unless you really expect to need the third controller.  The 9207-8i should be fine though I believe it will need flashing to IT mode.  That said, you should be aware of the Sky Lake / Kaby Lake hyper-threading issue:

 

 

 

I believe that for X11SSM there is a 2.0b BIOS update that fixes this issue :-)

  • Like 1

Share this post


Link to post

I'll second the ECC suggestion.  And the X11SSM.

 

Regarding SAS cards, the 9207's are solid performers; I have about six of those in various 16-to-24-drive setups.  Recently I've tried some 9300-8i's with good success.  Probably not as cost-effective as the 9207's but certainly more future-proof (12Gb/s versus 6Gb/s).

 

EDIT: Admittedly the 12Gb/s mentioned is only for SAS drives, which don't tend to be used as often in unRAID servers.  And of course the speed of the drives involved is usually the bottleneck, not the connection bandwidth.

Edited by bobkart
  • Like 1

Share this post


Link to post
17 hours ago, t33j4y said:

Have you considered a V6 Xeon? I also looked at V5 but I've just noticed that fx an E3-1245 v6 is actually marginally cheaper than the E3-1245 v5. Go figure.

Well I'm looking at E3-1230 due to pricing. The V6 is $10 more than the V5, so not much more. I may get the V6, still undecided.

 

17 hours ago, bobkart said:

Regarding SAS cards, the 9207's are solid performers; I have about six of those in various 16-to-24-drive setups.

Did you have to do any flashing?

Share this post


Link to post
1 hour ago, Adrian said:

Well I'm looking at E3-1230 due to pricing. The V6 is $10 more than the V5, so not much more. I may get the V6, still undecided.

 

 

Actually a fair point - I've just looked into the E3-1230v6 - I can shave almost USD100 of my cost by accepting a little lower frequency and no GPU. Since I'll be pairing it with the X11SSM which has on-board GPU and this is for server use, there's no actual need for the GPU fitted one.

 

Will consider this before I order tonight. Thanks for bring it to my attention.

  • Like 1

Share this post


Link to post
11 minutes ago, t33j4y said:

Actually a fair point - I've just looked into the E3-1230v6 - I can shave almost USD100 of my cost by accepting a little lower frequency and no GPU. Since I'll be pairing it with the X11SSM which has on-board GPU and this is for server use, there's no actual need for the GPU fitted one.

Oh crap, I completely forgot about GPU availability. Hmm, so are you sure the GPU is included in the MB and it doesn't require a GPU in the CPU? If so yea, that's also like a $70-80 savings for not having to get the E3-1245

 

Well I guess, in terms of direct console access, IPMI provides a VGA GPU?

Edited by Adrian

Share this post


Link to post

The X11SSM has the AST1400 onboard GPU. It's very much not fancy, but okay for basic stuff.

Edited by t33j4y
  • Like 1

Share this post


Link to post
13 minutes ago, t33j4y said:

The X11SSM has the AST1400 onboard GPU. It's very much not fancy, but okay for basic stuff.

Ah yes, I was looking at the wrong specs. Thanks!

Graphics
  • Aspeed AST1400
 

Share this post


Link to post

ok, so this possibly looks like my final build:

 

Supermicro X11SSM-F-O

Xeon E3-1230 V6 3.4 GHz

Hynix MEM-DR480L-HL01-EU24 2x 8GB ECC

 

$755 for it all. That's about $80 less than the i7-6700 route. I could shave another $20 if I go with the Micron memory instead.

 

I didn't include the LSI controllers in my comparison since I'm using the same controllers either way, but those will run about $340 more.

 

Share this post


Link to post
9 hours ago, Adrian said:

Did you have to do any flashing?

...

Hynix MEM-DR480L-HL01-EU24 2x 8GB ECC

 

No flashing; they're 9207-8e's as opposed to 8i's, mostly bought used on eBay, but I believe one was brand new.

 

Regarding memory, I'd lean towards a single 16GB ECC UDIMM instead of two 8GB's.  Of course pricing could work against that choice.  And: Micron/Crucial are pretty much the same, just FYI.

 

On your controller prices, I see them on eBay in the $60 range.  Don't know if that's an option for you.

  • Like 1

Share this post


Link to post
4 minutes ago, bobkart said:

No flashing; they're 9207-8e's as opposed to 8i's, mostly bought used on eBay, but I believe one was brand new.

So if I get an LSI 9201-16i and 9207-8i, I see them on Amazon, I'll have to flash them?

 

https://www.amazon.com/gp/product/B003UNP05O/ref=ox_sc_act_title_4?smid=A2J5EC07WROWMJ&psc=1

https://www.amazon.com/gp/product/B0085FT2JC/ref=ox_sc_act_title_1?smid=A2J5EC07WROWMJ&psc=1

 

11 minutes ago, bobkart said:

Regarding memory, I'd lean towards a single 16GB ECC UDIMM instead of two 8GB's.  Of course pricing could work against that choice.  And: Micron/Crucial are pretty much the same, just FYI.

Ok, I checked the price and a single 16GB is actually $50 less, thanks!

 

12 minutes ago, bobkart said:

On your controller prices, I see them on eBay in the $60 range.  Don't know if that's an option for you.

Yea, I'm staying out of ebay if I can for this upgrade if possible.

Share this post


Link to post

I can't be sure about the flashing; all I know is I've never needed to flash any 92xx cards I bought, and that count is around a dozen.  I did check my empty boxes and see that the one brand-new such card I bought was a 9211-4i, not a 9207-8e.  Hopefully someone else can chime in on that.

 

And, as long as we're talking about saving cost, three 9207-8i's look to come in $30-$40 under the two-card option you mention.  But maybe your PCIe slots are otherwise occupied.

 

Last thing I noticed is that the 9201-16i connects via PCIe2x8, so 16 channels over ~4GB/s leaves only ~250MB per channel . . . that might be cutting it close for some of the newer/faster drives (8TB and above).

  • Like 1

Share this post


Link to post
21 minutes ago, bobkart said:

And, as long as we're talking about saving cost, three 9207-8i's look to come in $30-$40 under the two-card option you mention.  But maybe your PCIe slots are otherwise occupied.

Well the 9207-8is uses x8 lanes, so I thought I could only have 2 x8 cards in the X11SSM-F-O. These would be the only cards, so am I mistaken? Could I have 3 x8 cards?

 I looked at the PCIe 3.0 24 port card (9305-24i), but that thing is $570. I'm only using 4TB drives (mostly the HGST NAS drives). I only have 20 drives, so I was thinking I'd put 8 on the 9207-8i and 12 on the 16 port card. Would that help deal with any limits I may hit?

Edited by Adrian

Share this post


Link to post

Good questions.  Moving drives away from the 'bottleneck' controller *will* help.  But the three-card options still wins; here's why.

 

That board has x8/x8/x4/x4 slots (all PCIe3).  The x4 slots are the concern.  They'll have ~4GB/s bandwidth instead of ~8GB/s.  Splitting that over eight channels leaves ~500MB/s per channel, plenty for a spinning drive.  This of course leaves out concerns of overall motherboard PCIe bandwidth, but that's present regardless of how the drives connect.

 

The -16i option loses only because of the revision 2 PCIe connection, which ends up quartering the per-channel bandwidth instead of just halving it, otherwise it'd be pretty much a toss-up between the two options (cost notwithstanding).

 

Probably what you weren't considering is that you *can* run an x8 card in an x4 slot (if physically compatible).  The card and host figure it out and you just get a bandwidth reduction.

Edited by bobkart
  • Like 1

Share this post


Link to post
19 hours ago, Adrian said:

ok, so this possibly looks like my final build:

 

Supermicro X11SSM-F-O

Xeon E3-1230 V6 3.4 GHz

Hynix MEM-DR480L-HL01-EU24 2x 8GB ECC

 

$755 for it all. That's about $80 less than the i7-6700 route. I could shave another $20 if I go with the Micron memory instead.

1

 

I went ahead and ordered the 1230 v6 last night, saved a lot based on your input. Thanks! 

 

10 hours ago, bobkart said:

Regarding memory, I'd lean towards a single 16GB ECC UDIMM instead of two 8GB's.  Of course pricing could work against that choice.  And: Micron/Crucial are pretty much the same, just FYI.

1

 

I think Crucial is actually the "consumer arm" of Micron?

 

9 hours ago, Adrian said:

Ok, I checked the price and a single 16GB is actually $50 less, thanks!

 

 

In my case, 2x8GB (Kingston Server Premier) were actually cheaper than going for the 1x16GB module. Just goes to show that good planning can lead to good savings :-)

Edited by t33j4y
leave=lead (correcting autocorrect
  • Like 1

Share this post


Link to post
4 hours ago, t33j4y said:

In my case, 2x8GB (Kingston Server Premier) were actually cheaper than going for the 1x16GB module. Just goes to show that good planning can leave to good savings :-)

Did you select it from the list on Kingston's site?

Share this post


Link to post

I found it the other way around by seeing what was available for immediate delivery and then checked on Kingstons site.

Share this post


Link to post
8 hours ago, bobkart said:

That board has x8/x8/x4/x4 slots (all PCIe3).  The x4 slots are the concern.  They'll have ~4GB/s bandwidth instead of ~8GB/s.  Splitting that over eight channels leaves ~500MB/s per channel, plenty for a spinning drive.  This of course leaves out concerns of overall motherboard PCIe bandwidth, but that's present regardless of how the drives connect.

ok, so considering that the drives I'm using won't come close to 500MB/s (at most with mechanical drives will be in the 200s?) using the 8x board in the 4x slot should be fine then right? Otherwise, in order to get 3x 8x slots, I'm looking at the workstation board, which even with the savings on getting 3x 8 port controllers, puts me at $40 more.

 

I have an SSD that I use for cache, but that will be on the onboard controller.

 

 

Share this post


Link to post
5 hours ago, t33j4y said:

I went ahead and ordered the 1230 v6 last night, saved a lot based on your input. Thanks! 

Did you by any chance get it on Amazon?

Share this post


Link to post
50 minutes ago, Adrian said:

Did you by any chance get it on Amazon?

 

Nope, a local retailer here in Denmark :-)

Share this post


Link to post
5 hours ago, Adrian said:

ok, so considering that the drives I'm using won't come close to 500MB/s (at most with mechanical drives will be in the 200s?) using the 8x board in the 4x slot should be fine then right? Otherwise, in order to get 3x 8x slots, I'm looking at the workstation board, which even with the savings on getting 3x 8 port controllers, puts me at $40 more.

 

I have an SSD that I use for cache, but that will be on the onboard controller.

 

Yes, using the x8 controller in an x4 slot will only reduce the per-channel bandwidth to ~500MB/s, which is easily enough for a spinning drive; a 7,200RPM 8TB drive might just break above 250MB/s for the outer cylinders.

 

And, note that the X11SSL, which drops one PCIex4 slot compared to the X11SSM, will suffice if all you need is the three total PCIe slots.  You might be able to find the SSL at a lower price than the SSM.

Share this post


Link to post
8 minutes ago, bobkart said:

And, note that the X11SSL, which drops one PCIex4 slot compared to the X11SSM, will suffice if all you need is the three total PCIe slots.  You might be able to find the SSL at a lower price than the SSM.

Thanks for all the help. I went ahead and ordered the MBD-X11SSM-F-O to have that extra free PCIe slot just in case I need it in the future. I should have everything by Wed so hopefully I can have it up and running over the weekend. I'll let you know how it goes.

Share this post


Link to post

Adrian, if you go for rebranded LSI controllers, be sure to check the advice given to me by johnnie.black in this thread:

 

 

It only pertains to some controllers, but I am unsure what you're ending up with, so just in case :-)

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now