Adrian Posted November 11, 2017 Share Posted November 11, 2017 (edited) So I'm looking at upgrading my unraid server. It's currently running on a Gigabyte Z77X-UD3H with an i3-3245 CPU in a Norco with 19 drives. I'm using 2x AOC-SAS2LP-MV8 plus the onboard SATA ports. On an off it's given me various issues and while it's still chugging along I'm just expecting any day now for it to take a crapper. I was going to upgrade it a while back, but then got distracted with other things. I'm trying to work on this again. I've been reading through various posts and also researching other sites regarding which MB to get and controllers. This is what I've put together so far: Supermicro X11SAT-F i7-6700 3.4 GHz 2x Supermicro Certified MEM-DR480L-HL01-UN21 Hynix Server Memory - 8GB DDR4-2133 2Rx8 Non-ECC UDIMM LSI 9201-16i LSI 9207-8i or Supermicro X11SSM-F-O (cheaper than the workstation motherboard) Xeon E3-1230 V5 3.4 GHz Do I need ECC memory? I got to this choice as it's supports the i7, has IPMI, 4 PCI-E 3.0 in case in the future I want to upgrade the controllers. For the controllers, I wanted to get away from the Marvel based controllers and saw several posts regarding these specific controllers being recommended. I couldn't find any 9201-8i (I'm getting them on Amazon), so hopefully the 9207-8i is good. Looking for thoughts, recommendations, warnings, etc... regarding this build. Anything else I should look at? Alternatives? Thank you, Adrian Edited November 11, 2017 by Adrian Quote Link to comment
t33j4y Posted November 11, 2017 Share Posted November 11, 2017 Apart from my server having taken the crapper yesterday, and not only fearing it, I am like you looking at a new build. Have also looked a lot at the X11SSM-F. I would definitely go for the ECC RAM. Where I am, it is like 10-20% more. If everything else is server level, I wouldn't skimp on getting that extra level of error prevention. Have you considered a V6 Xeon? I also looked at V5 but I've just noticed that fx an E3-1245 v6 is actually marginally cheaper than the E3-1245 v5. Go figure. 1 Quote Link to comment
tdallen Posted November 11, 2017 Share Posted November 11, 2017 Hi - Looks like good choices. I like ECC in an always-on fault tolerant server, so I'd go with your second choice unless you really expect to need the third controller. The 9207-8i should be fine though I believe it will need flashing to IT mode. That said, you should be aware of the Sky Lake / Kaby Lake hyper-threading issue: 1 Quote Link to comment
t33j4y Posted November 11, 2017 Share Posted November 11, 2017 14 minutes ago, tdallen said: Hi - Looks like good choices. I like ECC in an always-on fault tolerant server, so I'd go with your second choice unless you really expect to need the third controller. The 9207-8i should be fine though I believe it will need flashing to IT mode. That said, you should be aware of the Sky Lake / Kaby Lake hyper-threading issue: I believe that for X11SSM there is a 2.0b BIOS update that fixes this issue :-) 1 Quote Link to comment
bobkart Posted November 11, 2017 Share Posted November 11, 2017 (edited) I'll second the ECC suggestion. And the X11SSM. Regarding SAS cards, the 9207's are solid performers; I have about six of those in various 16-to-24-drive setups. Recently I've tried some 9300-8i's with good success. Probably not as cost-effective as the 9207's but certainly more future-proof (12Gb/s versus 6Gb/s). EDIT: Admittedly the 12Gb/s mentioned is only for SAS drives, which don't tend to be used as often in unRAID servers. And of course the speed of the drives involved is usually the bottleneck, not the connection bandwidth. Edited November 12, 2017 by bobkart 1 Quote Link to comment
Adrian Posted November 12, 2017 Author Share Posted November 12, 2017 17 hours ago, t33j4y said: Have you considered a V6 Xeon? I also looked at V5 but I've just noticed that fx an E3-1245 v6 is actually marginally cheaper than the E3-1245 v5. Go figure. Well I'm looking at E3-1230 due to pricing. The V6 is $10 more than the V5, so not much more. I may get the V6, still undecided. 17 hours ago, bobkart said: Regarding SAS cards, the 9207's are solid performers; I have about six of those in various 16-to-24-drive setups. Did you have to do any flashing? Quote Link to comment
t33j4y Posted November 12, 2017 Share Posted November 12, 2017 1 hour ago, Adrian said: Well I'm looking at E3-1230 due to pricing. The V6 is $10 more than the V5, so not much more. I may get the V6, still undecided. Actually a fair point - I've just looked into the E3-1230v6 - I can shave almost USD100 of my cost by accepting a little lower frequency and no GPU. Since I'll be pairing it with the X11SSM which has on-board GPU and this is for server use, there's no actual need for the GPU fitted one. Will consider this before I order tonight. Thanks for bring it to my attention. 1 Quote Link to comment
Adrian Posted November 12, 2017 Author Share Posted November 12, 2017 (edited) 11 minutes ago, t33j4y said: Actually a fair point - I've just looked into the E3-1230v6 - I can shave almost USD100 of my cost by accepting a little lower frequency and no GPU. Since I'll be pairing it with the X11SSM which has on-board GPU and this is for server use, there's no actual need for the GPU fitted one. Oh crap, I completely forgot about GPU availability. Hmm, so are you sure the GPU is included in the MB and it doesn't require a GPU in the CPU? If so yea, that's also like a $70-80 savings for not having to get the E3-1245 Well I guess, in terms of direct console access, IPMI provides a VGA GPU? Edited November 12, 2017 by Adrian Quote Link to comment
t33j4y Posted November 12, 2017 Share Posted November 12, 2017 (edited) The X11SSM has the AST1400 onboard GPU. It's very much not fancy, but okay for basic stuff. Edited November 12, 2017 by t33j4y 1 Quote Link to comment
Adrian Posted November 12, 2017 Author Share Posted November 12, 2017 13 minutes ago, t33j4y said: The X11SSM has the AST1400 onboard GPU. It's very much not fancy, but okay for basic stuff. Ah yes, I was looking at the wrong specs. Thanks! Graphics Aspeed AST1400 Quote Link to comment
Adrian Posted November 12, 2017 Author Share Posted November 12, 2017 ok, so this possibly looks like my final build: Supermicro X11SSM-F-O Xeon E3-1230 V6 3.4 GHz Hynix MEM-DR480L-HL01-EU24 2x 8GB ECC $755 for it all. That's about $80 less than the i7-6700 route. I could shave another $20 if I go with the Micron memory instead. I didn't include the LSI controllers in my comparison since I'm using the same controllers either way, but those will run about $340 more. Quote Link to comment
bobkart Posted November 13, 2017 Share Posted November 13, 2017 9 hours ago, Adrian said: Did you have to do any flashing? ... Hynix MEM-DR480L-HL01-EU24 2x 8GB ECC No flashing; they're 9207-8e's as opposed to 8i's, mostly bought used on eBay, but I believe one was brand new. Regarding memory, I'd lean towards a single 16GB ECC UDIMM instead of two 8GB's. Of course pricing could work against that choice. And: Micron/Crucial are pretty much the same, just FYI. On your controller prices, I see them on eBay in the $60 range. Don't know if that's an option for you. 1 Quote Link to comment
Adrian Posted November 13, 2017 Author Share Posted November 13, 2017 4 minutes ago, bobkart said: No flashing; they're 9207-8e's as opposed to 8i's, mostly bought used on eBay, but I believe one was brand new. So if I get an LSI 9201-16i and 9207-8i, I see them on Amazon, I'll have to flash them? https://www.amazon.com/gp/product/B003UNP05O/ref=ox_sc_act_title_4?smid=A2J5EC07WROWMJ&psc=1 https://www.amazon.com/gp/product/B0085FT2JC/ref=ox_sc_act_title_1?smid=A2J5EC07WROWMJ&psc=1 11 minutes ago, bobkart said: Regarding memory, I'd lean towards a single 16GB ECC UDIMM instead of two 8GB's. Of course pricing could work against that choice. And: Micron/Crucial are pretty much the same, just FYI. Ok, I checked the price and a single 16GB is actually $50 less, thanks! 12 minutes ago, bobkart said: On your controller prices, I see them on eBay in the $60 range. Don't know if that's an option for you. Yea, I'm staying out of ebay if I can for this upgrade if possible. Quote Link to comment
bobkart Posted November 13, 2017 Share Posted November 13, 2017 I can't be sure about the flashing; all I know is I've never needed to flash any 92xx cards I bought, and that count is around a dozen. I did check my empty boxes and see that the one brand-new such card I bought was a 9211-4i, not a 9207-8e. Hopefully someone else can chime in on that. And, as long as we're talking about saving cost, three 9207-8i's look to come in $30-$40 under the two-card option you mention. But maybe your PCIe slots are otherwise occupied. Last thing I noticed is that the 9201-16i connects via PCIe2x8, so 16 channels over ~4GB/s leaves only ~250MB per channel . . . that might be cutting it close for some of the newer/faster drives (8TB and above). 1 Quote Link to comment
Adrian Posted November 13, 2017 Author Share Posted November 13, 2017 (edited) 21 minutes ago, bobkart said: And, as long as we're talking about saving cost, three 9207-8i's look to come in $30-$40 under the two-card option you mention. But maybe your PCIe slots are otherwise occupied. Well the 9207-8is uses x8 lanes, so I thought I could only have 2 x8 cards in the X11SSM-F-O. These would be the only cards, so am I mistaken? Could I have 3 x8 cards? I looked at the PCIe 3.0 24 port card (9305-24i), but that thing is $570. I'm only using 4TB drives (mostly the HGST NAS drives). I only have 20 drives, so I was thinking I'd put 8 on the 9207-8i and 12 on the 16 port card. Would that help deal with any limits I may hit? Edited November 13, 2017 by Adrian Quote Link to comment
bobkart Posted November 13, 2017 Share Posted November 13, 2017 (edited) Good questions. Moving drives away from the 'bottleneck' controller *will* help. But the three-card options still wins; here's why. That board has x8/x8/x4/x4 slots (all PCIe3). The x4 slots are the concern. They'll have ~4GB/s bandwidth instead of ~8GB/s. Splitting that over eight channels leaves ~500MB/s per channel, plenty for a spinning drive. This of course leaves out concerns of overall motherboard PCIe bandwidth, but that's present regardless of how the drives connect. The -16i option loses only because of the revision 2 PCIe connection, which ends up quartering the per-channel bandwidth instead of just halving it, otherwise it'd be pretty much a toss-up between the two options (cost notwithstanding). Probably what you weren't considering is that you *can* run an x8 card in an x4 slot (if physically compatible). The card and host figure it out and you just get a bandwidth reduction. Edited November 13, 2017 by bobkart 1 Quote Link to comment
t33j4y Posted November 13, 2017 Share Posted November 13, 2017 (edited) 19 hours ago, Adrian said: ok, so this possibly looks like my final build: Supermicro X11SSM-F-O Xeon E3-1230 V6 3.4 GHz Hynix MEM-DR480L-HL01-EU24 2x 8GB ECC $755 for it all. That's about $80 less than the i7-6700 route. I could shave another $20 if I go with the Micron memory instead. 1 I went ahead and ordered the 1230 v6 last night, saved a lot based on your input. Thanks! 10 hours ago, bobkart said: Regarding memory, I'd lean towards a single 16GB ECC UDIMM instead of two 8GB's. Of course pricing could work against that choice. And: Micron/Crucial are pretty much the same, just FYI. 1 I think Crucial is actually the "consumer arm" of Micron? 9 hours ago, Adrian said: Ok, I checked the price and a single 16GB is actually $50 less, thanks! In my case, 2x8GB (Kingston Server Premier) were actually cheaper than going for the 1x16GB module. Just goes to show that good planning can lead to good savings :-) Edited November 13, 2017 by t33j4y leave=lead (correcting autocorrect 1 Quote Link to comment
Adrian Posted November 13, 2017 Author Share Posted November 13, 2017 4 hours ago, t33j4y said: In my case, 2x8GB (Kingston Server Premier) were actually cheaper than going for the 1x16GB module. Just goes to show that good planning can leave to good savings :-) Did you select it from the list on Kingston's site? Quote Link to comment
t33j4y Posted November 13, 2017 Share Posted November 13, 2017 I found it the other way around by seeing what was available for immediate delivery and then checked on Kingstons site. Quote Link to comment
Adrian Posted November 13, 2017 Author Share Posted November 13, 2017 8 hours ago, bobkart said: That board has x8/x8/x4/x4 slots (all PCIe3). The x4 slots are the concern. They'll have ~4GB/s bandwidth instead of ~8GB/s. Splitting that over eight channels leaves ~500MB/s per channel, plenty for a spinning drive. This of course leaves out concerns of overall motherboard PCIe bandwidth, but that's present regardless of how the drives connect. ok, so considering that the drives I'm using won't come close to 500MB/s (at most with mechanical drives will be in the 200s?) using the 8x board in the 4x slot should be fine then right? Otherwise, in order to get 3x 8x slots, I'm looking at the workstation board, which even with the savings on getting 3x 8 port controllers, puts me at $40 more. I have an SSD that I use for cache, but that will be on the onboard controller. Quote Link to comment
Adrian Posted November 13, 2017 Author Share Posted November 13, 2017 5 hours ago, t33j4y said: I went ahead and ordered the 1230 v6 last night, saved a lot based on your input. Thanks! Did you by any chance get it on Amazon? Quote Link to comment
t33j4y Posted November 13, 2017 Share Posted November 13, 2017 50 minutes ago, Adrian said: Did you by any chance get it on Amazon? Nope, a local retailer here in Denmark :-) Quote Link to comment
bobkart Posted November 13, 2017 Share Posted November 13, 2017 5 hours ago, Adrian said: ok, so considering that the drives I'm using won't come close to 500MB/s (at most with mechanical drives will be in the 200s?) using the 8x board in the 4x slot should be fine then right? Otherwise, in order to get 3x 8x slots, I'm looking at the workstation board, which even with the savings on getting 3x 8 port controllers, puts me at $40 more. I have an SSD that I use for cache, but that will be on the onboard controller. Yes, using the x8 controller in an x4 slot will only reduce the per-channel bandwidth to ~500MB/s, which is easily enough for a spinning drive; a 7,200RPM 8TB drive might just break above 250MB/s for the outer cylinders. And, note that the X11SSL, which drops one PCIex4 slot compared to the X11SSM, will suffice if all you need is the three total PCIe slots. You might be able to find the SSL at a lower price than the SSM. Quote Link to comment
Adrian Posted November 13, 2017 Author Share Posted November 13, 2017 8 minutes ago, bobkart said: And, note that the X11SSL, which drops one PCIex4 slot compared to the X11SSM, will suffice if all you need is the three total PCIe slots. You might be able to find the SSL at a lower price than the SSM. Thanks for all the help. I went ahead and ordered the MBD-X11SSM-F-O to have that extra free PCIe slot just in case I need it in the future. I should have everything by Wed so hopefully I can have it up and running over the weekend. I'll let you know how it goes. Quote Link to comment
t33j4y Posted November 13, 2017 Share Posted November 13, 2017 Adrian, if you go for rebranded LSI controllers, be sure to check the advice given to me by johnnie.black in this thread: It only pertains to some controllers, but I am unsure what you're ending up with, so just in case :-) Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.